2023-09-30T01:21:10.021 INFO:root:teuthology version: 0.0.1.dev194+g54e62bc 2023-09-30T01:21:10.021 DEBUG:teuthology.run:Teuthology command: teuthology --verbose --archive /home/teuthworker/archive/yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi/7406999 --name yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi --owner scheduled_yuriw@teuthology --description rados/monthrash/{ceph clusters/9-mons mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-hybrid rados supported-random-distro$/{ubuntu_latest} thrashers/many workloads/rados_mon_osdmap_prune} -- /home/teuthworker/archive/yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi/7406999/orig.config.yaml 2023-09-30T01:21:10.063 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-09-30T01:21:10.149 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi/7406999 branch: wip-yuri4-testing-2023-09-29-0928-quincy description: rados/monthrash/{ceph clusters/9-mons mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-hybrid rados supported-random-distro$/{ubuntu_latest} thrashers/many workloads/rados_mon_osdmap_prune} email: yweinste@redhat.com first_in_suite: false job_id: '7406999' kernel: kdb: true sha1: distro ktype: distro last_in_suite: false machine_type: smithi name: yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 3 size: 10 os_type: ubuntu os_version: '20.04' overrides: admin_socket: branch: wip-yuri4-testing-2023-09-29-0928-quincy ceph: conf: client: debug monc: 20 debug ms: 1 global: mon client directed command retry: 5 mon election default strategy: 3 ms bind msgr1: false ms bind msgr2: true ms inject socket failures: 5000 ms type: async mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon debug extra checks: true mon min osdmap epochs: 100 mon osdmap full prune enabled: true mon osdmap full prune interval: 10 mon osdmap full prune min: 200 mon osdmap full prune txsize: 100 mon scrub inject crc mismatch: 0.01 mon scrub inject missing keys: 0.05 mon scrub interval: 300 paxos service trim min: 5 osd: bluefs allocator: hybrid bluestore allocator: hybrid bluestore block size: 96636764160 bluestore fsck on mount: true bluestore zero block detection: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon client ping interval: 4 mon client ping timeout: 12 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd beacon report interval: 10 osd debug verify cached snaps: true osd debug verify missing on start: true osd failsafe full ratio: 0.95 osd mclock override recovery settings: true osd mclock profile: high_recovery_ops osd objectstore: bluestore osd op queue: debug_random osd op queue cut off: debug_random flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - ScrubResult - scrub mismatch - overall HEALTH_ - \(MGR_DOWN\) - \(PG_AVAILABILITY\) - \(SLOW_OPS\) - slow request - \(OSD_SLOW_PING_TIME - overall HEALTH_ - \(MON_DOWN\) - \(OSDMAP_FLAGS\) sha1: 93dce5e17918f8aca65993a7a2dc40233a47b481 ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: flavor: default sha1: 93dce5e17918f8aca65993a7a2dc40233a47b481 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-yuri4-testing-2023-09-29-0928-quincy sha1: 93dce5e17918f8aca65993a7a2dc40233a47b481 owner: scheduled_yuriw@teuthology priority: 75 repo: https://github.com/ceph/ceph-ci.git roles: - - mon.a - mon.b - mon.c - mon.d - mon.e - osd.0 - osd.1 - osd.2 - - mon.f - mon.g - mon.h - mon.i - mgr.x - osd.3 - osd.4 - osd.5 - client.0 seed: 9810 sha1: 93dce5e17918f8aca65993a7a2dc40233a47b481 sleep_before_teardown: 0 subset: 111/120000 suite: rados suite_branch: wip-yuri4-testing-2023-09-29-0928-quincy suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_93dce5e17918f8aca65993a7a2dc40233a47b481/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: 93dce5e17918f8aca65993a7a2dc40233a47b481 targets: smithi088.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFyT/r7xuKfmg50w0OdQebr4A/ptJo1Zk6DyZNZNZM/q5rW+sznh+wvHqeT3e4YtCdZLaXyvMms8o8u8H5nXOYM= smithi107.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFyT/r7xuKfmg50w0OdQebr4A/ptJo1Zk6DyZNZNZM/q5rW+sznh+wvHqeT3e4YtCdZLaXyvMms8o8u8H5nXOYM= tasks: - install: null - ceph: null - mon_thrash: freeze_mon_duration: 20 freeze_mon_probability: 10 revive_delay: 20 thrash_delay: 1 thrash_many: true - workunit: clients: client.0: - mon/test_mon_osdmap_prune.sh teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 54e62bcbac4e53d9685e08328b790d3b20d71cae timestamp: 2023-09-29_19:38:56 tube: smithi user: yuriw verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2830 2023-09-30T01:21:10.150 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_93dce5e17918f8aca65993a7a2dc40233a47b481/qa; will attempt to use it 2023-09-30T01:21:10.151 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_93dce5e17918f8aca65993a7a2dc40233a47b481/qa/tasks 2023-09-30T01:21:10.151 INFO:teuthology.run_tasks:Running task internal.check_packages... 2023-09-30T01:21:10.152 INFO:teuthology.task.internal:Checking packages... 2023-09-30T01:21:10.178 INFO:teuthology.task.internal:Checking packages for os_type 'ubuntu', flavor 'default' and ceph hash '93dce5e17918f8aca65993a7a2dc40233a47b481' 2023-09-30T01:21:10.178 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2023-09-30T01:21:10.179 INFO:teuthology.packaging:ref: None 2023-09-30T01:21:10.179 INFO:teuthology.packaging:tag: None 2023-09-30T01:21:10.179 INFO:teuthology.packaging:branch: wip-yuri4-testing-2023-09-29-0928-quincy 2023-09-30T01:21:10.179 INFO:teuthology.packaging:sha1: 93dce5e17918f8aca65993a7a2dc40233a47b481 2023-09-30T01:21:10.179 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&ref=wip-yuri4-testing-2023-09-29-0928-quincy 2023-09-30T01:21:10.394 INFO:teuthology.task.internal:Found packages for ceph version 17.2.6-1165-g93dce5e1-1focal 2023-09-30T01:21:10.394 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2023-09-30T01:21:10.401 INFO:teuthology.task.internal:no buildpackages task found 2023-09-30T01:21:10.402 INFO:teuthology.run_tasks:Running task internal.save_config... 2023-09-30T01:21:10.437 INFO:teuthology.task.internal:Saving configuration 2023-09-30T01:21:10.453 INFO:teuthology.run_tasks:Running task internal.check_lock... 2023-09-30T01:21:10.470 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-09-30T01:21:10.496 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi088.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi/7406999', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '20.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-09-30 01:14:57.489390', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIGETOy3xONfbx7kUFCNiUtRLNfkcrcEsWzPdq6Np/6Bw'} 2023-09-30T01:21:10.518 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi107.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi/7406999', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '20.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-09-30 01:14:57.490731', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIGETOy3xONfbx7kUFCNiUtRLNfkcrcEsWzPdq6Np/6Bw'} 2023-09-30T01:21:10.518 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2023-09-30T01:21:10.529 INFO:teuthology.task.internal:roles: ubuntu@smithi088.front.sepia.ceph.com - ['mon.a', 'mon.b', 'mon.c', 'mon.d', 'mon.e', 'osd.0', 'osd.1', 'osd.2'] 2023-09-30T01:21:10.529 INFO:teuthology.task.internal:roles: ubuntu@smithi107.front.sepia.ceph.com - ['mon.f', 'mon.g', 'mon.h', 'mon.i', 'mgr.x', 'osd.3', 'osd.4', 'osd.5', 'client.0'] 2023-09-30T01:21:10.530 INFO:teuthology.run_tasks:Running task console_log... 2023-09-30T01:21:10.616 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fdc572d8040>, signals=[15]) 2023-09-30T01:21:10.616 INFO:teuthology.run_tasks:Running task internal.connect... 2023-09-30T01:21:10.623 INFO:teuthology.task.internal:Opening connections... 2023-09-30T01:21:10.623 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi088.front.sepia.ceph.com 2023-09-30T01:21:10.625 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi088.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-09-30T01:21:10.670 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi107.front.sepia.ceph.com 2023-09-30T01:21:10.671 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi107.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-09-30T01:21:10.718 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2023-09-30T01:21:10.726 DEBUG:teuthology.orchestra.run.smithi088:> uname -m 2023-09-30T01:21:10.753 INFO:teuthology.orchestra.run.smithi088.stdout:x86_64 2023-09-30T01:21:10.754 DEBUG:teuthology.orchestra.run.smithi088:> cat /etc/os-release 2023-09-30T01:21:10.800 INFO:teuthology.orchestra.run.smithi088.stdout:NAME="Ubuntu" 2023-09-30T01:21:10.800 INFO:teuthology.orchestra.run.smithi088.stdout:VERSION="20.04.4 LTS (Focal Fossa)" 2023-09-30T01:21:10.800 INFO:teuthology.orchestra.run.smithi088.stdout:ID=ubuntu 2023-09-30T01:21:10.800 INFO:teuthology.orchestra.run.smithi088.stdout:ID_LIKE=debian 2023-09-30T01:21:10.800 INFO:teuthology.orchestra.run.smithi088.stdout:PRETTY_NAME="Ubuntu 20.04.4 LTS" 2023-09-30T01:21:10.800 INFO:teuthology.orchestra.run.smithi088.stdout:VERSION_ID="20.04" 2023-09-30T01:21:10.800 INFO:teuthology.orchestra.run.smithi088.stdout:HOME_URL="https://www.ubuntu.com/" 2023-09-30T01:21:10.801 INFO:teuthology.orchestra.run.smithi088.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2023-09-30T01:21:10.801 INFO:teuthology.orchestra.run.smithi088.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2023-09-30T01:21:10.801 INFO:teuthology.orchestra.run.smithi088.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2023-09-30T01:21:10.801 INFO:teuthology.orchestra.run.smithi088.stdout:VERSION_CODENAME=focal 2023-09-30T01:21:10.801 INFO:teuthology.orchestra.run.smithi088.stdout:UBUNTU_CODENAME=focal 2023-09-30T01:21:10.802 INFO:teuthology.lock.ops:Updating smithi088.front.sepia.ceph.com on lock server 2023-09-30T01:21:10.825 DEBUG:teuthology.orchestra.run.smithi107:> uname -m 2023-09-30T01:21:10.832 INFO:teuthology.orchestra.run.smithi107.stdout:x86_64 2023-09-30T01:21:10.832 DEBUG:teuthology.orchestra.run.smithi107:> cat /etc/os-release 2023-09-30T01:21:10.879 INFO:teuthology.orchestra.run.smithi107.stdout:NAME="Ubuntu" 2023-09-30T01:21:10.880 INFO:teuthology.orchestra.run.smithi107.stdout:VERSION="20.04.4 LTS (Focal Fossa)" 2023-09-30T01:21:10.880 INFO:teuthology.orchestra.run.smithi107.stdout:ID=ubuntu 2023-09-30T01:21:10.880 INFO:teuthology.orchestra.run.smithi107.stdout:ID_LIKE=debian 2023-09-30T01:21:10.880 INFO:teuthology.orchestra.run.smithi107.stdout:PRETTY_NAME="Ubuntu 20.04.4 LTS" 2023-09-30T01:21:10.880 INFO:teuthology.orchestra.run.smithi107.stdout:VERSION_ID="20.04" 2023-09-30T01:21:10.880 INFO:teuthology.orchestra.run.smithi107.stdout:HOME_URL="https://www.ubuntu.com/" 2023-09-30T01:21:10.880 INFO:teuthology.orchestra.run.smithi107.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2023-09-30T01:21:10.880 INFO:teuthology.orchestra.run.smithi107.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2023-09-30T01:21:10.880 INFO:teuthology.orchestra.run.smithi107.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2023-09-30T01:21:10.881 INFO:teuthology.orchestra.run.smithi107.stdout:VERSION_CODENAME=focal 2023-09-30T01:21:10.881 INFO:teuthology.orchestra.run.smithi107.stdout:UBUNTU_CODENAME=focal 2023-09-30T01:21:10.881 INFO:teuthology.lock.ops:Updating smithi107.front.sepia.ceph.com on lock server 2023-09-30T01:21:10.900 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2023-09-30T01:21:10.911 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2023-09-30T01:21:10.943 INFO:teuthology.task.internal:Checking for old test directory... 2023-09-30T01:21:10.943 DEBUG:teuthology.orchestra.run.smithi088:> test '!' -e /home/ubuntu/cephtest 2023-09-30T01:21:10.947 DEBUG:teuthology.orchestra.run.smithi107:> test '!' -e /home/ubuntu/cephtest 2023-09-30T01:21:10.953 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2023-09-30T01:21:10.976 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2023-09-30T01:21:10.977 DEBUG:teuthology.orchestra.run.smithi088:> test -z $(ls -A /var/lib/ceph) 2023-09-30T01:21:10.992 DEBUG:teuthology.orchestra.run.smithi107:> test -z $(ls -A /var/lib/ceph) 2023-09-30T01:21:11.008 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2023-09-30T01:21:11.067 INFO:teuthology.run_tasks:Running task kernel... 2023-09-30T01:21:11.081 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2023-09-30T01:21:11.081 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.f': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2023-09-30T01:21:11.081 DEBUG:teuthology.orchestra.run.smithi088:> test -f /run/.containerenv -o -f /.dockerenv 2023-09-30T01:21:11.082 DEBUG:teuthology.orchestra.run.smithi107:> test -f /run/.containerenv -o -f /.dockerenv 2023-09-30T01:21:11.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-09-30T01:21:11.087 DEBUG:teuthology.orchestra.run.smithi088:> uname -r 2023-09-30T01:21:11.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-09-30T01:21:11.088 DEBUG:teuthology.orchestra.run.smithi107:> uname -r 2023-09-30T01:21:11.095 INFO:teuthology.orchestra.run.smithi088.stdout:5.4.0-124-generic 2023-09-30T01:21:11.095 INFO:teuthology.task.kernel:Running kernel on smithi088: 5.4.0-124-generic 2023-09-30T01:21:11.095 DEBUG:teuthology.orchestra.run.smithi088:> sudo apt-get clean 2023-09-30T01:21:11.096 INFO:teuthology.orchestra.run.smithi107.stdout:5.4.0-124-generic 2023-09-30T01:21:11.096 INFO:teuthology.task.kernel:Running kernel on smithi107: 5.4.0-124-generic 2023-09-30T01:21:11.096 DEBUG:teuthology.orchestra.run.smithi107:> sudo apt-get clean 2023-09-30T01:21:11.204 DEBUG:teuthology.orchestra.run.smithi088:> sudo apt-get update 2023-09-30T01:21:11.245 DEBUG:teuthology.orchestra.run.smithi107:> sudo apt-get update 2023-09-30T01:21:11.425 INFO:teuthology.orchestra.run.smithi107.stdout:Hit:1 http://archive.ubuntu.com/ubuntu focal InRelease 2023-09-30T01:21:11.428 INFO:teuthology.orchestra.run.smithi107.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal-updates InRelease [114 kB] 2023-09-30T01:21:11.428 INFO:teuthology.orchestra.run.smithi107.stdout:Get:3 http://security.ubuntu.com/ubuntu focal-security InRelease [114 kB] 2023-09-30T01:21:11.486 INFO:teuthology.orchestra.run.smithi107.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease [108 kB] 2023-09-30T01:21:11.530 INFO:teuthology.orchestra.run.smithi088.stdout:Hit:1 http://archive.ubuntu.com/ubuntu focal InRelease 2023-09-30T01:21:11.531 INFO:teuthology.orchestra.run.smithi088.stdout:Get:2 http://security.ubuntu.com/ubuntu focal-security InRelease [114 kB] 2023-09-30T01:21:11.563 INFO:teuthology.orchestra.run.smithi088.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease [114 kB] 2023-09-30T01:21:11.813 INFO:teuthology.orchestra.run.smithi107.stdout:Get:5 http://security.ubuntu.com/ubuntu focal-security/main i386 Packages [652 kB] 2023-09-30T01:21:11.906 INFO:teuthology.orchestra.run.smithi088.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease [108 kB] 2023-09-30T01:21:11.913 INFO:teuthology.orchestra.run.smithi107.stdout:Get:6 http://security.ubuntu.com/ubuntu focal-security/main amd64 Packages [2,464 kB] 2023-09-30T01:21:11.949 INFO:teuthology.orchestra.run.smithi107.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal-updates/main i386 Packages [885 kB] 2023-09-30T01:21:11.960 INFO:teuthology.orchestra.run.smithi107.stdout:Get:8 http://security.ubuntu.com/ubuntu focal-security/main Translation-en [385 kB] 2023-09-30T01:21:11.964 INFO:teuthology.orchestra.run.smithi107.stdout:Get:9 http://security.ubuntu.com/ubuntu focal-security/main amd64 c-n-f Metadata [13.2 kB] 2023-09-30T01:21:11.964 INFO:teuthology.orchestra.run.smithi107.stdout:Get:10 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 Packages [2,218 kB] 2023-09-30T01:21:11.976 INFO:teuthology.orchestra.run.smithi088.stdout:Get:5 http://security.ubuntu.com/ubuntu focal-security/main amd64 Packages [2,464 kB] 2023-09-30T01:21:11.985 INFO:teuthology.orchestra.run.smithi107.stdout:Get:11 http://security.ubuntu.com/ubuntu focal-security/restricted i386 Packages [33.3 kB] 2023-09-30T01:21:11.986 INFO:teuthology.orchestra.run.smithi107.stdout:Get:12 http://security.ubuntu.com/ubuntu focal-security/restricted Translation-en [309 kB] 2023-09-30T01:21:11.988 INFO:teuthology.orchestra.run.smithi107.stdout:Get:13 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 c-n-f Metadata [572 B] 2023-09-30T01:21:11.989 INFO:teuthology.orchestra.run.smithi107.stdout:Get:14 http://security.ubuntu.com/ubuntu focal-security/universe amd64 Packages [886 kB] 2023-09-30T01:21:11.996 INFO:teuthology.orchestra.run.smithi107.stdout:Get:15 http://security.ubuntu.com/ubuntu focal-security/universe i386 Packages [615 kB] 2023-09-30T01:21:12.003 INFO:teuthology.orchestra.run.smithi107.stdout:Get:16 http://security.ubuntu.com/ubuntu focal-security/universe Translation-en [185 kB] 2023-09-30T01:21:12.004 INFO:teuthology.orchestra.run.smithi107.stdout:Get:17 http://security.ubuntu.com/ubuntu focal-security/universe amd64 c-n-f Metadata [19.1 kB] 2023-09-30T01:21:12.007 INFO:teuthology.orchestra.run.smithi107.stdout:Get:18 http://security.ubuntu.com/ubuntu focal-security/multiverse amd64 Packages [23.6 kB] 2023-09-30T01:21:12.007 INFO:teuthology.orchestra.run.smithi107.stdout:Get:19 http://security.ubuntu.com/ubuntu focal-security/multiverse i386 Packages [7,196 B] 2023-09-30T01:21:12.008 INFO:teuthology.orchestra.run.smithi107.stdout:Get:20 http://security.ubuntu.com/ubuntu focal-security/multiverse Translation-en [5,504 B] 2023-09-30T01:21:12.008 INFO:teuthology.orchestra.run.smithi107.stdout:Get:21 http://security.ubuntu.com/ubuntu focal-security/multiverse amd64 c-n-f Metadata [548 B] 2023-09-30T01:21:12.044 INFO:teuthology.orchestra.run.smithi107.stdout:Get:22 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages [2,852 kB] 2023-09-30T01:21:12.086 INFO:teuthology.orchestra.run.smithi107.stdout:Get:23 http://archive.ubuntu.com/ubuntu focal-updates/main Translation-en [469 kB] 2023-09-30T01:21:12.089 INFO:teuthology.orchestra.run.smithi107.stdout:Get:24 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 c-n-f Metadata [17.1 kB] 2023-09-30T01:21:12.089 INFO:teuthology.orchestra.run.smithi107.stdout:Get:25 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 Packages [2,330 kB] 2023-09-30T01:21:12.110 INFO:teuthology.orchestra.run.smithi107.stdout:Get:26 http://archive.ubuntu.com/ubuntu focal-updates/restricted i386 Packages [34.7 kB] 2023-09-30T01:21:12.110 INFO:teuthology.orchestra.run.smithi107.stdout:Get:27 http://archive.ubuntu.com/ubuntu focal-updates/restricted Translation-en [325 kB] 2023-09-30T01:21:12.113 INFO:teuthology.orchestra.run.smithi107.stdout:Get:28 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 c-n-f Metadata [572 B] 2023-09-30T01:21:12.113 INFO:teuthology.orchestra.run.smithi107.stdout:Get:29 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 Packages [1,117 kB] 2023-09-30T01:21:12.113 INFO:teuthology.orchestra.run.smithi088.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages [2,852 kB] 2023-09-30T01:21:12.122 INFO:teuthology.orchestra.run.smithi107.stdout:Get:30 http://archive.ubuntu.com/ubuntu focal-updates/universe i386 Packages [748 kB] 2023-09-30T01:21:12.130 INFO:teuthology.orchestra.run.smithi107.stdout:Get:31 http://archive.ubuntu.com/ubuntu focal-updates/universe Translation-en [267 kB] 2023-09-30T01:21:12.132 INFO:teuthology.orchestra.run.smithi107.stdout:Get:32 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 c-n-f Metadata [25.6 kB] 2023-09-30T01:21:12.136 INFO:teuthology.orchestra.run.smithi107.stdout:Get:33 http://archive.ubuntu.com/ubuntu focal-updates/multiverse i386 Packages [8,436 B] 2023-09-30T01:21:12.136 INFO:teuthology.orchestra.run.smithi107.stdout:Get:34 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 Packages [25.9 kB] 2023-09-30T01:21:12.136 INFO:teuthology.orchestra.run.smithi107.stdout:Get:35 http://archive.ubuntu.com/ubuntu focal-updates/multiverse Translation-en [7,484 B] 2023-09-30T01:21:12.137 INFO:teuthology.orchestra.run.smithi107.stdout:Get:36 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 c-n-f Metadata [620 B] 2023-09-30T01:21:12.155 INFO:teuthology.orchestra.run.smithi107.stdout:Get:37 http://archive.ubuntu.com/ubuntu focal-backports/main i386 Packages [36.1 kB] 2023-09-30T01:21:12.156 INFO:teuthology.orchestra.run.smithi107.stdout:Get:38 http://archive.ubuntu.com/ubuntu focal-backports/main amd64 Packages [45.7 kB] 2023-09-30T01:21:12.156 INFO:teuthology.orchestra.run.smithi107.stdout:Get:39 http://archive.ubuntu.com/ubuntu focal-backports/main amd64 c-n-f Metadata [1,420 B] 2023-09-30T01:21:12.156 INFO:teuthology.orchestra.run.smithi107.stdout:Get:40 http://archive.ubuntu.com/ubuntu focal-backports/universe i386 Packages [13.8 kB] 2023-09-30T01:21:12.157 INFO:teuthology.orchestra.run.smithi107.stdout:Get:41 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 Packages [25.0 kB] 2023-09-30T01:21:12.157 INFO:teuthology.orchestra.run.smithi107.stdout:Get:42 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 c-n-f Metadata [880 B] 2023-09-30T01:21:12.494 INFO:teuthology.orchestra.run.smithi088.stdout:Get:7 http://security.ubuntu.com/ubuntu focal-security/main i386 Packages [652 kB] 2023-09-30T01:21:12.509 INFO:teuthology.orchestra.run.smithi088.stdout:Get:8 http://security.ubuntu.com/ubuntu focal-security/main Translation-en [385 kB] 2023-09-30T01:21:12.518 INFO:teuthology.orchestra.run.smithi088.stdout:Get:9 http://security.ubuntu.com/ubuntu focal-security/main amd64 c-n-f Metadata [13.2 kB] 2023-09-30T01:21:12.518 INFO:teuthology.orchestra.run.smithi088.stdout:Get:10 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 Packages [2,218 kB] 2023-09-30T01:21:12.576 INFO:teuthology.orchestra.run.smithi088.stdout:Get:11 http://archive.ubuntu.com/ubuntu focal-updates/main i386 Packages [885 kB] 2023-09-30T01:21:12.591 INFO:teuthology.orchestra.run.smithi088.stdout:Get:12 http://security.ubuntu.com/ubuntu focal-security/restricted i386 Packages [33.3 kB] 2023-09-30T01:21:12.592 INFO:teuthology.orchestra.run.smithi088.stdout:Get:13 http://security.ubuntu.com/ubuntu focal-security/restricted Translation-en [309 kB] 2023-09-30T01:21:12.593 INFO:teuthology.orchestra.run.smithi088.stdout:Get:14 http://archive.ubuntu.com/ubuntu focal-updates/main Translation-en [469 kB] 2023-09-30T01:21:12.599 INFO:teuthology.orchestra.run.smithi088.stdout:Get:15 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 c-n-f Metadata [572 B] 2023-09-30T01:21:12.599 INFO:teuthology.orchestra.run.smithi088.stdout:Get:16 http://security.ubuntu.com/ubuntu focal-security/universe i386 Packages [615 kB] 2023-09-30T01:21:12.602 INFO:teuthology.orchestra.run.smithi088.stdout:Get:17 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 c-n-f Metadata [17.1 kB] 2023-09-30T01:21:12.603 INFO:teuthology.orchestra.run.smithi088.stdout:Get:18 http://archive.ubuntu.com/ubuntu focal-updates/restricted i386 Packages [34.7 kB] 2023-09-30T01:21:12.603 INFO:teuthology.orchestra.run.smithi088.stdout:Get:19 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 Packages [2,330 kB] 2023-09-30T01:21:12.614 INFO:teuthology.orchestra.run.smithi088.stdout:Get:20 http://security.ubuntu.com/ubuntu focal-security/universe amd64 Packages [886 kB] 2023-09-30T01:21:12.677 INFO:teuthology.orchestra.run.smithi088.stdout:Get:21 http://archive.ubuntu.com/ubuntu focal-updates/restricted Translation-en [325 kB] 2023-09-30T01:21:12.683 INFO:teuthology.orchestra.run.smithi088.stdout:Get:22 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 c-n-f Metadata [572 B] 2023-09-30T01:21:12.797 INFO:teuthology.orchestra.run.smithi088.stdout:Get:23 http://security.ubuntu.com/ubuntu focal-security/universe Translation-en [185 kB] 2023-09-30T01:21:12.847 INFO:teuthology.orchestra.run.smithi088.stdout:Get:24 http://archive.ubuntu.com/ubuntu focal-updates/universe i386 Packages [748 kB] 2023-09-30T01:21:13.218 INFO:teuthology.orchestra.run.smithi088.stdout:Get:25 http://security.ubuntu.com/ubuntu focal-security/universe amd64 c-n-f Metadata [19.1 kB] 2023-09-30T01:21:13.224 INFO:teuthology.orchestra.run.smithi088.stdout:Get:26 http://security.ubuntu.com/ubuntu focal-security/multiverse amd64 Packages [23.6 kB] 2023-09-30T01:21:13.232 INFO:teuthology.orchestra.run.smithi088.stdout:Get:27 http://security.ubuntu.com/ubuntu focal-security/multiverse i386 Packages [7,196 B] 2023-09-30T01:21:13.235 INFO:teuthology.orchestra.run.smithi088.stdout:Get:28 http://security.ubuntu.com/ubuntu focal-security/multiverse Translation-en [5,504 B] 2023-09-30T01:21:13.237 INFO:teuthology.orchestra.run.smithi088.stdout:Get:29 http://security.ubuntu.com/ubuntu focal-security/multiverse amd64 c-n-f Metadata [548 B] 2023-09-30T01:21:13.432 INFO:teuthology.orchestra.run.smithi088.stdout:Get:30 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 Packages [1,117 kB] 2023-09-30T01:21:13.541 INFO:teuthology.orchestra.run.smithi088.stdout:Get:31 http://archive.ubuntu.com/ubuntu focal-updates/universe Translation-en [267 kB] 2023-09-30T01:21:13.561 INFO:teuthology.orchestra.run.smithi088.stdout:Get:32 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 c-n-f Metadata [25.6 kB] 2023-09-30T01:21:13.563 INFO:teuthology.orchestra.run.smithi088.stdout:Get:33 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 Packages [25.9 kB] 2023-09-30T01:21:13.565 INFO:teuthology.orchestra.run.smithi088.stdout:Get:34 http://archive.ubuntu.com/ubuntu focal-updates/multiverse i386 Packages [8,436 B] 2023-09-30T01:21:13.566 INFO:teuthology.orchestra.run.smithi088.stdout:Get:35 http://archive.ubuntu.com/ubuntu focal-updates/multiverse Translation-en [7,484 B] 2023-09-30T01:21:13.566 INFO:teuthology.orchestra.run.smithi088.stdout:Get:36 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 c-n-f Metadata [620 B] 2023-09-30T01:21:13.566 INFO:teuthology.orchestra.run.smithi088.stdout:Get:37 http://archive.ubuntu.com/ubuntu focal-backports/main i386 Packages [36.1 kB] 2023-09-30T01:21:13.569 INFO:teuthology.orchestra.run.smithi088.stdout:Get:38 http://archive.ubuntu.com/ubuntu focal-backports/main amd64 Packages [45.7 kB] 2023-09-30T01:21:13.572 INFO:teuthology.orchestra.run.smithi088.stdout:Get:39 http://archive.ubuntu.com/ubuntu focal-backports/main amd64 c-n-f Metadata [1,420 B] 2023-09-30T01:21:13.736 INFO:teuthology.orchestra.run.smithi088.stdout:Get:40 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 Packages [25.0 kB] 2023-09-30T01:21:13.919 INFO:teuthology.orchestra.run.smithi088.stdout:Get:41 http://archive.ubuntu.com/ubuntu focal-backports/universe i386 Packages [13.8 kB] 2023-09-30T01:21:13.943 INFO:teuthology.orchestra.run.smithi088.stdout:Get:42 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 c-n-f Metadata [880 B] 2023-09-30T01:21:15.818 INFO:teuthology.orchestra.run.smithi107.stdout:Fetched 17.4 MB in 3s (6,935 kB/s) 2023-09-30T01:21:16.326 INFO:teuthology.orchestra.run.smithi088.stdout:Fetched 17.4 MB in 3s (5,653 kB/s) 2023-09-30T01:21:17.182 INFO:teuthology.orchestra.run.smithi107.stdout:Reading package lists... 2023-09-30T01:21:17.206 DEBUG:teuthology.orchestra.run.smithi107:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2023-09-30T01:21:17.276 INFO:teuthology.orchestra.run.smithi107.stdout:Reading package lists... 2023-09-30T01:21:17.455 INFO:teuthology.orchestra.run.smithi107.stdout:Building dependency tree... 2023-09-30T01:21:17.456 INFO:teuthology.orchestra.run.smithi107.stdout:Reading state information... 2023-09-30T01:21:17.626 INFO:teuthology.orchestra.run.smithi107.stdout:The following additional packages will be installed: 2023-09-30T01:21:17.626 INFO:teuthology.orchestra.run.smithi107.stdout: linux-generic linux-headers-5.4.0-163 linux-headers-5.4.0-163-generic 2023-09-30T01:21:17.626 INFO:teuthology.orchestra.run.smithi107.stdout: linux-headers-generic linux-image-5.4.0-163-generic 2023-09-30T01:21:17.627 INFO:teuthology.orchestra.run.smithi107.stdout: linux-modules-5.4.0-163-generic linux-modules-extra-5.4.0-163-generic 2023-09-30T01:21:17.627 INFO:teuthology.orchestra.run.smithi107.stdout:Suggested packages: 2023-09-30T01:21:17.627 INFO:teuthology.orchestra.run.smithi107.stdout: fdutils linux-doc | linux-source-5.4.0 linux-tools 2023-09-30T01:21:17.627 INFO:teuthology.orchestra.run.smithi107.stdout:Recommended packages: 2023-09-30T01:21:17.627 INFO:teuthology.orchestra.run.smithi107.stdout: thermald 2023-09-30T01:21:17.685 INFO:teuthology.orchestra.run.smithi088.stdout:Reading package lists... 2023-09-30T01:21:17.689 INFO:teuthology.orchestra.run.smithi107.stdout:The following NEW packages will be installed: 2023-09-30T01:21:17.689 INFO:teuthology.orchestra.run.smithi107.stdout: linux-headers-5.4.0-163 linux-headers-5.4.0-163-generic 2023-09-30T01:21:17.689 INFO:teuthology.orchestra.run.smithi107.stdout: linux-image-5.4.0-163-generic linux-modules-5.4.0-163-generic 2023-09-30T01:21:17.689 INFO:teuthology.orchestra.run.smithi107.stdout: linux-modules-extra-5.4.0-163-generic 2023-09-30T01:21:17.690 INFO:teuthology.orchestra.run.smithi107.stdout:The following packages will be upgraded: 2023-09-30T01:21:17.691 INFO:teuthology.orchestra.run.smithi107.stdout: linux-generic linux-headers-generic linux-image-generic 2023-09-30T01:21:17.708 DEBUG:teuthology.orchestra.run.smithi088:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2023-09-30T01:21:17.776 INFO:teuthology.orchestra.run.smithi088.stdout:Reading package lists... 2023-09-30T01:21:17.873 INFO:teuthology.orchestra.run.smithi107.stdout:3 upgraded, 5 newly installed, 0 to remove and 263 not upgraded. 2023-09-30T01:21:17.873 INFO:teuthology.orchestra.run.smithi107.stdout:Need to get 77.1 MB of archives. 2023-09-30T01:21:17.873 INFO:teuthology.orchestra.run.smithi107.stdout:After this operation, 380 MB of additional disk space will be used. 2023-09-30T01:21:17.874 INFO:teuthology.orchestra.run.smithi107.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-modules-5.4.0-163-generic amd64 5.4.0-163.180 [15.0 MB] 2023-09-30T01:21:17.972 INFO:teuthology.orchestra.run.smithi088.stdout:Building dependency tree... 2023-09-30T01:21:17.973 INFO:teuthology.orchestra.run.smithi088.stdout:Reading state information... 2023-09-30T01:21:18.148 INFO:teuthology.orchestra.run.smithi088.stdout:The following additional packages will be installed: 2023-09-30T01:21:18.149 INFO:teuthology.orchestra.run.smithi088.stdout: linux-generic linux-headers-5.4.0-163 linux-headers-5.4.0-163-generic 2023-09-30T01:21:18.149 INFO:teuthology.orchestra.run.smithi088.stdout: linux-headers-generic linux-image-5.4.0-163-generic 2023-09-30T01:21:18.149 INFO:teuthology.orchestra.run.smithi088.stdout: linux-modules-5.4.0-163-generic linux-modules-extra-5.4.0-163-generic 2023-09-30T01:21:18.150 INFO:teuthology.orchestra.run.smithi088.stdout:Suggested packages: 2023-09-30T01:21:18.150 INFO:teuthology.orchestra.run.smithi088.stdout: fdutils linux-doc | linux-source-5.4.0 linux-tools 2023-09-30T01:21:18.150 INFO:teuthology.orchestra.run.smithi088.stdout:Recommended packages: 2023-09-30T01:21:18.150 INFO:teuthology.orchestra.run.smithi088.stdout: thermald 2023-09-30T01:21:18.203 INFO:teuthology.orchestra.run.smithi088.stdout:The following NEW packages will be installed: 2023-09-30T01:21:18.203 INFO:teuthology.orchestra.run.smithi088.stdout: linux-headers-5.4.0-163 linux-headers-5.4.0-163-generic 2023-09-30T01:21:18.203 INFO:teuthology.orchestra.run.smithi088.stdout: linux-image-5.4.0-163-generic linux-modules-5.4.0-163-generic 2023-09-30T01:21:18.204 INFO:teuthology.orchestra.run.smithi088.stdout: linux-modules-extra-5.4.0-163-generic 2023-09-30T01:21:18.204 INFO:teuthology.orchestra.run.smithi088.stdout:The following packages will be upgraded: 2023-09-30T01:21:18.204 INFO:teuthology.orchestra.run.smithi088.stdout: linux-generic linux-headers-generic linux-image-generic 2023-09-30T01:21:18.258 INFO:teuthology.orchestra.run.smithi088.stdout:3 upgraded, 5 newly installed, 0 to remove and 263 not upgraded. 2023-09-30T01:21:18.258 INFO:teuthology.orchestra.run.smithi088.stdout:Need to get 77.1 MB of archives. 2023-09-30T01:21:18.258 INFO:teuthology.orchestra.run.smithi088.stdout:After this operation, 380 MB of additional disk space will be used. 2023-09-30T01:21:18.258 INFO:teuthology.orchestra.run.smithi088.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-modules-5.4.0-163-generic amd64 5.4.0-163.180 [15.0 MB] 2023-09-30T01:21:18.595 INFO:teuthology.orchestra.run.smithi088.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-image-5.4.0-163-generic amd64 5.4.0-163.180 [10.5 MB] 2023-09-30T01:21:18.686 INFO:teuthology.orchestra.run.smithi088.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-modules-extra-5.4.0-163-generic amd64 5.4.0-163.180 [39.2 MB] 2023-09-30T01:21:18.998 INFO:teuthology.orchestra.run.smithi107.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-image-5.4.0-163-generic amd64 5.4.0-163.180 [10.5 MB] 2023-09-30T01:21:19.172 INFO:teuthology.orchestra.run.smithi088.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-generic amd64 5.4.0.163.160 [1,904 B] 2023-09-30T01:21:19.173 INFO:teuthology.orchestra.run.smithi088.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-image-generic amd64 5.4.0.163.160 [2,604 B] 2023-09-30T01:21:19.173 INFO:teuthology.orchestra.run.smithi088.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-headers-5.4.0-163 all 5.4.0-163.180 [11.0 MB] 2023-09-30T01:21:19.266 INFO:teuthology.orchestra.run.smithi107.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-modules-extra-5.4.0-163-generic amd64 5.4.0-163.180 [39.2 MB] 2023-09-30T01:21:19.268 INFO:teuthology.orchestra.run.smithi088.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-headers-5.4.0-163-generic amd64 5.4.0-163.180 [1,358 kB] 2023-09-30T01:21:19.279 INFO:teuthology.orchestra.run.smithi088.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-headers-generic amd64 5.4.0.163.160 [2,468 B] 2023-09-30T01:21:19.918 INFO:teuthology.orchestra.run.smithi088.stdout:Fetched 77.1 MB in 1s (72.2 MB/s) 2023-09-30T01:21:20.112 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package linux-modules-5.4.0-163-generic. 2023-09-30T01:21:20.512 INFO:teuthology.orchestra.run.smithi107.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-generic amd64 5.4.0.163.160 [1,904 B] 2023-09-30T01:21:20.512 INFO:teuthology.orchestra.run.smithi107.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-image-generic amd64 5.4.0.163.160 [2,604 B] 2023-09-30T01:21:20.512 INFO:teuthology.orchestra.run.smithi107.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-headers-5.4.0-163 all 5.4.0-163.180 [11.0 MB] 2023-09-30T01:21:20.800 INFO:teuthology.orchestra.run.smithi107.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-headers-5.4.0-163-generic amd64 5.4.0-163.180 [1,358 kB] 2023-09-30T01:21:20.835 INFO:teuthology.orchestra.run.smithi107.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-headers-generic amd64 5.4.0.163.160 [2,468 B] 2023-09-30T01:21:21.536 INFO:teuthology.orchestra.run.smithi107.stdout:Fetched 77.1 MB in 3s (24.6 MB/s) 2023-09-30T01:21:21.775 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package linux-modules-5.4.0-163-generic. 2023-09-30T01:21:21.847 INFO:teuthology.orchestra.run.smithi088.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 104373 files and directories currently installed.) 2023-09-30T01:21:21.852 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../0-linux-modules-5.4.0-163-generic_5.4.0-163.180_amd64.deb ... 2023-09-30T01:21:21.935 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking linux-modules-5.4.0-163-generic (5.4.0-163.180) ... 2023-09-30T01:21:23.645 INFO:teuthology.orchestra.run.smithi107.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 104373 files and directories currently installed.) 2023-09-30T01:21:23.650 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../0-linux-modules-5.4.0-163-generic_5.4.0-163.180_amd64.deb ... 2023-09-30T01:21:23.914 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking linux-modules-5.4.0-163-generic (5.4.0-163.180) ... 2023-09-30T01:21:23.994 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package linux-image-5.4.0-163-generic. 2023-09-30T01:21:24.006 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../1-linux-image-5.4.0-163-generic_5.4.0-163.180_amd64.deb ... 2023-09-30T01:21:24.104 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking linux-image-5.4.0-163-generic (5.4.0-163.180) ... 2023-09-30T01:21:25.180 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package linux-modules-extra-5.4.0-163-generic. 2023-09-30T01:21:25.196 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../2-linux-modules-extra-5.4.0-163-generic_5.4.0-163.180_amd64.deb ... 2023-09-30T01:21:25.238 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking linux-modules-extra-5.4.0-163-generic (5.4.0-163.180) ... 2023-09-30T01:21:25.866 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package linux-image-5.4.0-163-generic. 2023-09-30T01:21:25.879 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../1-linux-image-5.4.0-163-generic_5.4.0-163.180_amd64.deb ... 2023-09-30T01:21:25.988 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking linux-image-5.4.0-163-generic (5.4.0-163.180) ... 2023-09-30T01:21:27.110 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package linux-modules-extra-5.4.0-163-generic. 2023-09-30T01:21:27.126 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../2-linux-modules-extra-5.4.0-163-generic_5.4.0-163.180_amd64.deb ... 2023-09-30T01:21:27.169 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking linux-modules-extra-5.4.0-163-generic (5.4.0-163.180) ... 2023-09-30T01:21:29.809 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../3-linux-generic_5.4.0.163.160_amd64.deb ... 2023-09-30T01:21:29.935 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking linux-generic (5.4.0.163.160) over (5.4.0.124.125) ... 2023-09-30T01:21:30.289 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../4-linux-image-generic_5.4.0.163.160_amd64.deb ... 2023-09-30T01:21:30.414 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking linux-image-generic (5.4.0.163.160) over (5.4.0.124.125) ... 2023-09-30T01:21:30.758 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package linux-headers-5.4.0-163. 2023-09-30T01:21:30.775 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../5-linux-headers-5.4.0-163_5.4.0-163.180_all.deb ... 2023-09-30T01:21:30.816 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking linux-headers-5.4.0-163 (5.4.0-163.180) ... 2023-09-30T01:21:31.943 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../3-linux-generic_5.4.0.163.160_amd64.deb ... 2023-09-30T01:21:32.096 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking linux-generic (5.4.0.163.160) over (5.4.0.124.125) ... 2023-09-30T01:21:32.414 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../4-linux-image-generic_5.4.0.163.160_amd64.deb ... 2023-09-30T01:21:32.540 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking linux-image-generic (5.4.0.163.160) over (5.4.0.124.125) ... 2023-09-30T01:21:32.902 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package linux-headers-5.4.0-163. 2023-09-30T01:21:32.916 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../5-linux-headers-5.4.0-163_5.4.0-163.180_all.deb ... 2023-09-30T01:21:32.959 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking linux-headers-5.4.0-163 (5.4.0-163.180) ... 2023-09-30T01:21:36.056 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package linux-headers-5.4.0-163-generic. 2023-09-30T01:21:36.079 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../6-linux-headers-5.4.0-163-generic_5.4.0-163.180_amd64.deb ... 2023-09-30T01:21:36.123 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking linux-headers-5.4.0-163-generic (5.4.0-163.180) ... 2023-09-30T01:21:38.177 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package linux-headers-5.4.0-163-generic. 2023-09-30T01:21:38.200 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../6-linux-headers-5.4.0-163-generic_5.4.0-163.180_amd64.deb ... 2023-09-30T01:21:38.243 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking linux-headers-5.4.0-163-generic (5.4.0-163.180) ... 2023-09-30T01:21:39.738 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../7-linux-headers-generic_5.4.0.163.160_amd64.deb ... 2023-09-30T01:21:39.866 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking linux-headers-generic (5.4.0.163.160) over (5.4.0.124.125) ... 2023-09-30T01:21:40.225 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up linux-modules-5.4.0-163-generic (5.4.0-163.180) ... 2023-09-30T01:21:41.015 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../7-linux-headers-generic_5.4.0.163.160_amd64.deb ... 2023-09-30T01:21:41.177 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking linux-headers-generic (5.4.0.163.160) over (5.4.0.124.125) ... 2023-09-30T01:21:41.431 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up linux-headers-5.4.0-163 (5.4.0-163.180) ... 2023-09-30T01:21:41.499 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up linux-modules-5.4.0-163-generic (5.4.0-163.180) ... 2023-09-30T01:21:41.557 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up linux-image-5.4.0-163-generic (5.4.0-163.180) ... 2023-09-30T01:21:42.710 INFO:teuthology.orchestra.run.smithi088.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.4.0-163-generic 2023-09-30T01:21:42.710 INFO:teuthology.orchestra.run.smithi088.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.4.0-163-generic 2023-09-30T01:21:42.766 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up linux-headers-5.4.0-163 (5.4.0-163.180) ... 2023-09-30T01:21:42.851 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up linux-headers-5.4.0-163-generic (5.4.0-163.180) ... 2023-09-30T01:21:42.909 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up linux-image-5.4.0-163-generic (5.4.0-163.180) ... 2023-09-30T01:21:42.977 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up linux-modules-extra-5.4.0-163-generic (5.4.0-163.180) ... 2023-09-30T01:21:44.055 INFO:teuthology.orchestra.run.smithi107.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.4.0-163-generic 2023-09-30T01:21:44.055 INFO:teuthology.orchestra.run.smithi107.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.4.0-163-generic 2023-09-30T01:21:44.194 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up linux-headers-5.4.0-163-generic (5.4.0-163.180) ... 2023-09-30T01:21:44.299 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up linux-image-generic (5.4.0.163.160) ... 2023-09-30T01:21:44.312 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up linux-modules-extra-5.4.0-163-generic (5.4.0-163.180) ... 2023-09-30T01:21:45.213 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up linux-headers-generic (5.4.0.163.160) ... 2023-09-30T01:21:45.339 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up linux-generic (5.4.0.163.160) ... 2023-09-30T01:21:45.466 INFO:teuthology.orchestra.run.smithi088.stdout:Processing triggers for linux-image-5.4.0-163-generic (5.4.0-163.180) ... 2023-09-30T01:21:45.480 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up linux-image-generic (5.4.0.163.160) ... 2023-09-30T01:21:45.539 INFO:teuthology.orchestra.run.smithi088.stdout:/etc/kernel/postinst.d/initramfs-tools: 2023-09-30T01:21:45.539 INFO:teuthology.orchestra.run.smithi088.stdout:update-initramfs: Generating /boot/initrd.img-5.4.0-163-generic 2023-09-30T01:21:45.639 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up linux-headers-generic (5.4.0.163.160) ... 2023-09-30T01:21:45.765 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up linux-generic (5.4.0.163.160) ... 2023-09-30T01:21:45.892 INFO:teuthology.orchestra.run.smithi107.stdout:Processing triggers for linux-image-5.4.0-163-generic (5.4.0-163.180) ... 2023-09-30T01:21:45.972 INFO:teuthology.orchestra.run.smithi107.stdout:/etc/kernel/postinst.d/initramfs-tools: 2023-09-30T01:21:45.972 INFO:teuthology.orchestra.run.smithi107.stdout:update-initramfs: Generating /boot/initrd.img-5.4.0-163-generic 2023-09-30T01:22:07.830 INFO:teuthology.orchestra.run.smithi088.stdout:/etc/kernel/postinst.d/zz-update-grub: 2023-09-30T01:22:07.830 INFO:teuthology.orchestra.run.smithi088.stdout:Sourcing file `/etc/default/grub' 2023-09-30T01:22:07.838 INFO:teuthology.orchestra.run.smithi088.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2023-09-30T01:22:07.849 INFO:teuthology.orchestra.run.smithi088.stdout:Generating grub configuration file ... 2023-09-30T01:22:07.949 INFO:teuthology.orchestra.run.smithi107.stdout:/etc/kernel/postinst.d/zz-update-grub: 2023-09-30T01:22:07.949 INFO:teuthology.orchestra.run.smithi107.stdout:Sourcing file `/etc/default/grub' 2023-09-30T01:22:07.967 INFO:teuthology.orchestra.run.smithi107.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2023-09-30T01:22:07.975 INFO:teuthology.orchestra.run.smithi107.stdout:Generating grub configuration file ... 2023-09-30T01:22:08.196 INFO:teuthology.orchestra.run.smithi088.stdout:Found linux image: /boot/vmlinuz-5.4.0-163-generic 2023-09-30T01:22:08.215 INFO:teuthology.orchestra.run.smithi088.stdout:Found initrd image: /boot/initrd.img-5.4.0-163-generic 2023-09-30T01:22:08.304 INFO:teuthology.orchestra.run.smithi107.stdout:Found linux image: /boot/vmlinuz-5.4.0-163-generic 2023-09-30T01:22:08.323 INFO:teuthology.orchestra.run.smithi107.stdout:Found initrd image: /boot/initrd.img-5.4.0-163-generic 2023-09-30T01:22:08.636 INFO:teuthology.orchestra.run.smithi088.stdout:Found linux image: /boot/vmlinuz-5.4.0-124-generic 2023-09-30T01:22:08.652 INFO:teuthology.orchestra.run.smithi088.stdout:Found initrd image: /boot/initrd.img-5.4.0-124-generic 2023-09-30T01:22:08.726 INFO:teuthology.orchestra.run.smithi107.stdout:Found linux image: /boot/vmlinuz-5.4.0-124-generic 2023-09-30T01:22:08.738 INFO:teuthology.orchestra.run.smithi107.stdout:Found initrd image: /boot/initrd.img-5.4.0-124-generic 2023-09-30T01:22:11.325 INFO:teuthology.orchestra.run.smithi088.stdout:done 2023-09-30T01:22:11.436 INFO:teuthology.orchestra.run.smithi107.stdout:done 2023-09-30T01:22:11.711 DEBUG:teuthology.orchestra.run.smithi088:> dpkg -s linux-image-generic 2023-09-30T01:22:11.758 INFO:teuthology.orchestra.run.smithi088.stdout:Package: linux-image-generic 2023-09-30T01:22:11.758 INFO:teuthology.orchestra.run.smithi088.stdout:Status: install ok installed 2023-09-30T01:22:11.758 INFO:teuthology.orchestra.run.smithi088.stdout:Priority: optional 2023-09-30T01:22:11.758 INFO:teuthology.orchestra.run.smithi088.stdout:Section: kernel 2023-09-30T01:22:11.758 INFO:teuthology.orchestra.run.smithi088.stdout:Installed-Size: 20 2023-09-30T01:22:11.759 INFO:teuthology.orchestra.run.smithi088.stdout:Maintainer: Ubuntu Kernel Team 2023-09-30T01:22:11.759 INFO:teuthology.orchestra.run.smithi088.stdout:Architecture: amd64 2023-09-30T01:22:11.759 INFO:teuthology.orchestra.run.smithi088.stdout:Source: linux-meta 2023-09-30T01:22:11.759 INFO:teuthology.orchestra.run.smithi088.stdout:Version: 5.4.0.163.160 2023-09-30T01:22:11.759 INFO:teuthology.orchestra.run.smithi088.stdout:Provides: virtualbox-guest-modules (= 6.1.38-dfsg-3~ubuntu1.20.04.1), wireguard-modules (= 1.0.20201112-1~20.04.1), zfs-modules (= 0.8.3-1ubuntu12.15) 2023-09-30T01:22:11.759 INFO:teuthology.orchestra.run.smithi088.stdout:Depends: linux-image-5.4.0-163-generic, linux-modules-extra-5.4.0-163-generic, linux-firmware, intel-microcode, amd64-microcode 2023-09-30T01:22:11.759 INFO:teuthology.orchestra.run.smithi088.stdout:Recommends: thermald 2023-09-30T01:22:11.759 INFO:teuthology.orchestra.run.smithi088.stdout:Description: Generic Linux kernel image 2023-09-30T01:22:11.759 INFO:teuthology.orchestra.run.smithi088.stdout: This package will always depend on the latest generic kernel image 2023-09-30T01:22:11.759 INFO:teuthology.orchestra.run.smithi088.stdout: available. 2023-09-30T01:22:11.760 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.4.0-124-generic Expected: 5.4.0-163-generic 2023-09-30T01:22:11.760 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-09-30T01:22:11.760 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-09-30T01:22:11.760 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2023-09-30T01:22:11.760 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi088.front.sepia.ceph.com, path=None, version=distro) 2023-09-30T01:22:11.760 DEBUG:teuthology.orchestra.run.smithi088:> sudo apt-get clean 2023-09-30T01:22:11.851 DEBUG:teuthology.orchestra.run.smithi088:> sudo apt-get update 2023-09-30T01:22:11.858 DEBUG:teuthology.orchestra.run.smithi107:> dpkg -s linux-image-generic 2023-09-30T01:22:11.896 INFO:teuthology.orchestra.run.smithi107.stdout:Package: linux-image-generic 2023-09-30T01:22:11.897 INFO:teuthology.orchestra.run.smithi107.stdout:Status: install ok installed 2023-09-30T01:22:11.897 INFO:teuthology.orchestra.run.smithi107.stdout:Priority: optional 2023-09-30T01:22:11.897 INFO:teuthology.orchestra.run.smithi107.stdout:Section: kernel 2023-09-30T01:22:11.897 INFO:teuthology.orchestra.run.smithi107.stdout:Installed-Size: 20 2023-09-30T01:22:11.897 INFO:teuthology.orchestra.run.smithi107.stdout:Maintainer: Ubuntu Kernel Team 2023-09-30T01:22:11.897 INFO:teuthology.orchestra.run.smithi107.stdout:Architecture: amd64 2023-09-30T01:22:11.897 INFO:teuthology.orchestra.run.smithi107.stdout:Source: linux-meta 2023-09-30T01:22:11.897 INFO:teuthology.orchestra.run.smithi107.stdout:Version: 5.4.0.163.160 2023-09-30T01:22:11.897 INFO:teuthology.orchestra.run.smithi107.stdout:Provides: virtualbox-guest-modules (= 6.1.38-dfsg-3~ubuntu1.20.04.1), wireguard-modules (= 1.0.20201112-1~20.04.1), zfs-modules (= 0.8.3-1ubuntu12.15) 2023-09-30T01:22:11.897 INFO:teuthology.orchestra.run.smithi107.stdout:Depends: linux-image-5.4.0-163-generic, linux-modules-extra-5.4.0-163-generic, linux-firmware, intel-microcode, amd64-microcode 2023-09-30T01:22:11.897 INFO:teuthology.orchestra.run.smithi107.stdout:Recommends: thermald 2023-09-30T01:22:11.898 INFO:teuthology.orchestra.run.smithi107.stdout:Description: Generic Linux kernel image 2023-09-30T01:22:11.898 INFO:teuthology.orchestra.run.smithi107.stdout: This package will always depend on the latest generic kernel image 2023-09-30T01:22:11.898 INFO:teuthology.orchestra.run.smithi107.stdout: available. 2023-09-30T01:22:11.898 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.4.0-124-generic Expected: 5.4.0-163-generic 2023-09-30T01:22:11.899 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-09-30T01:22:11.899 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-09-30T01:22:11.899 INFO:teuthology.task.kernel:Installing distro kernel on mon.f... 2023-09-30T01:22:11.899 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi107.front.sepia.ceph.com, path=None, version=distro) 2023-09-30T01:22:11.899 DEBUG:teuthology.orchestra.run.smithi107:> sudo apt-get clean 2023-09-30T01:22:11.999 DEBUG:teuthology.orchestra.run.smithi107:> sudo apt-get update 2023-09-30T01:22:12.129 INFO:teuthology.orchestra.run.smithi107.stdout:Hit:1 http://archive.ubuntu.com/ubuntu focal InRelease 2023-09-30T01:22:12.129 INFO:teuthology.orchestra.run.smithi107.stdout:Hit:2 http://security.ubuntu.com/ubuntu focal-security InRelease 2023-09-30T01:22:12.136 INFO:teuthology.orchestra.run.smithi107.stdout:Hit:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2023-09-30T01:22:12.153 INFO:teuthology.orchestra.run.smithi107.stdout:Hit:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2023-09-30T01:22:12.199 INFO:teuthology.orchestra.run.smithi088.stdout:Hit:1 http://archive.ubuntu.com/ubuntu focal InRelease 2023-09-30T01:22:12.199 INFO:teuthology.orchestra.run.smithi088.stdout:Hit:2 http://security.ubuntu.com/ubuntu focal-security InRelease 2023-09-30T01:22:12.269 INFO:teuthology.orchestra.run.smithi088.stdout:Hit:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2023-09-30T01:22:12.358 INFO:teuthology.orchestra.run.smithi088.stdout:Hit:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2023-09-30T01:22:13.835 INFO:teuthology.orchestra.run.smithi107.stdout:Reading package lists... 2023-09-30T01:22:13.860 DEBUG:teuthology.orchestra.run.smithi107:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2023-09-30T01:22:13.930 INFO:teuthology.orchestra.run.smithi107.stdout:Reading package lists... 2023-09-30T01:22:13.944 INFO:teuthology.orchestra.run.smithi088.stdout:Reading package lists... 2023-09-30T01:22:13.969 DEBUG:teuthology.orchestra.run.smithi088:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2023-09-30T01:22:14.037 INFO:teuthology.orchestra.run.smithi088.stdout:Reading package lists... 2023-09-30T01:22:14.108 INFO:teuthology.orchestra.run.smithi107.stdout:Building dependency tree... 2023-09-30T01:22:14.108 INFO:teuthology.orchestra.run.smithi107.stdout:Reading state information... 2023-09-30T01:22:14.237 INFO:teuthology.orchestra.run.smithi088.stdout:Building dependency tree... 2023-09-30T01:22:14.238 INFO:teuthology.orchestra.run.smithi088.stdout:Reading state information... 2023-09-30T01:22:14.314 INFO:teuthology.orchestra.run.smithi107.stdout:linux-image-generic is already the newest version (5.4.0.163.160). 2023-09-30T01:22:14.314 INFO:teuthology.orchestra.run.smithi107.stdout:0 upgraded, 0 newly installed, 0 to remove and 263 not upgraded. 2023-09-30T01:22:14.316 DEBUG:teuthology.orchestra.run.smithi107:> dpkg -s linux-image-generic 2023-09-30T01:22:14.334 INFO:teuthology.orchestra.run.smithi107.stdout:Package: linux-image-generic 2023-09-30T01:22:14.334 INFO:teuthology.orchestra.run.smithi107.stdout:Status: install ok installed 2023-09-30T01:22:14.334 INFO:teuthology.orchestra.run.smithi107.stdout:Priority: optional 2023-09-30T01:22:14.334 INFO:teuthology.orchestra.run.smithi107.stdout:Section: kernel 2023-09-30T01:22:14.334 INFO:teuthology.orchestra.run.smithi107.stdout:Installed-Size: 20 2023-09-30T01:22:14.335 INFO:teuthology.orchestra.run.smithi107.stdout:Maintainer: Ubuntu Kernel Team 2023-09-30T01:22:14.335 INFO:teuthology.orchestra.run.smithi107.stdout:Architecture: amd64 2023-09-30T01:22:14.335 INFO:teuthology.orchestra.run.smithi107.stdout:Source: linux-meta 2023-09-30T01:22:14.335 INFO:teuthology.orchestra.run.smithi107.stdout:Version: 5.4.0.163.160 2023-09-30T01:22:14.335 INFO:teuthology.orchestra.run.smithi107.stdout:Provides: virtualbox-guest-modules (= 6.1.38-dfsg-3~ubuntu1.20.04.1), wireguard-modules (= 1.0.20201112-1~20.04.1), zfs-modules (= 0.8.3-1ubuntu12.15) 2023-09-30T01:22:14.335 INFO:teuthology.orchestra.run.smithi107.stdout:Depends: linux-image-5.4.0-163-generic, linux-modules-extra-5.4.0-163-generic, linux-firmware, intel-microcode, amd64-microcode 2023-09-30T01:22:14.335 INFO:teuthology.orchestra.run.smithi107.stdout:Recommends: thermald 2023-09-30T01:22:14.335 INFO:teuthology.orchestra.run.smithi107.stdout:Description: Generic Linux kernel image 2023-09-30T01:22:14.335 INFO:teuthology.orchestra.run.smithi107.stdout: This package will always depend on the latest generic kernel image 2023-09-30T01:22:14.335 INFO:teuthology.orchestra.run.smithi107.stdout: available. 2023-09-30T01:22:14.336 DEBUG:teuthology.orchestra.run.smithi107:> mktemp 2023-09-30T01:22:14.340 INFO:teuthology.orchestra.run.smithi107.stdout:/tmp/tmp.AvgMUJSnYc 2023-09-30T01:22:14.340 DEBUG:teuthology.orchestra.run.smithi107:> sudo cp /boot/grub/grub.cfg /tmp/tmp.AvgMUJSnYc 2023-09-30T01:22:14.412 DEBUG:teuthology.orchestra.run.smithi107:> sudo chmod 0666 /tmp/tmp.AvgMUJSnYc 2023-09-30T01:22:14.455 INFO:teuthology.orchestra.run.smithi088.stdout:linux-image-generic is already the newest version (5.4.0.163.160). 2023-09-30T01:22:14.456 INFO:teuthology.orchestra.run.smithi088.stdout:0 upgraded, 0 newly installed, 0 to remove and 263 not upgraded. 2023-09-30T01:22:14.457 DEBUG:teuthology.orchestra.run.smithi088:> dpkg -s linux-image-generic 2023-09-30T01:22:14.461 DEBUG:teuthology.orchestra.remote:smithi107:/tmp/tmp.AvgMUJSnYc is 9KB 2023-09-30T01:22:14.478 DEBUG:teuthology.orchestra.run.smithi107:> rm -fr /tmp/tmp.AvgMUJSnYc 2023-09-30T01:22:14.480 INFO:teuthology.orchestra.run.smithi088.stdout:Package: linux-image-generic 2023-09-30T01:22:14.480 INFO:teuthology.orchestra.run.smithi088.stdout:Status: install ok installed 2023-09-30T01:22:14.480 INFO:teuthology.orchestra.run.smithi088.stdout:Priority: optional 2023-09-30T01:22:14.480 INFO:teuthology.orchestra.run.smithi088.stdout:Section: kernel 2023-09-30T01:22:14.481 INFO:teuthology.orchestra.run.smithi088.stdout:Installed-Size: 20 2023-09-30T01:22:14.481 INFO:teuthology.orchestra.run.smithi088.stdout:Maintainer: Ubuntu Kernel Team 2023-09-30T01:22:14.481 INFO:teuthology.orchestra.run.smithi088.stdout:Architecture: amd64 2023-09-30T01:22:14.481 INFO:teuthology.orchestra.run.smithi088.stdout:Source: linux-meta 2023-09-30T01:22:14.481 INFO:teuthology.orchestra.run.smithi088.stdout:Version: 5.4.0.163.160 2023-09-30T01:22:14.481 INFO:teuthology.orchestra.run.smithi088.stdout:Provides: virtualbox-guest-modules (= 6.1.38-dfsg-3~ubuntu1.20.04.1), wireguard-modules (= 1.0.20201112-1~20.04.1), zfs-modules (= 0.8.3-1ubuntu12.15) 2023-09-30T01:22:14.481 INFO:teuthology.orchestra.run.smithi088.stdout:Depends: linux-image-5.4.0-163-generic, linux-modules-extra-5.4.0-163-generic, linux-firmware, intel-microcode, amd64-microcode 2023-09-30T01:22:14.482 INFO:teuthology.orchestra.run.smithi088.stdout:Recommends: thermald 2023-09-30T01:22:14.482 INFO:teuthology.orchestra.run.smithi088.stdout:Description: Generic Linux kernel image 2023-09-30T01:22:14.482 INFO:teuthology.orchestra.run.smithi088.stdout: This package will always depend on the latest generic kernel image 2023-09-30T01:22:14.482 INFO:teuthology.orchestra.run.smithi088.stdout: available. 2023-09-30T01:22:14.482 DEBUG:teuthology.orchestra.run.smithi088:> mktemp 2023-09-30T01:22:14.488 DEBUG:teuthology.orchestra.run.smithi107:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2023-09-30T01:22:14.530 INFO:teuthology.orchestra.run.smithi088.stdout:/tmp/tmp.ni6QrVU9Eh 2023-09-30T01:22:14.530 DEBUG:teuthology.orchestra.run.smithi088:> sudo cp /boot/grub/grub.cfg /tmp/tmp.ni6QrVU9Eh 2023-09-30T01:22:14.544 DEBUG:teuthology.orchestra.run.smithi088:> sudo chmod 0666 /tmp/tmp.ni6QrVU9Eh 2023-09-30T01:22:14.546 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-09-30T01:22:14.546 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd of=/etc/grub.d/01_ceph_kernel 2023-09-30T01:22:14.546 DEBUG:teuthology.orchestra.run.smithi107:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2023-09-30T01:22:14.585 INFO:teuthology.task.kernel:Distro Kernel Version: 5.4.0-163-generic 2023-09-30T01:22:14.585 DEBUG:teuthology.orchestra.run.smithi107:> sudo update-grub 2023-09-30T01:22:14.630 DEBUG:teuthology.orchestra.remote:smithi088:/tmp/tmp.ni6QrVU9Eh is 9KB 2023-09-30T01:22:14.644 DEBUG:teuthology.orchestra.run.smithi088:> rm -fr /tmp/tmp.ni6QrVU9Eh 2023-09-30T01:22:14.650 DEBUG:teuthology.orchestra.run.smithi088:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2023-09-30T01:22:14.706 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2023-09-30T01:22:14.706 DEBUG:teuthology.orchestra.run.smithi088:> sudo dd of=/etc/grub.d/01_ceph_kernel 2023-09-30T01:22:14.707 DEBUG:teuthology.orchestra.run.smithi088:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2023-09-30T01:22:14.752 INFO:teuthology.task.kernel:Distro Kernel Version: 5.4.0-163-generic 2023-09-30T01:22:14.752 DEBUG:teuthology.orchestra.run.smithi088:> sudo update-grub 2023-09-30T01:22:15.889 INFO:teuthology.orchestra.run.smithi107.stderr:Sourcing file `/etc/default/grub' 2023-09-30T01:22:15.898 INFO:teuthology.orchestra.run.smithi107.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2023-09-30T01:22:15.910 INFO:teuthology.orchestra.run.smithi107.stderr:Generating grub configuration file ... 2023-09-30T01:22:16.063 INFO:teuthology.orchestra.run.smithi088.stderr:Sourcing file `/etc/default/grub' 2023-09-30T01:22:16.068 INFO:teuthology.orchestra.run.smithi088.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2023-09-30T01:22:16.080 INFO:teuthology.orchestra.run.smithi088.stderr:Generating grub configuration file ... 2023-09-30T01:22:16.161 INFO:teuthology.orchestra.run.smithi107.stderr:Found linux image: /boot/vmlinuz-5.4.0-163-generic 2023-09-30T01:22:16.173 INFO:teuthology.orchestra.run.smithi107.stderr:Found initrd image: /boot/initrd.img-5.4.0-163-generic 2023-09-30T01:22:16.362 INFO:teuthology.orchestra.run.smithi088.stderr:Found linux image: /boot/vmlinuz-5.4.0-163-generic 2023-09-30T01:22:16.373 INFO:teuthology.orchestra.run.smithi088.stderr:Found initrd image: /boot/initrd.img-5.4.0-163-generic 2023-09-30T01:22:16.555 INFO:teuthology.orchestra.run.smithi107.stderr:Found linux image: /boot/vmlinuz-5.4.0-124-generic 2023-09-30T01:22:16.566 INFO:teuthology.orchestra.run.smithi107.stderr:Found initrd image: /boot/initrd.img-5.4.0-124-generic 2023-09-30T01:22:16.817 INFO:teuthology.orchestra.run.smithi088.stderr:Found linux image: /boot/vmlinuz-5.4.0-124-generic 2023-09-30T01:22:16.828 INFO:teuthology.orchestra.run.smithi088.stderr:Found initrd image: /boot/initrd.img-5.4.0-124-generic 2023-09-30T01:22:18.212 INFO:teuthology.orchestra.run.smithi107.stderr:done 2023-09-30T01:22:18.214 DEBUG:teuthology.orchestra.run.smithi107:> sudo shutdown -r now 2023-09-30T01:22:18.490 INFO:teuthology.orchestra.run.smithi088.stderr:done 2023-09-30T01:22:18.492 DEBUG:teuthology.orchestra.run.smithi088:> sudo shutdown -r now 2023-09-30T01:22:48.246 INFO:teuthology.task.kernel:Checking client mon.f for new kernel version... 2023-09-30T01:22:48.247 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi107.front.sepia.ceph.com' 2023-09-30T01:22:48.247 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi107.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-09-30T01:22:48.495 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2023-09-30T01:22:48.495 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi088.front.sepia.ceph.com' 2023-09-30T01:22:48.496 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi088.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-09-30T01:23:22.874 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.107 2023-09-30T01:23:23.130 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.88 2023-09-30T01:23:31.918 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi107.front.sepia.ceph.com' 2023-09-30T01:23:31.919 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi107.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-09-30T01:23:32.131 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi088.front.sepia.ceph.com' 2023-09-30T01:23:32.132 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi088.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-09-30T01:23:35.194 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.88 2023-09-30T01:23:47.206 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi088.front.sepia.ceph.com' 2023-09-30T01:23:47.207 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi088.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-09-30T01:23:50.266 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.107 2023-09-30T01:24:02.278 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi107.front.sepia.ceph.com' 2023-09-30T01:24:02.279 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi107.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-09-30T01:24:02.650 DEBUG:teuthology.orchestra.run.smithi107:> true 2023-09-30T01:24:03.515 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi107.front.sepia.ceph.com' 2023-09-30T01:24:03.515 INFO:teuthology.task.kernel:Checking kernel version of mon.f, want "5.4.0-163-generic"... 2023-09-30T01:24:03.515 DEBUG:teuthology.orchestra.run.smithi107:> uname -r 2023-09-30T01:24:03.567 INFO:teuthology.orchestra.run.smithi107.stdout:5.4.0-163-generic 2023-09-30T01:24:03.567 DEBUG:teuthology.task.kernel:current kernel version is 5.4.0-163-generic vs 5.4.0-163-generic 2023-09-30T01:24:03.567 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-09-30T01:24:03.567 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2023-09-30T01:24:04.569 INFO:teuthology.task.kernel:Enabling kdb on mon.f... 2023-09-30T01:24:04.570 DEBUG:teuthology.orchestra.run.smithi107:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-09-30T01:24:04.804 INFO:teuthology.orchestra.run.smithi107.stdout:ttyS1 2023-09-30T01:24:04.831 DEBUG:teuthology.parallel:result is None 2023-09-30T01:24:18.895 DEBUG:teuthology.orchestra.run.smithi088:> true 2023-09-30T01:24:19.451 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi088.front.sepia.ceph.com' 2023-09-30T01:24:19.452 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.4.0-163-generic"... 2023-09-30T01:24:19.452 DEBUG:teuthology.orchestra.run.smithi088:> uname -r 2023-09-30T01:24:19.458 INFO:teuthology.orchestra.run.smithi088.stdout:5.4.0-163-generic 2023-09-30T01:24:19.458 DEBUG:teuthology.task.kernel:current kernel version is 5.4.0-163-generic vs 5.4.0-163-generic 2023-09-30T01:24:19.458 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-09-30T01:24:19.458 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2023-09-30T01:24:20.459 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2023-09-30T01:24:20.459 DEBUG:teuthology.orchestra.run.smithi088:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-09-30T01:24:20.570 INFO:teuthology.orchestra.run.smithi088.stdout:ttyS1 2023-09-30T01:24:20.596 DEBUG:teuthology.parallel:result is None 2023-09-30T01:24:20.596 INFO:teuthology.run_tasks:Running task internal.base... 2023-09-30T01:24:20.604 INFO:teuthology.task.internal:Creating test directory... 2023-09-30T01:24:20.604 DEBUG:teuthology.orchestra.run.smithi088:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-09-30T01:24:20.607 DEBUG:teuthology.orchestra.run.smithi107:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-09-30T01:24:20.614 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2023-09-30T01:24:20.641 INFO:teuthology.run_tasks:Running task internal.archive... 2023-09-30T01:24:20.674 INFO:teuthology.task.internal:Creating archive directory... 2023-09-30T01:24:20.674 DEBUG:teuthology.orchestra.run.smithi088:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-09-30T01:24:20.676 DEBUG:teuthology.orchestra.run.smithi107:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-09-30T01:24:20.695 INFO:teuthology.run_tasks:Running task internal.coredump... 2023-09-30T01:24:20.707 INFO:teuthology.task.internal:Enabling coredump saving... 2023-09-30T01:24:20.708 DEBUG:teuthology.orchestra.run.smithi088:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-09-30T01:24:20.731 DEBUG:teuthology.orchestra.run.smithi107:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-09-30T01:24:20.746 INFO:teuthology.orchestra.run.smithi088.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-09-30T01:24:20.757 INFO:teuthology.orchestra.run.smithi088.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-09-30T01:24:20.774 INFO:teuthology.orchestra.run.smithi107.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-09-30T01:24:20.785 INFO:teuthology.orchestra.run.smithi107.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-09-30T01:24:20.787 INFO:teuthology.run_tasks:Running task internal.sudo... 2023-09-30T01:24:20.799 INFO:teuthology.task.internal:Configuring sudo... 2023-09-30T01:24:20.799 DEBUG:teuthology.orchestra.run.smithi088:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-09-30T01:24:20.802 DEBUG:teuthology.orchestra.run.smithi107:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-09-30T01:24:20.846 INFO:teuthology.run_tasks:Running task internal.syslog... 2023-09-30T01:24:20.855 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2023-09-30T01:24:20.855 DEBUG:teuthology.orchestra.run.smithi088:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-09-30T01:24:20.859 DEBUG:teuthology.orchestra.run.smithi107:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-09-30T01:24:20.866 DEBUG:teuthology.orchestra.run.smithi088:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-09-30T01:24:20.872 DEBUG:teuthology.orchestra.run.smithi088:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-09-30T01:24:20.879 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2023-09-30T01:24:20.880 DEBUG:teuthology.orchestra.run.smithi088:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-09-30T01:24:20.955 DEBUG:teuthology.orchestra.run.smithi107:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-09-30T01:24:20.963 DEBUG:teuthology.orchestra.run.smithi107:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-09-30T01:24:21.011 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-09-30T01:24:21.011 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-09-30T01:24:21.032 DEBUG:teuthology.orchestra.run.smithi088:> sudo service rsyslog restart 2023-09-30T01:24:21.034 DEBUG:teuthology.orchestra.run.smithi107:> sudo service rsyslog restart 2023-09-30T01:24:21.094 INFO:teuthology.run_tasks:Running task internal.timer... 2023-09-30T01:24:21.104 INFO:teuthology.task.internal:Starting timer... 2023-09-30T01:24:21.104 INFO:teuthology.run_tasks:Running task pcp... 2023-09-30T01:24:21.127 INFO:teuthology.run_tasks:Running task selinux... 2023-09-30T01:24:21.161 DEBUG:teuthology.task.selinux:Excluding smithi088: OS 'ubuntu' does not support SELinux 2023-09-30T01:24:21.161 DEBUG:teuthology.task.selinux:Excluding smithi107: OS 'ubuntu' does not support SELinux 2023-09-30T01:24:21.161 DEBUG:teuthology.task.selinux:Getting current SELinux state 2023-09-30T01:24:21.161 DEBUG:teuthology.task.selinux:Existing SELinux modes: {} 2023-09-30T01:24:21.162 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2023-09-30T01:24:21.162 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2023-09-30T01:24:21.194 INFO:teuthology.repo_utils:/home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main was just updated or references a specific commit; assuming it is current 2023-09-30T01:24:21.194 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2023-09-30T01:24:21.214 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2023-09-30T01:24:21.215 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi088.front.sepia.ceph.com,smithi107.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2023-09-30T01:30:05.673 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi088.front.sepia.ceph.com'), Remote(name='ubuntu@smithi107.front.sepia.ceph.com')] 2023-09-30T01:30:05.674 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi088.front.sepia.ceph.com' 2023-09-30T01:30:05.675 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi088.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-09-30T01:30:05.725 DEBUG:teuthology.orchestra.run.smithi088:> true 2023-09-30T01:30:05.848 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi088.front.sepia.ceph.com' 2023-09-30T01:30:05.848 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi107.front.sepia.ceph.com' 2023-09-30T01:30:05.849 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi107.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-09-30T01:30:05.898 DEBUG:teuthology.orchestra.run.smithi107:> true 2023-09-30T01:30:06.015 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi107.front.sepia.ceph.com' 2023-09-30T01:30:06.016 INFO:teuthology.run_tasks:Running task clock... 2023-09-30T01:30:06.028 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2023-09-30T01:30:06.028 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-09-30T01:30:06.028 DEBUG:teuthology.orchestra.run.smithi088:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-09-30T01:30:06.031 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-09-30T01:30:06.031 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-09-30T01:30:06.063 INFO:teuthology.orchestra.run.smithi088.stdout:30 Sep 01:30:06 ntpd[22733]: ntpd 4.2.8p12@1.3728-o (1): Starting 2023-09-30T01:30:06.063 INFO:teuthology.orchestra.run.smithi088.stdout:30 Sep 01:30:06 ntpd[22733]: Command line: ntpd -gq 2023-09-30T01:30:06.064 INFO:teuthology.orchestra.run.smithi088.stdout:30 Sep 01:30:06 ntpd[22733]: proto: precision = 0.082 usec (-23) 2023-09-30T01:30:06.064 INFO:teuthology.orchestra.run.smithi107.stdout:30 Sep 01:30:06 ntpd[22728]: ntpd 4.2.8p12@1.3728-o (1): Starting 2023-09-30T01:30:06.064 INFO:teuthology.orchestra.run.smithi107.stdout:30 Sep 01:30:06 ntpd[22728]: Command line: ntpd -gq 2023-09-30T01:30:06.064 INFO:teuthology.orchestra.run.smithi088.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2023-09-30T01:30:06.065 INFO:teuthology.orchestra.run.smithi088.stdout:30 Sep 01:30:06 ntpd[22733]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2023-09-30T01:30:06.065 INFO:teuthology.orchestra.run.smithi088.stderr:restrict ::: KOD does nothing without LIMITED. 2023-09-30T01:30:06.066 INFO:teuthology.orchestra.run.smithi088.stdout:30 Sep 01:30:06 ntpd[22733]: restrict ::: KOD does nothing without LIMITED. 2023-09-30T01:30:06.066 INFO:teuthology.orchestra.run.smithi088.stdout:30 Sep 01:30:06 ntpd[22733]: Listen and drop on 0 v6wildcard [::]:123 2023-09-30T01:30:06.066 INFO:teuthology.orchestra.run.smithi088.stdout:30 Sep 01:30:06 ntpd[22733]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2023-09-30T01:30:06.066 INFO:teuthology.orchestra.run.smithi088.stdout:30 Sep 01:30:06 ntpd[22733]: Listen normally on 2 lo 127.0.0.1:123 2023-09-30T01:30:06.066 INFO:teuthology.orchestra.run.smithi088.stdout:30 Sep 01:30:06 ntpd[22733]: Listen normally on 3 enp3s0f1 172.21.15.88:123 2023-09-30T01:30:06.066 INFO:teuthology.orchestra.run.smithi088.stdout:30 Sep 01:30:06 ntpd[22733]: Listen normally on 4 lo [::1]:123 2023-09-30T01:30:06.066 INFO:teuthology.orchestra.run.smithi088.stdout:30 Sep 01:30:06 ntpd[22733]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe8f:d42d%5]:123 2023-09-30T01:30:06.066 INFO:teuthology.orchestra.run.smithi088.stdout:30 Sep 01:30:06 ntpd[22733]: Listening on routing socket on fd #22 for interface updates 2023-09-30T01:30:06.067 INFO:teuthology.orchestra.run.smithi107.stdout:30 Sep 01:30:06 ntpd[22728]: proto: precision = 0.087 usec (-23) 2023-09-30T01:30:06.067 INFO:teuthology.orchestra.run.smithi107.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2023-09-30T01:30:06.067 INFO:teuthology.orchestra.run.smithi107.stdout:30 Sep 01:30:06 ntpd[22728]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2023-09-30T01:30:06.068 INFO:teuthology.orchestra.run.smithi107.stderr:restrict ::: KOD does nothing without LIMITED. 2023-09-30T01:30:06.069 INFO:teuthology.orchestra.run.smithi107.stdout:30 Sep 01:30:06 ntpd[22728]: restrict ::: KOD does nothing without LIMITED. 2023-09-30T01:30:06.069 INFO:teuthology.orchestra.run.smithi107.stdout:30 Sep 01:30:06 ntpd[22728]: Listen and drop on 0 v6wildcard [::]:123 2023-09-30T01:30:06.069 INFO:teuthology.orchestra.run.smithi107.stdout:30 Sep 01:30:06 ntpd[22728]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2023-09-30T01:30:06.069 INFO:teuthology.orchestra.run.smithi107.stdout:30 Sep 01:30:06 ntpd[22728]: Listen normally on 2 lo 127.0.0.1:123 2023-09-30T01:30:06.069 INFO:teuthology.orchestra.run.smithi107.stdout:30 Sep 01:30:06 ntpd[22728]: Listen normally on 3 enp3s0f1 172.21.15.107:123 2023-09-30T01:30:06.069 INFO:teuthology.orchestra.run.smithi107.stdout:30 Sep 01:30:06 ntpd[22728]: Listen normally on 4 lo [::1]:123 2023-09-30T01:30:06.069 INFO:teuthology.orchestra.run.smithi107.stdout:30 Sep 01:30:06 ntpd[22728]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe8f:47eb%5]:123 2023-09-30T01:30:06.069 INFO:teuthology.orchestra.run.smithi107.stdout:30 Sep 01:30:06 ntpd[22728]: Listening on routing socket on fd #22 for interface updates 2023-09-30T01:30:07.064 INFO:teuthology.orchestra.run.smithi088.stderr:30 Sep 01:30:07 ntpd[22733]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-09-30T01:30:07.064 INFO:teuthology.orchestra.run.smithi088.stderr:30 Sep 01:30:07 ntpd[22733]: can't open /var/log/ntpstats/rawstats.20230930: Permission denied 2023-09-30T01:30:07.064 INFO:teuthology.orchestra.run.smithi088.stderr:30 Sep 01:30:07 ntpd[22733]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-09-30T01:30:07.065 INFO:teuthology.orchestra.run.smithi088.stderr:30 Sep 01:30:07 ntpd[22733]: can't open /var/log/ntpstats/rawstats.20230930: Permission denied 2023-09-30T01:30:07.065 INFO:teuthology.orchestra.run.smithi088.stderr:30 Sep 01:30:07 ntpd[22733]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-09-30T01:30:07.065 INFO:teuthology.orchestra.run.smithi088.stderr:30 Sep 01:30:07 ntpd[22733]: can't open /var/log/ntpstats/peerstats.20230930: Permission denied 2023-09-30T01:30:07.065 INFO:teuthology.orchestra.run.smithi107.stderr:30 Sep 01:30:07 ntpd[22728]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-09-30T01:30:07.066 INFO:teuthology.orchestra.run.smithi107.stderr:30 Sep 01:30:07 ntpd[22728]: can't open /var/log/ntpstats/rawstats.20230930: Permission denied 2023-09-30T01:30:08.064 INFO:teuthology.orchestra.run.smithi088.stderr:30 Sep 01:30:08 ntpd[22733]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-09-30T01:30:08.064 INFO:teuthology.orchestra.run.smithi088.stderr:30 Sep 01:30:08 ntpd[22733]: can't open /var/log/ntpstats/rawstats.20230930: Permission denied 2023-09-30T01:30:08.065 INFO:teuthology.orchestra.run.smithi088.stderr:30 Sep 01:30:08 ntpd[22733]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-09-30T01:30:08.065 INFO:teuthology.orchestra.run.smithi088.stderr:30 Sep 01:30:08 ntpd[22733]: can't open /var/log/ntpstats/rawstats.20230930: Permission denied 2023-09-30T01:30:08.065 INFO:teuthology.orchestra.run.smithi088.stderr:30 Sep 01:30:08 ntpd[22733]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-09-30T01:30:08.065 INFO:teuthology.orchestra.run.smithi088.stderr:30 Sep 01:30:08 ntpd[22733]: can't open /var/log/ntpstats/peerstats.20230930: Permission denied 2023-09-30T01:30:08.066 INFO:teuthology.orchestra.run.smithi107.stderr:30 Sep 01:30:08 ntpd[22728]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-09-30T01:30:08.066 INFO:teuthology.orchestra.run.smithi107.stderr:30 Sep 01:30:08 ntpd[22728]: can't open /var/log/ntpstats/rawstats.20230930: Permission denied 2023-09-30T01:30:08.066 INFO:teuthology.orchestra.run.smithi107.stderr:30 Sep 01:30:08 ntpd[22728]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-09-30T01:30:08.066 INFO:teuthology.orchestra.run.smithi107.stderr:30 Sep 01:30:08 ntpd[22728]: can't open /var/log/ntpstats/rawstats.20230930: Permission denied 2023-09-30T01:30:08.066 INFO:teuthology.orchestra.run.smithi107.stderr:30 Sep 01:30:08 ntpd[22728]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-09-30T01:30:08.066 INFO:teuthology.orchestra.run.smithi107.stderr:30 Sep 01:30:08 ntpd[22728]: can't open /var/log/ntpstats/peerstats.20230930: Permission denied 2023-09-30T01:30:09.064 INFO:teuthology.orchestra.run.smithi088.stderr:30 Sep 01:30:09 ntpd[22733]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-09-30T01:30:09.064 INFO:teuthology.orchestra.run.smithi088.stderr:30 Sep 01:30:09 ntpd[22733]: can't open /var/log/ntpstats/rawstats.20230930: Permission denied 2023-09-30T01:30:09.064 INFO:teuthology.orchestra.run.smithi088.stderr:30 Sep 01:30:09 ntpd[22733]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-09-30T01:30:09.064 INFO:teuthology.orchestra.run.smithi088.stderr:30 Sep 01:30:09 ntpd[22733]: can't open /var/log/ntpstats/rawstats.20230930: Permission denied 2023-09-30T01:30:09.065 INFO:teuthology.orchestra.run.smithi088.stderr:30 Sep 01:30:09 ntpd[22733]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-09-30T01:30:09.065 INFO:teuthology.orchestra.run.smithi088.stderr:30 Sep 01:30:09 ntpd[22733]: can't open /var/log/ntpstats/rawstats.20230930: Permission denied 2023-09-30T01:30:09.065 INFO:teuthology.orchestra.run.smithi088.stderr:30 Sep 01:30:09 ntpd[22733]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-09-30T01:30:09.065 INFO:teuthology.orchestra.run.smithi088.stderr:30 Sep 01:30:09 ntpd[22733]: can't open /var/log/ntpstats/peerstats.20230930: Permission denied 2023-09-30T01:30:10.064 INFO:teuthology.orchestra.run.smithi088.stderr:30 Sep 01:30:10 ntpd[22733]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-09-30T01:30:10.064 INFO:teuthology.orchestra.run.smithi088.stderr:30 Sep 01:30:10 ntpd[22733]: can't open /var/log/ntpstats/rawstats.20230930: Permission denied 2023-09-30T01:30:10.064 INFO:teuthology.orchestra.run.smithi088.stderr:30 Sep 01:30:10 ntpd[22733]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-09-30T01:30:10.065 INFO:teuthology.orchestra.run.smithi088.stderr:30 Sep 01:30:10 ntpd[22733]: can't open /var/log/ntpstats/rawstats.20230930: Permission denied 2023-09-30T01:30:10.065 INFO:teuthology.orchestra.run.smithi088.stderr:30 Sep 01:30:10 ntpd[22733]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-09-30T01:30:10.065 INFO:teuthology.orchestra.run.smithi088.stderr:30 Sep 01:30:10 ntpd[22733]: can't open /var/log/ntpstats/rawstats.20230930: Permission denied 2023-09-30T01:30:10.065 INFO:teuthology.orchestra.run.smithi088.stderr:30 Sep 01:30:10 ntpd[22733]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-09-30T01:30:10.065 INFO:teuthology.orchestra.run.smithi088.stderr:30 Sep 01:30:10 ntpd[22733]: can't open /var/log/ntpstats/peerstats.20230930: Permission denied 2023-09-30T01:30:10.065 INFO:teuthology.orchestra.run.smithi088.stderr:30 Sep 01:30:10 ntpd[22733]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-09-30T01:30:10.065 INFO:teuthology.orchestra.run.smithi088.stderr:30 Sep 01:30:10 ntpd[22733]: can't open /var/log/ntpstats/rawstats.20230930: Permission denied 2023-09-30T01:30:10.065 INFO:teuthology.orchestra.run.smithi088.stderr:30 Sep 01:30:10 ntpd[22733]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-09-30T01:30:10.065 INFO:teuthology.orchestra.run.smithi088.stderr:30 Sep 01:30:10 ntpd[22733]: can't open /var/log/ntpstats/peerstats.20230930: Permission denied 2023-09-30T01:30:10.067 INFO:teuthology.orchestra.run.smithi107.stderr:30 Sep 01:30:10 ntpd[22728]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-09-30T01:30:10.067 INFO:teuthology.orchestra.run.smithi107.stderr:30 Sep 01:30:10 ntpd[22728]: can't open /var/log/ntpstats/rawstats.20230930: Permission denied 2023-09-30T01:30:10.067 INFO:teuthology.orchestra.run.smithi107.stderr:30 Sep 01:30:10 ntpd[22728]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-09-30T01:30:10.067 INFO:teuthology.orchestra.run.smithi107.stderr:30 Sep 01:30:10 ntpd[22728]: can't open /var/log/ntpstats/rawstats.20230930: Permission denied 2023-09-30T01:30:10.067 INFO:teuthology.orchestra.run.smithi107.stderr:30 Sep 01:30:10 ntpd[22728]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-09-30T01:30:10.067 INFO:teuthology.orchestra.run.smithi107.stderr:30 Sep 01:30:10 ntpd[22728]: can't open /var/log/ntpstats/rawstats.20230930: Permission denied 2023-09-30T01:30:10.067 INFO:teuthology.orchestra.run.smithi107.stderr:30 Sep 01:30:10 ntpd[22728]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-09-30T01:30:10.067 INFO:teuthology.orchestra.run.smithi107.stderr:30 Sep 01:30:10 ntpd[22728]: can't open /var/log/ntpstats/peerstats.20230930: Permission denied 2023-09-30T01:30:10.067 INFO:teuthology.orchestra.run.smithi107.stderr:30 Sep 01:30:10 ntpd[22728]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-09-30T01:30:10.067 INFO:teuthology.orchestra.run.smithi107.stderr:30 Sep 01:30:10 ntpd[22728]: can't open /var/log/ntpstats/rawstats.20230930: Permission denied 2023-09-30T01:30:10.067 INFO:teuthology.orchestra.run.smithi107.stderr:30 Sep 01:30:10 ntpd[22728]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-09-30T01:30:10.067 INFO:teuthology.orchestra.run.smithi107.stderr:30 Sep 01:30:10 ntpd[22728]: can't open /var/log/ntpstats/peerstats.20230930: Permission denied 2023-09-30T01:30:11.064 INFO:teuthology.orchestra.run.smithi088.stderr:30 Sep 01:30:11 ntpd[22733]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-09-30T01:30:11.064 INFO:teuthology.orchestra.run.smithi088.stderr:30 Sep 01:30:11 ntpd[22733]: can't open /var/log/ntpstats/rawstats.20230930: Permission denied 2023-09-30T01:30:11.064 INFO:teuthology.orchestra.run.smithi088.stderr:30 Sep 01:30:11 ntpd[22733]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-09-30T01:30:11.064 INFO:teuthology.orchestra.run.smithi088.stderr:30 Sep 01:30:11 ntpd[22733]: can't open /var/log/ntpstats/rawstats.20230930: Permission denied 2023-09-30T01:30:11.064 INFO:teuthology.orchestra.run.smithi088.stderr:30 Sep 01:30:11 ntpd[22733]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-09-30T01:30:11.064 INFO:teuthology.orchestra.run.smithi088.stderr:30 Sep 01:30:11 ntpd[22733]: can't open /var/log/ntpstats/peerstats.20230930: Permission denied 2023-09-30T01:30:11.065 INFO:teuthology.orchestra.run.smithi107.stderr:30 Sep 01:30:11 ntpd[22728]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-09-30T01:30:11.065 INFO:teuthology.orchestra.run.smithi107.stderr:30 Sep 01:30:11 ntpd[22728]: can't open /var/log/ntpstats/rawstats.20230930: Permission denied 2023-09-30T01:30:11.066 INFO:teuthology.orchestra.run.smithi107.stderr:30 Sep 01:30:11 ntpd[22728]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-09-30T01:30:11.066 INFO:teuthology.orchestra.run.smithi107.stderr:30 Sep 01:30:11 ntpd[22728]: can't open /var/log/ntpstats/rawstats.20230930: Permission denied 2023-09-30T01:30:11.066 INFO:teuthology.orchestra.run.smithi107.stderr:30 Sep 01:30:11 ntpd[22728]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-09-30T01:30:11.066 INFO:teuthology.orchestra.run.smithi107.stderr:30 Sep 01:30:11 ntpd[22728]: can't open /var/log/ntpstats/peerstats.20230930: Permission denied 2023-09-30T01:30:12.064 INFO:teuthology.orchestra.run.smithi088.stderr:30 Sep 01:30:12 ntpd[22733]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-09-30T01:30:12.064 INFO:teuthology.orchestra.run.smithi088.stderr:30 Sep 01:30:12 ntpd[22733]: can't open /var/log/ntpstats/rawstats.20230930: Permission denied 2023-09-30T01:30:12.064 INFO:teuthology.orchestra.run.smithi088.stderr:30 Sep 01:30:12 ntpd[22733]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-09-30T01:30:12.065 INFO:teuthology.orchestra.run.smithi088.stderr:30 Sep 01:30:12 ntpd[22733]: can't open /var/log/ntpstats/rawstats.20230930: Permission denied 2023-09-30T01:30:12.065 INFO:teuthology.orchestra.run.smithi088.stderr:30 Sep 01:30:12 ntpd[22733]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-09-30T01:30:12.065 INFO:teuthology.orchestra.run.smithi088.stderr:30 Sep 01:30:12 ntpd[22733]: can't open /var/log/ntpstats/rawstats.20230930: Permission denied 2023-09-30T01:30:12.065 INFO:teuthology.orchestra.run.smithi088.stderr:30 Sep 01:30:12 ntpd[22733]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-09-30T01:30:12.065 INFO:teuthology.orchestra.run.smithi088.stderr:30 Sep 01:30:12 ntpd[22733]: can't open /var/log/ntpstats/peerstats.20230930: Permission denied 2023-09-30T01:30:12.065 INFO:teuthology.orchestra.run.smithi088.stderr:30 Sep 01:30:12 ntpd[22733]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-09-30T01:30:12.065 INFO:teuthology.orchestra.run.smithi088.stderr:30 Sep 01:30:12 ntpd[22733]: can't open /var/log/ntpstats/rawstats.20230930: Permission denied 2023-09-30T01:30:12.065 INFO:teuthology.orchestra.run.smithi088.stderr:30 Sep 01:30:12 ntpd[22733]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-09-30T01:30:12.065 INFO:teuthology.orchestra.run.smithi088.stderr:30 Sep 01:30:12 ntpd[22733]: can't open /var/log/ntpstats/peerstats.20230930: Permission denied 2023-09-30T01:30:12.066 INFO:teuthology.orchestra.run.smithi107.stderr:30 Sep 01:30:12 ntpd[22728]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-09-30T01:30:12.067 INFO:teuthology.orchestra.run.smithi107.stderr:30 Sep 01:30:12 ntpd[22728]: can't open /var/log/ntpstats/rawstats.20230930: Permission denied 2023-09-30T01:30:12.067 INFO:teuthology.orchestra.run.smithi107.stderr:30 Sep 01:30:12 ntpd[22728]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-09-30T01:30:12.067 INFO:teuthology.orchestra.run.smithi107.stderr:30 Sep 01:30:12 ntpd[22728]: can't open /var/log/ntpstats/rawstats.20230930: Permission denied 2023-09-30T01:30:12.067 INFO:teuthology.orchestra.run.smithi107.stderr:30 Sep 01:30:12 ntpd[22728]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-09-30T01:30:12.067 INFO:teuthology.orchestra.run.smithi107.stderr:30 Sep 01:30:12 ntpd[22728]: can't open /var/log/ntpstats/rawstats.20230930: Permission denied 2023-09-30T01:30:12.067 INFO:teuthology.orchestra.run.smithi107.stderr:30 Sep 01:30:12 ntpd[22728]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-09-30T01:30:12.067 INFO:teuthology.orchestra.run.smithi107.stderr:30 Sep 01:30:12 ntpd[22728]: can't open /var/log/ntpstats/peerstats.20230930: Permission denied 2023-09-30T01:30:12.067 INFO:teuthology.orchestra.run.smithi107.stderr:30 Sep 01:30:12 ntpd[22728]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-09-30T01:30:12.067 INFO:teuthology.orchestra.run.smithi107.stderr:30 Sep 01:30:12 ntpd[22728]: can't open /var/log/ntpstats/rawstats.20230930: Permission denied 2023-09-30T01:30:12.067 INFO:teuthology.orchestra.run.smithi107.stderr:30 Sep 01:30:12 ntpd[22728]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-09-30T01:30:12.067 INFO:teuthology.orchestra.run.smithi107.stderr:30 Sep 01:30:12 ntpd[22728]: can't open /var/log/ntpstats/peerstats.20230930: Permission denied 2023-09-30T01:30:13.064 INFO:teuthology.orchestra.run.smithi088.stderr:30 Sep 01:30:13 ntpd[22733]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-09-30T01:30:13.064 INFO:teuthology.orchestra.run.smithi088.stderr:30 Sep 01:30:13 ntpd[22733]: can't open /var/log/ntpstats/rawstats.20230930: Permission denied 2023-09-30T01:30:13.064 INFO:teuthology.orchestra.run.smithi088.stderr:30 Sep 01:30:13 ntpd[22733]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-09-30T01:30:13.064 INFO:teuthology.orchestra.run.smithi088.stderr:30 Sep 01:30:13 ntpd[22733]: can't open /var/log/ntpstats/rawstats.20230930: Permission denied 2023-09-30T01:30:13.065 INFO:teuthology.orchestra.run.smithi088.stderr:30 Sep 01:30:13 ntpd[22733]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-09-30T01:30:13.065 INFO:teuthology.orchestra.run.smithi088.stderr:30 Sep 01:30:13 ntpd[22733]: can't open /var/log/ntpstats/peerstats.20230930: Permission denied 2023-09-30T01:30:13.065 INFO:teuthology.orchestra.run.smithi088.stdout:30 Sep 01:30:13 ntpd[22733]: ntpd: time slew +0.000617 s 2023-09-30T01:30:13.065 INFO:teuthology.orchestra.run.smithi088.stdout:ntpd: time slew +0.000617s 2023-09-30T01:30:13.065 INFO:teuthology.orchestra.run.smithi088.stderr:30 Sep 01:30:13 ntpd[22733]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2023-09-30T01:30:13.066 INFO:teuthology.orchestra.run.smithi088.stderr:30 Sep 01:30:13 ntpd[22733]: can't open /var/log/ntpstats/loopstats.20230930: Permission denied 2023-09-30T01:30:13.067 INFO:teuthology.orchestra.run.smithi107.stderr:30 Sep 01:30:13 ntpd[22728]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-09-30T01:30:13.067 INFO:teuthology.orchestra.run.smithi107.stderr:30 Sep 01:30:13 ntpd[22728]: can't open /var/log/ntpstats/rawstats.20230930: Permission denied 2023-09-30T01:30:13.067 INFO:teuthology.orchestra.run.smithi107.stderr:30 Sep 01:30:13 ntpd[22728]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-09-30T01:30:13.067 INFO:teuthology.orchestra.run.smithi107.stderr:30 Sep 01:30:13 ntpd[22728]: can't open /var/log/ntpstats/rawstats.20230930: Permission denied 2023-09-30T01:30:13.067 INFO:teuthology.orchestra.run.smithi107.stderr:30 Sep 01:30:13 ntpd[22728]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-09-30T01:30:13.067 INFO:teuthology.orchestra.run.smithi107.stderr:30 Sep 01:30:13 ntpd[22728]: can't open /var/log/ntpstats/peerstats.20230930: Permission denied 2023-09-30T01:30:13.138 INFO:teuthology.orchestra.run.smithi088.stdout: remote refid st t when poll reach delay offset jitter 2023-09-30T01:30:13.138 INFO:teuthology.orchestra.run.smithi088.stdout:============================================================================== 2023-09-30T01:30:13.138 INFO:teuthology.orchestra.run.smithi088.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2023-09-30T01:30:13.138 INFO:teuthology.orchestra.run.smithi088.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2023-09-30T01:30:13.138 INFO:teuthology.orchestra.run.smithi088.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2023-09-30T01:30:13.138 INFO:teuthology.orchestra.run.smithi088.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2023-09-30T01:30:14.065 INFO:teuthology.orchestra.run.smithi107.stderr:30 Sep 01:30:14 ntpd[22728]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-09-30T01:30:14.066 INFO:teuthology.orchestra.run.smithi107.stderr:30 Sep 01:30:14 ntpd[22728]: can't open /var/log/ntpstats/rawstats.20230930: Permission denied 2023-09-30T01:30:14.066 INFO:teuthology.orchestra.run.smithi107.stderr:30 Sep 01:30:14 ntpd[22728]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-09-30T01:30:14.066 INFO:teuthology.orchestra.run.smithi107.stderr:30 Sep 01:30:14 ntpd[22728]: can't open /var/log/ntpstats/rawstats.20230930: Permission denied 2023-09-30T01:30:14.066 INFO:teuthology.orchestra.run.smithi107.stderr:30 Sep 01:30:14 ntpd[22728]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-09-30T01:30:14.066 INFO:teuthology.orchestra.run.smithi107.stderr:30 Sep 01:30:14 ntpd[22728]: can't open /var/log/ntpstats/rawstats.20230930: Permission denied 2023-09-30T01:30:14.066 INFO:teuthology.orchestra.run.smithi107.stderr:30 Sep 01:30:14 ntpd[22728]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-09-30T01:30:14.066 INFO:teuthology.orchestra.run.smithi107.stderr:30 Sep 01:30:14 ntpd[22728]: can't open /var/log/ntpstats/peerstats.20230930: Permission denied 2023-09-30T01:30:14.066 INFO:teuthology.orchestra.run.smithi107.stderr:30 Sep 01:30:14 ntpd[22728]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-09-30T01:30:14.066 INFO:teuthology.orchestra.run.smithi107.stderr:30 Sep 01:30:14 ntpd[22728]: can't open /var/log/ntpstats/rawstats.20230930: Permission denied 2023-09-30T01:30:14.066 INFO:teuthology.orchestra.run.smithi107.stderr:30 Sep 01:30:14 ntpd[22728]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-09-30T01:30:14.067 INFO:teuthology.orchestra.run.smithi107.stderr:30 Sep 01:30:14 ntpd[22728]: can't open /var/log/ntpstats/peerstats.20230930: Permission denied 2023-09-30T01:30:14.067 INFO:teuthology.orchestra.run.smithi107.stdout:30 Sep 01:30:14 ntpd[22728]: ntpd: time slew +0.001321 s 2023-09-30T01:30:14.067 INFO:teuthology.orchestra.run.smithi107.stdout:ntpd: time slew +0.001321s 2023-09-30T01:30:14.067 INFO:teuthology.orchestra.run.smithi107.stderr:30 Sep 01:30:14 ntpd[22728]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2023-09-30T01:30:14.067 INFO:teuthology.orchestra.run.smithi107.stderr:30 Sep 01:30:14 ntpd[22728]: can't open /var/log/ntpstats/loopstats.20230930: Permission denied 2023-09-30T01:30:14.137 INFO:teuthology.orchestra.run.smithi107.stdout: remote refid st t when poll reach delay offset jitter 2023-09-30T01:30:14.138 INFO:teuthology.orchestra.run.smithi107.stdout:============================================================================== 2023-09-30T01:30:14.138 INFO:teuthology.orchestra.run.smithi107.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2023-09-30T01:30:14.138 INFO:teuthology.orchestra.run.smithi107.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2023-09-30T01:30:14.138 INFO:teuthology.orchestra.run.smithi107.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2023-09-30T01:30:14.138 INFO:teuthology.orchestra.run.smithi107.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2023-09-30T01:30:14.139 INFO:teuthology.run_tasks:Running task install... 2023-09-30T01:30:14.147 DEBUG:teuthology.task.install:project ceph 2023-09-30T01:30:14.148 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '93dce5e17918f8aca65993a7a2dc40233a47b481'}} 2023-09-30T01:30:14.148 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': '93dce5e17918f8aca65993a7a2dc40233a47b481'} 2023-09-30T01:30:14.148 INFO:teuthology.task.install:Using flavor: default 2023-09-30T01:30:14.153 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2023-09-30T01:30:14.154 INFO:teuthology.task.install:extra packages: [] 2023-09-30T01:30:14.154 DEBUG:teuthology.orchestra.run.smithi088:> sudo apt-key list | grep Ceph 2023-09-30T01:30:14.154 DEBUG:teuthology.orchestra.run.smithi107:> sudo apt-key list | grep Ceph 2023-09-30T01:30:14.248 INFO:teuthology.orchestra.run.smithi107.stderr:Warning: apt-key output should not be parsed (stdout is not a terminal) 2023-09-30T01:30:14.252 INFO:teuthology.orchestra.run.smithi088.stderr:Warning: apt-key output should not be parsed (stdout is not a terminal) 2023-09-30T01:30:14.311 INFO:teuthology.orchestra.run.smithi107.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2023-09-30T01:30:14.311 INFO:teuthology.orchestra.run.smithi107.stdout:uid [ unknown] Ceph.com (release key) 2023-09-30T01:30:14.312 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on remote deb x86_64 2023-09-30T01:30:14.313 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&sha1=93dce5e17918f8aca65993a7a2dc40233a47b481 2023-09-30T01:30:14.317 INFO:teuthology.orchestra.run.smithi088.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2023-09-30T01:30:14.317 INFO:teuthology.orchestra.run.smithi088.stdout:uid [ unknown] Ceph.com (release key) 2023-09-30T01:30:14.318 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on remote deb x86_64 2023-09-30T01:30:14.318 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&sha1=93dce5e17918f8aca65993a7a2dc40233a47b481 2023-09-30T01:30:14.460 INFO:teuthology.task.install.deb:Pulling from https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/ubuntu/focal/flavors/default/ 2023-09-30T01:30:14.460 INFO:teuthology.task.install.deb:Package version is 17.2.6-1165-g93dce5e1-1focal 2023-09-30T01:30:14.531 INFO:teuthology.task.install.deb:Pulling from https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/ubuntu/focal/flavors/default/ 2023-09-30T01:30:14.531 INFO:teuthology.task.install.deb:Package version is 17.2.6-1165-g93dce5e1-1focal 2023-09-30T01:30:14.568 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2023-09-30T01:30:14.569 DEBUG:teuthology.orchestra.run.smithi088:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2023-09-30T01:30:14.585 DEBUG:teuthology.orchestra.run.smithi088:> sudo apt-get update 2023-09-30T01:30:14.701 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-09-30T01:30:14.701 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2023-09-30T01:30:14.718 DEBUG:teuthology.orchestra.run.smithi107:> sudo apt-get update 2023-09-30T01:30:14.746 INFO:teuthology.orchestra.run.smithi088.stdout:Hit:1 http://archive.ubuntu.com/ubuntu focal InRelease 2023-09-30T01:30:14.746 INFO:teuthology.orchestra.run.smithi088.stdout:Hit:2 http://security.ubuntu.com/ubuntu focal-security InRelease 2023-09-30T01:30:14.753 INFO:teuthology.orchestra.run.smithi088.stdout:Hit:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2023-09-30T01:30:14.773 INFO:teuthology.orchestra.run.smithi088.stdout:Hit:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2023-09-30T01:30:14.855 INFO:teuthology.orchestra.run.smithi107.stdout:Hit:1 http://security.ubuntu.com/ubuntu focal-security InRelease 2023-09-30T01:30:14.856 INFO:teuthology.orchestra.run.smithi107.stdout:Hit:2 http://archive.ubuntu.com/ubuntu focal InRelease 2023-09-30T01:30:14.862 INFO:teuthology.orchestra.run.smithi107.stdout:Hit:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2023-09-30T01:30:14.862 INFO:teuthology.orchestra.run.smithi088.stdout:Ign:5 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/ubuntu/focal/flavors/default focal InRelease 2023-09-30T01:30:14.879 INFO:teuthology.orchestra.run.smithi107.stdout:Hit:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2023-09-30T01:30:14.899 INFO:teuthology.orchestra.run.smithi088.stdout:Get:6 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/ubuntu/focal/flavors/default focal Release [7,662 B] 2023-09-30T01:30:14.908 INFO:teuthology.orchestra.run.smithi107.stdout:Ign:5 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/ubuntu/focal/flavors/default focal InRelease 2023-09-30T01:30:14.929 INFO:teuthology.orchestra.run.smithi107.stdout:Get:6 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/ubuntu/focal/flavors/default focal Release [7,662 B] 2023-09-30T01:30:14.941 INFO:teuthology.orchestra.run.smithi088.stdout:Ign:7 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/ubuntu/focal/flavors/default focal Release.gpg 2023-09-30T01:30:14.953 INFO:teuthology.orchestra.run.smithi107.stdout:Ign:7 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/ubuntu/focal/flavors/default focal Release.gpg 2023-09-30T01:30:14.975 INFO:teuthology.orchestra.run.smithi107.stdout:Get:8 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/ubuntu/focal/flavors/default focal/main i386 Packages [3,375 B] 2023-09-30T01:30:14.980 INFO:teuthology.orchestra.run.smithi088.stdout:Get:8 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/ubuntu/focal/flavors/default focal/main i386 Packages [3,375 B] 2023-09-30T01:30:15.018 INFO:teuthology.orchestra.run.smithi107.stdout:Get:9 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/ubuntu/focal/flavors/default focal/main amd64 Packages [17.4 kB] 2023-09-30T01:30:15.040 INFO:teuthology.orchestra.run.smithi088.stdout:Get:9 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/ubuntu/focal/flavors/default focal/main amd64 Packages [17.4 kB] 2023-09-30T01:30:15.405 INFO:teuthology.orchestra.run.smithi088.stdout:Fetched 28.4 kB in 1s (43.6 kB/s) 2023-09-30T01:30:15.506 INFO:teuthology.orchestra.run.smithi107.stdout:Fetched 28.4 kB in 1s (45.3 kB/s) 2023-09-30T01:30:16.424 INFO:teuthology.orchestra.run.smithi088.stdout:Reading package lists... 2023-09-30T01:30:16.449 DEBUG:teuthology.orchestra.run.smithi088:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=17.2.6-1165-g93dce5e1-1focal cephadm=17.2.6-1165-g93dce5e1-1focal ceph-mds=17.2.6-1165-g93dce5e1-1focal ceph-mgr=17.2.6-1165-g93dce5e1-1focal ceph-common=17.2.6-1165-g93dce5e1-1focal ceph-fuse=17.2.6-1165-g93dce5e1-1focal ceph-test=17.2.6-1165-g93dce5e1-1focal ceph-volume=17.2.6-1165-g93dce5e1-1focal radosgw=17.2.6-1165-g93dce5e1-1focal python3-rados=17.2.6-1165-g93dce5e1-1focal python3-rgw=17.2.6-1165-g93dce5e1-1focal python3-cephfs=17.2.6-1165-g93dce5e1-1focal python3-rbd=17.2.6-1165-g93dce5e1-1focal libcephfs2=17.2.6-1165-g93dce5e1-1focal libcephfs-dev=17.2.6-1165-g93dce5e1-1focal librados2=17.2.6-1165-g93dce5e1-1focal librbd1=17.2.6-1165-g93dce5e1-1focal rbd-fuse=17.2.6-1165-g93dce5e1-1focal 2023-09-30T01:30:16.521 INFO:teuthology.orchestra.run.smithi088.stdout:Reading package lists... 2023-09-30T01:30:16.523 INFO:teuthology.orchestra.run.smithi107.stdout:Reading package lists... 2023-09-30T01:30:16.548 DEBUG:teuthology.orchestra.run.smithi107:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=17.2.6-1165-g93dce5e1-1focal cephadm=17.2.6-1165-g93dce5e1-1focal ceph-mds=17.2.6-1165-g93dce5e1-1focal ceph-mgr=17.2.6-1165-g93dce5e1-1focal ceph-common=17.2.6-1165-g93dce5e1-1focal ceph-fuse=17.2.6-1165-g93dce5e1-1focal ceph-test=17.2.6-1165-g93dce5e1-1focal ceph-volume=17.2.6-1165-g93dce5e1-1focal radosgw=17.2.6-1165-g93dce5e1-1focal python3-rados=17.2.6-1165-g93dce5e1-1focal python3-rgw=17.2.6-1165-g93dce5e1-1focal python3-cephfs=17.2.6-1165-g93dce5e1-1focal python3-rbd=17.2.6-1165-g93dce5e1-1focal libcephfs2=17.2.6-1165-g93dce5e1-1focal libcephfs-dev=17.2.6-1165-g93dce5e1-1focal librados2=17.2.6-1165-g93dce5e1-1focal librbd1=17.2.6-1165-g93dce5e1-1focal rbd-fuse=17.2.6-1165-g93dce5e1-1focal 2023-09-30T01:30:16.614 INFO:teuthology.orchestra.run.smithi107.stdout:Reading package lists... 2023-09-30T01:30:16.720 INFO:teuthology.orchestra.run.smithi088.stdout:Building dependency tree... 2023-09-30T01:30:16.720 INFO:teuthology.orchestra.run.smithi088.stdout:Reading state information... 2023-09-30T01:30:16.815 INFO:teuthology.orchestra.run.smithi107.stdout:Building dependency tree... 2023-09-30T01:30:16.816 INFO:teuthology.orchestra.run.smithi107.stdout:Reading state information... 2023-09-30T01:30:16.902 INFO:teuthology.orchestra.run.smithi088.stdout:The following packages were automatically installed and are no longer required: 2023-09-30T01:30:16.903 INFO:teuthology.orchestra.run.smithi088.stdout: libboost-iostreams1.71.0 libboost-thread1.71.0 2023-09-30T01:30:16.903 INFO:teuthology.orchestra.run.smithi088.stdout:Use 'sudo apt autoremove' to remove them. 2023-09-30T01:30:16.903 INFO:teuthology.orchestra.run.smithi088.stdout:The following additional packages will be installed: 2023-09-30T01:30:16.903 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2023-09-30T01:30:16.904 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd fonts-mathjax jq 2023-09-30T01:30:16.904 INFO:teuthology.orchestra.run.smithi088.stdout: libdouble-conversion3 libjq1 libjs-mathjax libjs-sphinxdoc libjs-underscore 2023-09-30T01:30:16.904 INFO:teuthology.orchestra.run.smithi088.stdout: liblbfgsb0 liblttng-ust0 liblua5.3-0 liboath0 libonig5 libpcre2-16-0 2023-09-30T01:30:16.904 INFO:teuthology.orchestra.run.smithi088.stdout: libqt5core5a libqt5dbus5 libqt5network5 librabbitmq4 libradosstriper1 2023-09-30T01:30:16.905 INFO:teuthology.orchestra.run.smithi088.stdout: librdkafka1 librgw2 libsqlite3-mod-ceph libthrift-0.13.0 nvme-cli 2023-09-30T01:30:16.905 INFO:teuthology.orchestra.run.smithi088.stdout: python-natsort-doc python-pastedeploy-tpl python3-atomicwrites python3-attr 2023-09-30T01:30:16.906 INFO:teuthology.orchestra.run.smithi088.stdout: python3-bcrypt python3-blinker python3-bs4 python3-cachetools 2023-09-30T01:30:16.906 INFO:teuthology.orchestra.run.smithi088.stdout: python3-ceph-argparse python3-ceph-common python3-cherrypy3 2023-09-30T01:30:16.906 INFO:teuthology.orchestra.run.smithi088.stdout: python3-cryptography python3-decorator python3-google-auth python3-html5lib 2023-09-30T01:30:16.906 INFO:teuthology.orchestra.run.smithi088.stdout: python3-jinja2 python3-joblib python3-jwt python3-kubernetes 2023-09-30T01:30:16.906 INFO:teuthology.orchestra.run.smithi088.stdout: python3-logutils python3-lxml python3-mako python3-markupsafe 2023-09-30T01:30:16.906 INFO:teuthology.orchestra.run.smithi088.stdout: python3-natsort python3-oauthlib python3-openssl python3-packaging 2023-09-30T01:30:16.906 INFO:teuthology.orchestra.run.smithi088.stdout: python3-paste python3-pastedeploy python3-pastescript python3-pecan 2023-09-30T01:30:16.906 INFO:teuthology.orchestra.run.smithi088.stdout: python3-pluggy python3-prettytable python3-psutil python3-py python3-pyasn1 2023-09-30T01:30:16.906 INFO:teuthology.orchestra.run.smithi088.stdout: python3-pyasn1-modules python3-pyinotify python3-pytest python3-repoze.lru 2023-09-30T01:30:16.906 INFO:teuthology.orchestra.run.smithi088.stdout: python3-requests-oauthlib python3-routes python3-rsa python3-scipy 2023-09-30T01:30:16.906 INFO:teuthology.orchestra.run.smithi088.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2023-09-30T01:30:16.907 INFO:teuthology.orchestra.run.smithi088.stdout: python3-sklearn python3-sklearn-lib python3-soupsieve python3-tempita 2023-09-30T01:30:16.907 INFO:teuthology.orchestra.run.smithi088.stdout: python3-waitress python3-wcwidth python3-webencodings python3-webob 2023-09-30T01:30:16.907 INFO:teuthology.orchestra.run.smithi088.stdout: python3-websocket python3-webtest python3-werkzeug qttranslations5-l10n 2023-09-30T01:30:16.907 INFO:teuthology.orchestra.run.smithi088.stdout: socat xmlstarlet 2023-09-30T01:30:16.907 INFO:teuthology.orchestra.run.smithi088.stdout:Suggested packages: 2023-09-30T01:30:16.907 INFO:teuthology.orchestra.run.smithi088.stdout: python3-influxdb fonts-mathjax-extras fonts-stix libjs-mathjax-doc 2023-09-30T01:30:16.907 INFO:teuthology.orchestra.run.smithi088.stdout: python-attr-doc python-blinker-doc python-cryptography-doc 2023-09-30T01:30:16.907 INFO:teuthology.orchestra.run.smithi088.stdout: python3-cryptography-vectors python3-genshi python-jinja2-doc python3-crypto 2023-09-30T01:30:16.907 INFO:teuthology.orchestra.run.smithi088.stdout: python3-lxml-dbg python-lxml-doc python3-beaker python-mako-doc 2023-09-30T01:30:16.907 INFO:teuthology.orchestra.run.smithi088.stdout: python-openssl-doc python3-openssl-dbg httpd-wsgi libapache2-mod-python 2023-09-30T01:30:16.908 INFO:teuthology.orchestra.run.smithi088.stdout: libapache2-mod-scgi libjs-mochikit python-pecan-doc python-psutil-doc 2023-09-30T01:30:16.908 INFO:teuthology.orchestra.run.smithi088.stdout: subversion python-pyinotify-doc python-scipy-doc python3-dap 2023-09-30T01:30:16.908 INFO:teuthology.orchestra.run.smithi088.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2023-09-30T01:30:16.908 INFO:teuthology.orchestra.run.smithi088.stdout: python-webtest-doc python-werkzeug-doc python3-termcolor python3-watchdog 2023-09-30T01:30:16.908 INFO:teuthology.orchestra.run.smithi088.stdout:Recommended packages: 2023-09-30T01:30:16.908 INFO:teuthology.orchestra.run.smithi088.stdout: btrfs-tools 2023-09-30T01:30:16.991 INFO:teuthology.orchestra.run.smithi107.stdout:The following packages were automatically installed and are no longer required: 2023-09-30T01:30:16.992 INFO:teuthology.orchestra.run.smithi107.stdout: libboost-iostreams1.71.0 libboost-thread1.71.0 2023-09-30T01:30:16.993 INFO:teuthology.orchestra.run.smithi107.stdout:Use 'sudo apt autoremove' to remove them. 2023-09-30T01:30:16.993 INFO:teuthology.orchestra.run.smithi107.stdout:The following additional packages will be installed: 2023-09-30T01:30:16.993 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2023-09-30T01:30:16.993 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd fonts-mathjax jq 2023-09-30T01:30:16.993 INFO:teuthology.orchestra.run.smithi107.stdout: libdouble-conversion3 libjq1 libjs-mathjax libjs-sphinxdoc libjs-underscore 2023-09-30T01:30:16.993 INFO:teuthology.orchestra.run.smithi107.stdout: liblbfgsb0 liblttng-ust0 liblua5.3-0 liboath0 libonig5 libpcre2-16-0 2023-09-30T01:30:16.993 INFO:teuthology.orchestra.run.smithi107.stdout: libqt5core5a libqt5dbus5 libqt5network5 librabbitmq4 libradosstriper1 2023-09-30T01:30:16.994 INFO:teuthology.orchestra.run.smithi107.stdout: librdkafka1 librgw2 libsqlite3-mod-ceph libthrift-0.13.0 nvme-cli 2023-09-30T01:30:16.994 INFO:teuthology.orchestra.run.smithi107.stdout: python-natsort-doc python-pastedeploy-tpl python3-atomicwrites python3-attr 2023-09-30T01:30:16.994 INFO:teuthology.orchestra.run.smithi107.stdout: python3-bcrypt python3-blinker python3-bs4 python3-cachetools 2023-09-30T01:30:16.994 INFO:teuthology.orchestra.run.smithi107.stdout: python3-ceph-argparse python3-ceph-common python3-cherrypy3 2023-09-30T01:30:16.994 INFO:teuthology.orchestra.run.smithi107.stdout: python3-cryptography python3-decorator python3-google-auth python3-html5lib 2023-09-30T01:30:16.994 INFO:teuthology.orchestra.run.smithi107.stdout: python3-jinja2 python3-joblib python3-jwt python3-kubernetes 2023-09-30T01:30:16.994 INFO:teuthology.orchestra.run.smithi107.stdout: python3-logutils python3-lxml python3-mako python3-markupsafe 2023-09-30T01:30:16.994 INFO:teuthology.orchestra.run.smithi107.stdout: python3-natsort python3-oauthlib python3-openssl python3-packaging 2023-09-30T01:30:16.995 INFO:teuthology.orchestra.run.smithi107.stdout: python3-paste python3-pastedeploy python3-pastescript python3-pecan 2023-09-30T01:30:16.995 INFO:teuthology.orchestra.run.smithi107.stdout: python3-pluggy python3-prettytable python3-psutil python3-py python3-pyasn1 2023-09-30T01:30:16.995 INFO:teuthology.orchestra.run.smithi107.stdout: python3-pyasn1-modules python3-pyinotify python3-pytest python3-repoze.lru 2023-09-30T01:30:16.995 INFO:teuthology.orchestra.run.smithi107.stdout: python3-requests-oauthlib python3-routes python3-rsa python3-scipy 2023-09-30T01:30:16.995 INFO:teuthology.orchestra.run.smithi107.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2023-09-30T01:30:16.995 INFO:teuthology.orchestra.run.smithi107.stdout: python3-sklearn python3-sklearn-lib python3-soupsieve python3-tempita 2023-09-30T01:30:16.995 INFO:teuthology.orchestra.run.smithi107.stdout: python3-waitress python3-wcwidth python3-webencodings python3-webob 2023-09-30T01:30:16.995 INFO:teuthology.orchestra.run.smithi107.stdout: python3-websocket python3-webtest python3-werkzeug qttranslations5-l10n 2023-09-30T01:30:16.995 INFO:teuthology.orchestra.run.smithi107.stdout: socat xmlstarlet 2023-09-30T01:30:16.995 INFO:teuthology.orchestra.run.smithi107.stdout:Suggested packages: 2023-09-30T01:30:16.996 INFO:teuthology.orchestra.run.smithi107.stdout: python3-influxdb fonts-mathjax-extras fonts-stix libjs-mathjax-doc 2023-09-30T01:30:16.996 INFO:teuthology.orchestra.run.smithi107.stdout: python-attr-doc python-blinker-doc python-cryptography-doc 2023-09-30T01:30:16.996 INFO:teuthology.orchestra.run.smithi107.stdout: python3-cryptography-vectors python3-genshi python-jinja2-doc python3-crypto 2023-09-30T01:30:16.996 INFO:teuthology.orchestra.run.smithi107.stdout: python3-lxml-dbg python-lxml-doc python3-beaker python-mako-doc 2023-09-30T01:30:16.996 INFO:teuthology.orchestra.run.smithi107.stdout: python-openssl-doc python3-openssl-dbg httpd-wsgi libapache2-mod-python 2023-09-30T01:30:16.996 INFO:teuthology.orchestra.run.smithi107.stdout: libapache2-mod-scgi libjs-mochikit python-pecan-doc python-psutil-doc 2023-09-30T01:30:16.996 INFO:teuthology.orchestra.run.smithi107.stdout: subversion python-pyinotify-doc python-scipy-doc python3-dap 2023-09-30T01:30:16.996 INFO:teuthology.orchestra.run.smithi107.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2023-09-30T01:30:16.996 INFO:teuthology.orchestra.run.smithi107.stdout: python-webtest-doc python-werkzeug-doc python3-termcolor python3-watchdog 2023-09-30T01:30:16.996 INFO:teuthology.orchestra.run.smithi107.stdout:Recommended packages: 2023-09-30T01:30:16.996 INFO:teuthology.orchestra.run.smithi107.stdout: btrfs-tools 2023-09-30T01:30:17.024 INFO:teuthology.orchestra.run.smithi088.stdout:The following NEW packages will be installed: 2023-09-30T01:30:17.024 INFO:teuthology.orchestra.run.smithi088.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2023-09-30T01:30:17.024 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2023-09-30T01:30:17.024 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm 2023-09-30T01:30:17.024 INFO:teuthology.orchestra.run.smithi088.stdout: fonts-mathjax jq libcephfs-dev libcephfs2 libdouble-conversion3 libjq1 2023-09-30T01:30:17.025 INFO:teuthology.orchestra.run.smithi088.stdout: libjs-mathjax libjs-sphinxdoc libjs-underscore liblbfgsb0 liblttng-ust0 2023-09-30T01:30:17.025 INFO:teuthology.orchestra.run.smithi088.stdout: liblua5.3-0 liboath0 libonig5 libpcre2-16-0 libqt5core5a libqt5dbus5 2023-09-30T01:30:17.025 INFO:teuthology.orchestra.run.smithi088.stdout: libqt5network5 librabbitmq4 libradosstriper1 librdkafka1 librgw2 2023-09-30T01:30:17.026 INFO:teuthology.orchestra.run.smithi088.stdout: libsqlite3-mod-ceph libthrift-0.13.0 nvme-cli python-natsort-doc 2023-09-30T01:30:17.026 INFO:teuthology.orchestra.run.smithi088.stdout: python-pastedeploy-tpl python3-atomicwrites python3-attr python3-bcrypt 2023-09-30T01:30:17.026 INFO:teuthology.orchestra.run.smithi088.stdout: python3-blinker python3-bs4 python3-cachetools python3-ceph-argparse 2023-09-30T01:30:17.026 INFO:teuthology.orchestra.run.smithi088.stdout: python3-ceph-common python3-cephfs python3-cherrypy3 python3-cryptography 2023-09-30T01:30:17.026 INFO:teuthology.orchestra.run.smithi088.stdout: python3-decorator python3-google-auth python3-html5lib python3-jinja2 2023-09-30T01:30:17.026 INFO:teuthology.orchestra.run.smithi088.stdout: python3-joblib python3-jwt python3-kubernetes python3-logutils python3-lxml 2023-09-30T01:30:17.027 INFO:teuthology.orchestra.run.smithi088.stdout: python3-mako python3-markupsafe python3-natsort python3-oauthlib 2023-09-30T01:30:17.027 INFO:teuthology.orchestra.run.smithi088.stdout: python3-openssl python3-packaging python3-paste python3-pastedeploy 2023-09-30T01:30:17.027 INFO:teuthology.orchestra.run.smithi088.stdout: python3-pastescript python3-pecan python3-pluggy python3-prettytable 2023-09-30T01:30:17.027 INFO:teuthology.orchestra.run.smithi088.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2023-09-30T01:30:17.027 INFO:teuthology.orchestra.run.smithi088.stdout: python3-pyinotify python3-pytest python3-rados python3-rbd 2023-09-30T01:30:17.027 INFO:teuthology.orchestra.run.smithi088.stdout: python3-repoze.lru python3-requests-oauthlib python3-rgw python3-routes 2023-09-30T01:30:17.027 INFO:teuthology.orchestra.run.smithi088.stdout: python3-rsa python3-scipy python3-simplegeneric python3-simplejson 2023-09-30T01:30:17.027 INFO:teuthology.orchestra.run.smithi088.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib python3-soupsieve 2023-09-30T01:30:17.027 INFO:teuthology.orchestra.run.smithi088.stdout: python3-tempita python3-waitress python3-wcwidth python3-webencodings 2023-09-30T01:30:17.027 INFO:teuthology.orchestra.run.smithi088.stdout: python3-webob python3-websocket python3-webtest python3-werkzeug 2023-09-30T01:30:17.027 INFO:teuthology.orchestra.run.smithi088.stdout: qttranslations5-l10n radosgw rbd-fuse socat xmlstarlet 2023-09-30T01:30:17.027 INFO:teuthology.orchestra.run.smithi088.stdout:The following packages will be upgraded: 2023-09-30T01:30:17.029 INFO:teuthology.orchestra.run.smithi088.stdout: librados2 librbd1 2023-09-30T01:30:17.110 INFO:teuthology.orchestra.run.smithi107.stdout:The following NEW packages will be installed: 2023-09-30T01:30:17.110 INFO:teuthology.orchestra.run.smithi107.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2023-09-30T01:30:17.110 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2023-09-30T01:30:17.110 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm 2023-09-30T01:30:17.111 INFO:teuthology.orchestra.run.smithi107.stdout: fonts-mathjax jq libcephfs-dev libcephfs2 libdouble-conversion3 libjq1 2023-09-30T01:30:17.111 INFO:teuthology.orchestra.run.smithi107.stdout: libjs-mathjax libjs-sphinxdoc libjs-underscore liblbfgsb0 liblttng-ust0 2023-09-30T01:30:17.111 INFO:teuthology.orchestra.run.smithi107.stdout: liblua5.3-0 liboath0 libonig5 libpcre2-16-0 libqt5core5a libqt5dbus5 2023-09-30T01:30:17.111 INFO:teuthology.orchestra.run.smithi107.stdout: libqt5network5 librabbitmq4 libradosstriper1 librdkafka1 librgw2 2023-09-30T01:30:17.112 INFO:teuthology.orchestra.run.smithi107.stdout: libsqlite3-mod-ceph libthrift-0.13.0 nvme-cli python-natsort-doc 2023-09-30T01:30:17.112 INFO:teuthology.orchestra.run.smithi107.stdout: python-pastedeploy-tpl python3-atomicwrites python3-attr python3-bcrypt 2023-09-30T01:30:17.112 INFO:teuthology.orchestra.run.smithi107.stdout: python3-blinker python3-bs4 python3-cachetools python3-ceph-argparse 2023-09-30T01:30:17.112 INFO:teuthology.orchestra.run.smithi107.stdout: python3-ceph-common python3-cephfs python3-cherrypy3 python3-cryptography 2023-09-30T01:30:17.112 INFO:teuthology.orchestra.run.smithi107.stdout: python3-decorator python3-google-auth python3-html5lib python3-jinja2 2023-09-30T01:30:17.112 INFO:teuthology.orchestra.run.smithi107.stdout: python3-joblib python3-jwt python3-kubernetes python3-logutils python3-lxml 2023-09-30T01:30:17.112 INFO:teuthology.orchestra.run.smithi107.stdout: python3-mako python3-markupsafe python3-natsort python3-oauthlib 2023-09-30T01:30:17.112 INFO:teuthology.orchestra.run.smithi107.stdout: python3-openssl python3-packaging python3-paste python3-pastedeploy 2023-09-30T01:30:17.112 INFO:teuthology.orchestra.run.smithi107.stdout: python3-pastescript python3-pecan python3-pluggy python3-prettytable 2023-09-30T01:30:17.112 INFO:teuthology.orchestra.run.smithi107.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2023-09-30T01:30:17.112 INFO:teuthology.orchestra.run.smithi107.stdout: python3-pyinotify python3-pytest python3-rados python3-rbd 2023-09-30T01:30:17.112 INFO:teuthology.orchestra.run.smithi107.stdout: python3-repoze.lru python3-requests-oauthlib python3-rgw python3-routes 2023-09-30T01:30:17.113 INFO:teuthology.orchestra.run.smithi107.stdout: python3-rsa python3-scipy python3-simplegeneric python3-simplejson 2023-09-30T01:30:17.113 INFO:teuthology.orchestra.run.smithi107.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib python3-soupsieve 2023-09-30T01:30:17.113 INFO:teuthology.orchestra.run.smithi107.stdout: python3-tempita python3-waitress python3-wcwidth python3-webencodings 2023-09-30T01:30:17.113 INFO:teuthology.orchestra.run.smithi107.stdout: python3-webob python3-websocket python3-webtest python3-werkzeug 2023-09-30T01:30:17.113 INFO:teuthology.orchestra.run.smithi107.stdout: qttranslations5-l10n radosgw rbd-fuse socat xmlstarlet 2023-09-30T01:30:17.113 INFO:teuthology.orchestra.run.smithi107.stdout:The following packages will be upgraded: 2023-09-30T01:30:17.113 INFO:teuthology.orchestra.run.smithi107.stdout: librados2 librbd1 2023-09-30T01:30:17.219 INFO:teuthology.orchestra.run.smithi088.stdout:2 upgraded, 108 newly installed, 0 to remove and 245 not upgraded. 2023-09-30T01:30:17.219 INFO:teuthology.orchestra.run.smithi088.stdout:Need to get 209 MB of archives. 2023-09-30T01:30:17.219 INFO:teuthology.orchestra.run.smithi088.stdout:After this operation, 980 MB of additional disk space will be used. 2023-09-30T01:30:17.219 INFO:teuthology.orchestra.run.smithi088.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal/main amd64 liblttng-ust0 amd64 2.11.0-1 [160 kB] 2023-09-30T01:30:17.250 INFO:teuthology.orchestra.run.smithi088.stdout:Get:2 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/ubuntu/focal/flavors/default focal/main amd64 librbd1 amd64 17.2.6-1165-g93dce5e1-1focal [3,501 kB] 2023-09-30T01:30:17.303 INFO:teuthology.orchestra.run.smithi107.stdout:2 upgraded, 108 newly installed, 0 to remove and 245 not upgraded. 2023-09-30T01:30:17.303 INFO:teuthology.orchestra.run.smithi107.stdout:Need to get 209 MB of archives. 2023-09-30T01:30:17.303 INFO:teuthology.orchestra.run.smithi107.stdout:After this operation, 980 MB of additional disk space will be used. 2023-09-30T01:30:17.303 INFO:teuthology.orchestra.run.smithi107.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal/main amd64 liblttng-ust0 amd64 2.11.0-1 [160 kB] 2023-09-30T01:30:17.336 INFO:teuthology.orchestra.run.smithi107.stdout:Get:2 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/ubuntu/focal/flavors/default focal/main amd64 librbd1 amd64 17.2.6-1165-g93dce5e1-1focal [3,501 kB] 2023-09-30T01:30:17.567 INFO:teuthology.orchestra.run.smithi088.stdout:Get:3 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/ubuntu/focal/flavors/default focal/main amd64 librados2 amd64 17.2.6-1165-g93dce5e1-1focal [3,552 kB] 2023-09-30T01:30:17.615 INFO:teuthology.orchestra.run.smithi088.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-prettytable all 0.7.2-5 [20.1 kB] 2023-09-30T01:30:17.622 INFO:teuthology.orchestra.run.smithi088.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal/main amd64 liblua5.3-0 amd64 5.3.3-1.1ubuntu2 [116 kB] 2023-09-30T01:30:17.666 INFO:teuthology.orchestra.run.smithi088.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal/main amd64 librabbitmq4 amd64 0.10.0-1 [35.8 kB] 2023-09-30T01:30:17.680 INFO:teuthology.orchestra.run.smithi088.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal/universe amd64 librdkafka1 amd64 1.2.1-1ubuntu1 [410 kB] 2023-09-30T01:30:17.699 INFO:teuthology.orchestra.run.smithi107.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-prettytable all 0.7.2-5 [20.1 kB] 2023-09-30T01:30:17.706 INFO:teuthology.orchestra.run.smithi107.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal/main amd64 liblua5.3-0 amd64 5.3.3-1.1ubuntu2 [116 kB] 2023-09-30T01:30:17.724 INFO:teuthology.orchestra.run.smithi088.stdout:Get:8 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/ubuntu/focal/flavors/default focal/main amd64 libcephfs2 amd64 17.2.6-1165-g93dce5e1-1focal [758 kB] 2023-09-30T01:30:17.750 INFO:teuthology.orchestra.run.smithi107.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal/main amd64 librabbitmq4 amd64 0.10.0-1 [35.8 kB] 2023-09-30T01:30:17.763 INFO:teuthology.orchestra.run.smithi088.stdout:Get:9 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/ubuntu/focal/flavors/default focal/main amd64 python3-rados amd64 17.2.6-1165-g93dce5e1-1focal [414 kB] 2023-09-30T01:30:17.764 INFO:teuthology.orchestra.run.smithi107.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal/universe amd64 librdkafka1 amd64 1.2.1-1ubuntu1 [410 kB] 2023-09-30T01:30:17.776 INFO:teuthology.orchestra.run.smithi088.stdout:Get:10 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/ubuntu/focal/flavors/default focal/main amd64 python3-ceph-argparse all 17.2.6-1165-g93dce5e1-1focal [22.5 kB] 2023-09-30T01:30:17.777 INFO:teuthology.orchestra.run.smithi088.stdout:Get:11 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/ubuntu/focal/flavors/default focal/main amd64 python3-cephfs amd64 17.2.6-1165-g93dce5e1-1focal [200 kB] 2023-09-30T01:30:17.784 INFO:teuthology.orchestra.run.smithi088.stdout:Get:12 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/ubuntu/focal/flavors/default focal/main amd64 python3-ceph-common all 17.2.6-1165-g93dce5e1-1focal [47.1 kB] 2023-09-30T01:30:17.786 INFO:teuthology.orchestra.run.smithi088.stdout:Get:13 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/ubuntu/focal/flavors/default focal/main amd64 python3-rbd amd64 17.2.6-1165-g93dce5e1-1focal [385 kB] 2023-09-30T01:30:17.798 INFO:teuthology.orchestra.run.smithi088.stdout:Get:14 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/ubuntu/focal/flavors/default focal/main amd64 librgw2 amd64 17.2.6-1165-g93dce5e1-1focal [5,569 kB] 2023-09-30T01:30:17.806 INFO:teuthology.orchestra.run.smithi088.stdout:Get:15 http://archive.ubuntu.com/ubuntu focal/universe amd64 libdouble-conversion3 amd64 3.1.5-4ubuntu1 [37.9 kB] 2023-09-30T01:30:17.808 INFO:teuthology.orchestra.run.smithi107.stdout:Get:7 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/ubuntu/focal/flavors/default focal/main amd64 librados2 amd64 17.2.6-1165-g93dce5e1-1focal [3,552 kB] 2023-09-30T01:30:17.811 INFO:teuthology.orchestra.run.smithi088.stdout:Get:16 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libpcre2-16-0 amd64 10.34-7ubuntu0.1 [181 kB] 2023-09-30T01:30:17.841 INFO:teuthology.orchestra.run.smithi088.stdout:Get:17 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libqt5core5a amd64 5.12.8+dfsg-0ubuntu2.1 [2,006 kB] 2023-09-30T01:30:17.890 INFO:teuthology.orchestra.run.smithi107.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal/universe amd64 libdouble-conversion3 amd64 3.1.5-4ubuntu1 [37.9 kB] 2023-09-30T01:30:17.895 INFO:teuthology.orchestra.run.smithi107.stdout:Get:9 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libpcre2-16-0 amd64 10.34-7ubuntu0.1 [181 kB] 2023-09-30T01:30:17.925 INFO:teuthology.orchestra.run.smithi107.stdout:Get:10 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libqt5core5a amd64 5.12.8+dfsg-0ubuntu2.1 [2,006 kB] 2023-09-30T01:30:17.983 INFO:teuthology.orchestra.run.smithi088.stdout:Get:18 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libqt5dbus5 amd64 5.12.8+dfsg-0ubuntu2.1 [208 kB] 2023-09-30T01:30:17.991 INFO:teuthology.orchestra.run.smithi088.stdout:Get:19 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libqt5network5 amd64 5.12.8+dfsg-0ubuntu2.1 [673 kB] 2023-09-30T01:30:18.014 INFO:teuthology.orchestra.run.smithi088.stdout:Get:20 http://archive.ubuntu.com/ubuntu focal/universe amd64 libthrift-0.13.0 amd64 0.13.0-2build2 [249 kB] 2023-09-30T01:30:18.068 INFO:teuthology.orchestra.run.smithi107.stdout:Get:11 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libqt5dbus5 amd64 5.12.8+dfsg-0ubuntu2.1 [208 kB] 2023-09-30T01:30:18.076 INFO:teuthology.orchestra.run.smithi107.stdout:Get:12 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libqt5network5 amd64 5.12.8+dfsg-0ubuntu2.1 [673 kB] 2023-09-30T01:30:18.097 INFO:teuthology.orchestra.run.smithi088.stdout:Get:21 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/ubuntu/focal/flavors/default focal/main amd64 python3-rgw amd64 17.2.6-1165-g93dce5e1-1focal [116 kB] 2023-09-30T01:30:18.099 INFO:teuthology.orchestra.run.smithi107.stdout:Get:13 http://archive.ubuntu.com/ubuntu focal/universe amd64 libthrift-0.13.0 amd64 0.13.0-2build2 [249 kB] 2023-09-30T01:30:18.100 INFO:teuthology.orchestra.run.smithi088.stdout:Get:22 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/ubuntu/focal/flavors/default focal/main amd64 libradosstriper1 amd64 17.2.6-1165-g93dce5e1-1focal [480 kB] 2023-09-30T01:30:18.116 INFO:teuthology.orchestra.run.smithi088.stdout:Get:23 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/ubuntu/focal/flavors/default focal/main amd64 ceph-common amd64 17.2.6-1165-g93dce5e1-1focal [27.7 MB] 2023-09-30T01:30:18.152 INFO:teuthology.orchestra.run.smithi107.stdout:Get:14 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/ubuntu/focal/flavors/default focal/main amd64 libcephfs2 amd64 17.2.6-1165-g93dce5e1-1focal [758 kB] 2023-09-30T01:30:18.179 INFO:teuthology.orchestra.run.smithi107.stdout:Get:15 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/ubuntu/focal/flavors/default focal/main amd64 python3-rados amd64 17.2.6-1165-g93dce5e1-1focal [414 kB] 2023-09-30T01:30:18.184 INFO:teuthology.orchestra.run.smithi088.stdout:Get:24 http://archive.ubuntu.com/ubuntu focal/main amd64 liboath0 amd64 2.6.1-1.3 [44.7 kB] 2023-09-30T01:30:18.254 INFO:teuthology.orchestra.run.smithi107.stdout:Get:16 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/ubuntu/focal/flavors/default focal/main amd64 python3-ceph-argparse all 17.2.6-1165-g93dce5e1-1focal [22.5 kB] 2023-09-30T01:30:18.255 INFO:teuthology.orchestra.run.smithi107.stdout:Get:17 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/ubuntu/focal/flavors/default focal/main amd64 python3-cephfs amd64 17.2.6-1165-g93dce5e1-1focal [200 kB] 2023-09-30T01:30:18.270 INFO:teuthology.orchestra.run.smithi107.stdout:Get:18 http://archive.ubuntu.com/ubuntu focal/main amd64 liboath0 amd64 2.6.1-1.3 [44.7 kB] 2023-09-30T01:30:18.353 INFO:teuthology.orchestra.run.smithi107.stdout:Get:19 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/ubuntu/focal/flavors/default focal/main amd64 python3-ceph-common all 17.2.6-1165-g93dce5e1-1focal [47.1 kB] 2023-09-30T01:30:18.354 INFO:teuthology.orchestra.run.smithi107.stdout:Get:20 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/ubuntu/focal/flavors/default focal/main amd64 python3-rbd amd64 17.2.6-1165-g93dce5e1-1focal [385 kB] 2023-09-30T01:30:18.363 INFO:teuthology.orchestra.run.smithi107.stdout:Get:21 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/ubuntu/focal/flavors/default focal/main amd64 librgw2 amd64 17.2.6-1165-g93dce5e1-1focal [5,569 kB] 2023-09-30T01:30:18.433 INFO:teuthology.orchestra.run.smithi088.stdout:Get:25 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-cryptography amd64 2.8-3ubuntu0.1 [211 kB] 2023-09-30T01:30:18.519 INFO:teuthology.orchestra.run.smithi107.stdout:Get:22 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-cryptography amd64 2.8-3ubuntu0.1 [211 kB] 2023-09-30T01:30:18.627 INFO:teuthology.orchestra.run.smithi088.stdout:Get:26 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-openssl all 19.0.0-1build1 [43.3 kB] 2023-09-30T01:30:18.646 INFO:teuthology.orchestra.run.smithi088.stdout:Get:27 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-bcrypt amd64 3.1.7-2ubuntu1 [30.4 kB] 2023-09-30T01:30:18.660 INFO:teuthology.orchestra.run.smithi088.stdout:Get:28 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-cherrypy3 all 8.9.1-6 [160 kB] 2023-09-30T01:30:18.712 INFO:teuthology.orchestra.run.smithi088.stdout:Get:29 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-natsort all 7.0.1-1 [29.2 kB] 2023-09-30T01:30:18.712 INFO:teuthology.orchestra.run.smithi107.stdout:Get:23 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-openssl all 19.0.0-1build1 [43.3 kB] 2023-09-30T01:30:18.719 INFO:teuthology.orchestra.run.smithi088.stdout:Get:30 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-jwt all 1.7.1-2ubuntu2.1 [18.0 kB] 2023-09-30T01:30:18.722 INFO:teuthology.orchestra.run.smithi088.stdout:Get:31 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-logutils all 0.3.3-7 [17.3 kB] 2023-09-30T01:30:18.726 INFO:teuthology.orchestra.run.smithi088.stdout:Get:32 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-markupsafe amd64 1.1.0-1build2 [13.9 kB] 2023-09-30T01:30:18.729 INFO:teuthology.orchestra.run.smithi088.stdout:Get:33 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-mako all 1.1.0+ds1-1ubuntu2.1 [59.2 kB] 2023-09-30T01:30:18.732 INFO:teuthology.orchestra.run.smithi107.stdout:Get:24 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-bcrypt amd64 3.1.7-2ubuntu1 [30.4 kB] 2023-09-30T01:30:18.741 INFO:teuthology.orchestra.run.smithi088.stdout:Get:34 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2023-09-30T01:30:18.741 INFO:teuthology.orchestra.run.smithi107.stdout:Get:25 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/ubuntu/focal/flavors/default focal/main amd64 python3-rgw amd64 17.2.6-1165-g93dce5e1-1focal [116 kB] 2023-09-30T01:30:18.744 INFO:teuthology.orchestra.run.smithi107.stdout:Get:26 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/ubuntu/focal/flavors/default focal/main amd64 libradosstriper1 amd64 17.2.6-1165-g93dce5e1-1focal [480 kB] 2023-09-30T01:30:18.746 INFO:teuthology.orchestra.run.smithi107.stdout:Get:27 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-cherrypy3 all 8.9.1-6 [160 kB] 2023-09-30T01:30:18.768 INFO:teuthology.orchestra.run.smithi107.stdout:Get:28 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/ubuntu/focal/flavors/default focal/main amd64 ceph-common amd64 17.2.6-1165-g93dce5e1-1focal [27.7 MB] 2023-09-30T01:30:18.798 INFO:teuthology.orchestra.run.smithi107.stdout:Get:29 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-natsort all 7.0.1-1 [29.2 kB] 2023-09-30T01:30:18.805 INFO:teuthology.orchestra.run.smithi107.stdout:Get:30 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-jwt all 1.7.1-2ubuntu2.1 [18.0 kB] 2023-09-30T01:30:18.809 INFO:teuthology.orchestra.run.smithi107.stdout:Get:31 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-logutils all 0.3.3-7 [17.3 kB] 2023-09-30T01:30:18.813 INFO:teuthology.orchestra.run.smithi107.stdout:Get:32 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-markupsafe amd64 1.1.0-1build2 [13.9 kB] 2023-09-30T01:30:18.816 INFO:teuthology.orchestra.run.smithi107.stdout:Get:33 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-mako all 1.1.0+ds1-1ubuntu2.1 [59.2 kB] 2023-09-30T01:30:18.827 INFO:teuthology.orchestra.run.smithi107.stdout:Get:34 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2023-09-30T01:30:18.906 INFO:teuthology.orchestra.run.smithi088.stdout:Get:35 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-singledispatch all 3.4.0.3-2build1 [7,128 B] 2023-09-30T01:30:18.990 INFO:teuthology.orchestra.run.smithi088.stdout:Get:36 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-webob all 1:1.8.5-2 [84.4 kB] 2023-09-30T01:30:18.993 INFO:teuthology.orchestra.run.smithi107.stdout:Get:35 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-singledispatch all 3.4.0.3-2build1 [7,128 B] 2023-09-30T01:30:19.077 INFO:teuthology.orchestra.run.smithi107.stdout:Get:36 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-webob all 1:1.8.5-2 [84.4 kB] 2023-09-30T01:30:19.174 INFO:teuthology.orchestra.run.smithi088.stdout:Get:37 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-soupsieve all 1.9.5+dfsg-1 [29.1 kB] 2023-09-30T01:30:19.208 INFO:teuthology.orchestra.run.smithi088.stdout:Get:38 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-bs4 all 4.8.2-1 [83.0 kB] 2023-09-30T01:30:19.262 INFO:teuthology.orchestra.run.smithi107.stdout:Get:37 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-soupsieve all 1.9.5+dfsg-1 [29.1 kB] 2023-09-30T01:30:19.274 INFO:teuthology.orchestra.run.smithi088.stdout:Get:39 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-waitress all 1.4.1-1ubuntu0.1 [81.4 kB] 2023-09-30T01:30:19.296 INFO:teuthology.orchestra.run.smithi107.stdout:Get:38 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-bs4 all 4.8.2-1 [83.0 kB] 2023-09-30T01:30:19.313 INFO:teuthology.orchestra.run.smithi088.stdout:Get:40 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-tempita all 0.5.2-6 [14.8 kB] 2023-09-30T01:30:19.320 INFO:teuthology.orchestra.run.smithi088.stdout:Get:41 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-paste all 3.4.0+dfsg1-1 [456 kB] 2023-09-30T01:30:19.361 INFO:teuthology.orchestra.run.smithi107.stdout:Get:39 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-waitress all 1.4.1-1ubuntu0.1 [81.4 kB] 2023-09-30T01:30:19.401 INFO:teuthology.orchestra.run.smithi107.stdout:Get:40 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-tempita all 0.5.2-6 [14.8 kB] 2023-09-30T01:30:19.407 INFO:teuthology.orchestra.run.smithi107.stdout:Get:41 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-paste all 3.4.0+dfsg1-1 [456 kB] 2023-09-30T01:30:19.429 INFO:teuthology.orchestra.run.smithi088.stdout:Get:42 http://archive.ubuntu.com/ubuntu focal/main amd64 python-pastedeploy-tpl all 2.1.0-1 [5,016 B] 2023-09-30T01:30:19.430 INFO:teuthology.orchestra.run.smithi088.stdout:Get:43 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pastedeploy all 2.1.0-1 [26.5 kB] 2023-09-30T01:30:19.434 INFO:teuthology.orchestra.run.smithi088.stdout:Get:44 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-webtest all 2.0.32-2 [28.2 kB] 2023-09-30T01:30:19.438 INFO:teuthology.orchestra.run.smithi088.stdout:Get:45 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pecan all 1.3.3-3ubuntu1 [85.8 kB] 2023-09-30T01:30:19.517 INFO:teuthology.orchestra.run.smithi107.stdout:Get:42 http://archive.ubuntu.com/ubuntu focal/main amd64 python-pastedeploy-tpl all 2.1.0-1 [5,016 B] 2023-09-30T01:30:19.518 INFO:teuthology.orchestra.run.smithi107.stdout:Get:43 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pastedeploy all 2.1.0-1 [26.5 kB] 2023-09-30T01:30:19.522 INFO:teuthology.orchestra.run.smithi107.stdout:Get:44 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-webtest all 2.0.32-2 [28.2 kB] 2023-09-30T01:30:19.526 INFO:teuthology.orchestra.run.smithi107.stdout:Get:45 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pecan all 1.3.3-3ubuntu1 [85.8 kB] 2023-09-30T01:30:19.614 INFO:teuthology.orchestra.run.smithi088.stdout:Get:46 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-werkzeug all 0.16.1+dfsg1-2ubuntu0.1 [183 kB] 2023-09-30T01:30:19.702 INFO:teuthology.orchestra.run.smithi107.stdout:Get:46 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-werkzeug all 0.16.1+dfsg1-2ubuntu0.1 [183 kB] 2023-09-30T01:30:19.717 INFO:teuthology.orchestra.run.smithi088.stdout:Get:47 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/ubuntu/focal/flavors/default focal/main amd64 ceph-base amd64 17.2.6-1165-g93dce5e1-1focal [6,453 kB] 2023-09-30T01:30:19.993 INFO:teuthology.orchestra.run.smithi088.stdout:Get:48 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-modules-core all 17.2.6-1165-g93dce5e1-1focal [221 kB] 2023-09-30T01:30:20.005 INFO:teuthology.orchestra.run.smithi088.stdout:Get:49 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/ubuntu/focal/flavors/default focal/main amd64 libsqlite3-mod-ceph amd64 17.2.6-1165-g93dce5e1-1focal [153 kB] 2023-09-30T01:30:20.009 INFO:teuthology.orchestra.run.smithi088.stdout:Get:50 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr amd64 17.2.6-1165-g93dce5e1-1focal [1,589 kB] 2023-09-30T01:30:20.030 INFO:teuthology.orchestra.run.smithi088.stdout:Get:51 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-jinja2 all 2.10.1-2 [95.5 kB] 2023-09-30T01:30:20.061 INFO:teuthology.orchestra.run.smithi088.stdout:Get:52 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2023-09-30T01:30:20.067 INFO:teuthology.orchestra.run.smithi088.stdout:Get:53 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libjs-underscore all 1.9.1~dfsg-1ubuntu0.20.04.1 [99.5 kB] 2023-09-30T01:30:20.091 INFO:teuthology.orchestra.run.smithi088.stdout:Get:54 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/ubuntu/focal/flavors/default focal/main amd64 ceph-mon amd64 17.2.6-1165-g93dce5e1-1focal [7,845 kB] 2023-09-30T01:30:20.114 INFO:teuthology.orchestra.run.smithi088.stdout:Get:55 http://archive.ubuntu.com/ubuntu focal/main amd64 libjs-sphinxdoc all 1.8.5-7ubuntu3 [97.1 kB] 2023-09-30T01:30:20.118 INFO:teuthology.orchestra.run.smithi107.stdout:Get:47 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-jinja2 all 2.10.1-2 [95.5 kB] 2023-09-30T01:30:20.147 INFO:teuthology.orchestra.run.smithi088.stdout:Get:56 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-routes all 2.4.1-2 [88.1 kB] 2023-09-30T01:30:20.150 INFO:teuthology.orchestra.run.smithi107.stdout:Get:48 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2023-09-30T01:30:20.154 INFO:teuthology.orchestra.run.smithi107.stdout:Get:49 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libjs-underscore all 1.9.1~dfsg-1ubuntu0.20.04.1 [99.5 kB] 2023-09-30T01:30:20.174 INFO:teuthology.orchestra.run.smithi088.stdout:Get:57 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-decorator all 4.4.2-0ubuntu1 [10.3 kB] 2023-09-30T01:30:20.176 INFO:teuthology.orchestra.run.smithi088.stdout:Get:58 http://archive.ubuntu.com/ubuntu focal/universe amd64 liblbfgsb0 amd64 3.0+dfsg.3-7build1 [29.5 kB] 2023-09-30T01:30:20.184 INFO:teuthology.orchestra.run.smithi088.stdout:Get:59 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-scipy amd64 1.3.3-3build1 [11.1 MB] 2023-09-30T01:30:20.203 INFO:teuthology.orchestra.run.smithi107.stdout:Get:50 http://archive.ubuntu.com/ubuntu focal/main amd64 libjs-sphinxdoc all 1.8.5-7ubuntu3 [97.1 kB] 2023-09-30T01:30:20.235 INFO:teuthology.orchestra.run.smithi107.stdout:Get:51 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-routes all 2.4.1-2 [88.1 kB] 2023-09-30T01:30:20.263 INFO:teuthology.orchestra.run.smithi107.stdout:Get:52 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-decorator all 4.4.2-0ubuntu1 [10.3 kB] 2023-09-30T01:30:20.266 INFO:teuthology.orchestra.run.smithi107.stdout:Get:53 http://archive.ubuntu.com/ubuntu focal/universe amd64 liblbfgsb0 amd64 3.0+dfsg.3-7build1 [29.5 kB] 2023-09-30T01:30:20.274 INFO:teuthology.orchestra.run.smithi107.stdout:Get:54 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-scipy amd64 1.3.3-3build1 [11.1 MB] 2023-09-30T01:30:20.524 INFO:teuthology.orchestra.run.smithi088.stdout:Get:60 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/ubuntu/focal/flavors/default focal/main amd64 ceph-osd amd64 17.2.6-1165-g93dce5e1-1focal [28.7 MB] 2023-09-30T01:30:20.627 INFO:teuthology.orchestra.run.smithi088.stdout:Get:61 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-sklearn-lib amd64 0.22.2.post1+dfsg-5 [2,358 kB] 2023-09-30T01:30:20.699 INFO:teuthology.orchestra.run.smithi088.stdout:Get:62 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-joblib all 0.14.0-3 [192 kB] 2023-09-30T01:30:20.736 INFO:teuthology.orchestra.run.smithi107.stdout:Get:55 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-sklearn-lib amd64 0.22.2.post1+dfsg-5 [2,358 kB] 2023-09-30T01:30:20.786 INFO:teuthology.orchestra.run.smithi107.stdout:Get:56 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-joblib all 0.14.0-3 [192 kB] 2023-09-30T01:30:20.864 INFO:teuthology.orchestra.run.smithi088.stdout:Get:63 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-sklearn all 0.22.2.post1+dfsg-5 [1,773 kB] 2023-09-30T01:30:20.953 INFO:teuthology.orchestra.run.smithi107.stdout:Get:57 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-sklearn all 0.22.2.post1+dfsg-5 [1,773 kB] 2023-09-30T01:30:21.213 INFO:teuthology.orchestra.run.smithi107.stdout:Get:58 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/ubuntu/focal/flavors/default focal/main amd64 ceph-base amd64 17.2.6-1165-g93dce5e1-1focal [6,453 kB] 2023-09-30T01:30:21.552 INFO:teuthology.orchestra.run.smithi088.stdout:Get:64 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-cachetools all 4.0.0-1 [9,820 B] 2023-09-30T01:30:21.552 INFO:teuthology.orchestra.run.smithi088.stdout:Get:65 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyasn1 all 0.4.2-3build1 [46.7 kB] 2023-09-30T01:30:21.554 INFO:teuthology.orchestra.run.smithi088.stdout:Get:66 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyasn1-modules all 0.2.1-0.2build1 [32.9 kB] 2023-09-30T01:30:21.555 INFO:teuthology.orchestra.run.smithi088.stdout:Get:67 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-rsa all 4.0-3 [24.7 kB] 2023-09-30T01:30:21.556 INFO:teuthology.orchestra.run.smithi088.stdout:Get:68 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-google-auth all 1.5.1-2 [35.6 kB] 2023-09-30T01:30:21.557 INFO:teuthology.orchestra.run.smithi088.stdout:Get:69 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-blinker all 1.4+dfsg1-0.3ubuntu1 [13.2 kB] 2023-09-30T01:30:21.558 INFO:teuthology.orchestra.run.smithi088.stdout:Get:70 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-oauthlib all 3.1.0-1ubuntu2 [84.8 kB] 2023-09-30T01:30:21.561 INFO:teuthology.orchestra.run.smithi088.stdout:Get:71 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-requests-oauthlib all 1.0.0-1.1build1 [16.8 kB] 2023-09-30T01:30:21.561 INFO:teuthology.orchestra.run.smithi088.stdout:Get:72 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-websocket all 0.53.0-2ubuntu1 [32.3 kB] 2023-09-30T01:30:21.632 INFO:teuthology.orchestra.run.smithi088.stdout:Get:73 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-kubernetes all 7.0.0~a1-2 [271 kB] 2023-09-30T01:30:21.641 INFO:teuthology.orchestra.run.smithi107.stdout:Get:59 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-cachetools all 4.0.0-1 [9,820 B] 2023-09-30T01:30:21.641 INFO:teuthology.orchestra.run.smithi107.stdout:Get:60 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyasn1 all 0.4.2-3build1 [46.7 kB] 2023-09-30T01:30:21.643 INFO:teuthology.orchestra.run.smithi107.stdout:Get:61 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyasn1-modules all 0.2.1-0.2build1 [32.9 kB] 2023-09-30T01:30:21.644 INFO:teuthology.orchestra.run.smithi107.stdout:Get:62 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-rsa all 4.0-3 [24.7 kB] 2023-09-30T01:30:21.645 INFO:teuthology.orchestra.run.smithi107.stdout:Get:63 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-google-auth all 1.5.1-2 [35.6 kB] 2023-09-30T01:30:21.646 INFO:teuthology.orchestra.run.smithi107.stdout:Get:64 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-blinker all 1.4+dfsg1-0.3ubuntu1 [13.2 kB] 2023-09-30T01:30:21.647 INFO:teuthology.orchestra.run.smithi107.stdout:Get:65 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-oauthlib all 3.1.0-1ubuntu2 [84.8 kB] 2023-09-30T01:30:21.649 INFO:teuthology.orchestra.run.smithi107.stdout:Get:66 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-requests-oauthlib all 1.0.0-1.1build1 [16.8 kB] 2023-09-30T01:30:21.650 INFO:teuthology.orchestra.run.smithi107.stdout:Get:67 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-websocket all 0.53.0-2ubuntu1 [32.3 kB] 2023-09-30T01:30:21.671 INFO:teuthology.orchestra.run.smithi107.stdout:Get:68 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-modules-core all 17.2.6-1165-g93dce5e1-1focal [221 kB] 2023-09-30T01:30:21.677 INFO:teuthology.orchestra.run.smithi107.stdout:Get:69 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/ubuntu/focal/flavors/default focal/main amd64 libsqlite3-mod-ceph amd64 17.2.6-1165-g93dce5e1-1focal [153 kB] 2023-09-30T01:30:21.702 INFO:teuthology.orchestra.run.smithi107.stdout:Get:70 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr amd64 17.2.6-1165-g93dce5e1-1focal [1,589 kB] 2023-09-30T01:30:21.723 INFO:teuthology.orchestra.run.smithi107.stdout:Get:71 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-kubernetes all 7.0.0~a1-2 [271 kB] 2023-09-30T01:30:21.805 INFO:teuthology.orchestra.run.smithi088.stdout:Get:74 http://archive.ubuntu.com/ubuntu focal/universe amd64 libonig5 amd64 6.9.4-1 [142 kB] 2023-09-30T01:30:21.843 INFO:teuthology.orchestra.run.smithi107.stdout:Get:72 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/ubuntu/focal/flavors/default focal/main amd64 ceph-mon amd64 17.2.6-1165-g93dce5e1-1focal [7,845 kB] 2023-09-30T01:30:21.896 INFO:teuthology.orchestra.run.smithi107.stdout:Get:73 http://archive.ubuntu.com/ubuntu focal/universe amd64 libonig5 amd64 6.9.4-1 [142 kB] 2023-09-30T01:30:22.168 INFO:teuthology.orchestra.run.smithi088.stdout:Get:75 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/ubuntu/focal/flavors/default focal/main amd64 ceph amd64 17.2.6-1165-g93dce5e1-1focal [3,928 B] 2023-09-30T01:30:22.169 INFO:teuthology.orchestra.run.smithi088.stdout:Get:76 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/ubuntu/focal/flavors/default focal/main amd64 ceph-fuse amd64 17.2.6-1165-g93dce5e1-1focal [868 kB] 2023-09-30T01:30:22.181 INFO:teuthology.orchestra.run.smithi088.stdout:Get:77 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/ubuntu/focal/flavors/default focal/main amd64 ceph-mds amd64 17.2.6-1165-g93dce5e1-1focal [2,404 kB] 2023-09-30T01:30:22.186 INFO:teuthology.orchestra.run.smithi088.stdout:Get:78 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libjq1 amd64 1.6-1ubuntu0.20.04.1 [121 kB] 2023-09-30T01:30:22.199 INFO:teuthology.orchestra.run.smithi088.stdout:Get:79 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/ubuntu/focal/flavors/default focal/main amd64 cephadm amd64 17.2.6-1165-g93dce5e1-1focal [78.9 kB] 2023-09-30T01:30:22.200 INFO:teuthology.orchestra.run.smithi088.stdout:Get:80 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-cephadm all 17.2.6-1165-g93dce5e1-1focal [117 kB] 2023-09-30T01:30:22.221 INFO:teuthology.orchestra.run.smithi088.stdout:Get:81 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-dashboard all 17.2.6-1165-g93dce5e1-1focal [1,760 kB] 2023-09-30T01:30:22.238 INFO:teuthology.orchestra.run.smithi088.stdout:Get:82 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 jq amd64 1.6-1ubuntu0.20.04.1 [50.2 kB] 2023-09-30T01:30:22.274 INFO:teuthology.orchestra.run.smithi088.stdout:Get:83 http://archive.ubuntu.com/ubuntu focal/main amd64 socat amd64 1.7.3.3-2 [323 kB] 2023-09-30T01:30:22.277 INFO:teuthology.orchestra.run.smithi107.stdout:Get:74 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libjq1 amd64 1.6-1ubuntu0.20.04.1 [121 kB] 2023-09-30T01:30:22.313 INFO:teuthology.orchestra.run.smithi088.stdout:Get:84 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-diskprediction-local all 17.2.6-1165-g93dce5e1-1focal [7,861 kB] 2023-09-30T01:30:22.329 INFO:teuthology.orchestra.run.smithi107.stdout:Get:75 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 jq amd64 1.6-1ubuntu0.20.04.1 [50.2 kB] 2023-09-30T01:30:22.365 INFO:teuthology.orchestra.run.smithi107.stdout:Get:76 http://archive.ubuntu.com/ubuntu focal/main amd64 socat amd64 1.7.3.3-2 [323 kB] 2023-09-30T01:30:22.374 INFO:teuthology.orchestra.run.smithi088.stdout:Get:85 http://archive.ubuntu.com/ubuntu focal/universe amd64 xmlstarlet amd64 1.6.1-2build1 [267 kB] 2023-09-30T01:30:22.420 INFO:teuthology.orchestra.run.smithi088.stdout:Get:86 http://archive.ubuntu.com/ubuntu focal/universe amd64 fonts-mathjax all 2.7.4+dfsg-1 [2,208 kB] 2023-09-30T01:30:22.465 INFO:teuthology.orchestra.run.smithi107.stdout:Get:77 http://archive.ubuntu.com/ubuntu focal/universe amd64 xmlstarlet amd64 1.6.1-2build1 [267 kB] 2023-09-30T01:30:22.482 INFO:teuthology.orchestra.run.smithi107.stdout:Get:78 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/ubuntu/focal/flavors/default focal/main amd64 ceph-osd amd64 17.2.6-1165-g93dce5e1-1focal [28.7 MB] 2023-09-30T01:30:22.513 INFO:teuthology.orchestra.run.smithi107.stdout:Get:79 http://archive.ubuntu.com/ubuntu focal/universe amd64 fonts-mathjax all 2.7.4+dfsg-1 [2,208 kB] 2023-09-30T01:30:22.574 INFO:teuthology.orchestra.run.smithi088.stdout:Get:87 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 nvme-cli amd64 1.9-1ubuntu0.1 [283 kB] 2023-09-30T01:30:22.586 INFO:teuthology.orchestra.run.smithi088.stdout:Get:88 http://archive.ubuntu.com/ubuntu focal/universe amd64 libjs-mathjax all 2.7.4+dfsg-1 [5,654 kB] 2023-09-30T01:30:22.667 INFO:teuthology.orchestra.run.smithi107.stdout:Get:80 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 nvme-cli amd64 1.9-1ubuntu0.1 [283 kB] 2023-09-30T01:30:22.677 INFO:teuthology.orchestra.run.smithi107.stdout:Get:81 http://archive.ubuntu.com/ubuntu focal/universe amd64 libjs-mathjax all 2.7.4+dfsg-1 [5,654 kB] 2023-09-30T01:30:22.721 INFO:teuthology.orchestra.run.smithi088.stdout:Get:89 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-k8sevents all 17.2.6-1165-g93dce5e1-1focal [4,048 B] 2023-09-30T01:30:22.733 INFO:teuthology.orchestra.run.smithi088.stdout:Get:90 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/ubuntu/focal/flavors/default focal/main amd64 ceph-test amd64 17.2.6-1165-g93dce5e1-1focal [56.2 MB] 2023-09-30T01:30:22.813 INFO:teuthology.orchestra.run.smithi088.stdout:Get:91 http://archive.ubuntu.com/ubuntu focal/universe amd64 python-natsort-doc all 7.0.1-1 [103 kB] 2023-09-30T01:30:22.815 INFO:teuthology.orchestra.run.smithi088.stdout:Get:92 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-atomicwrites all 1.1.5-2build1 [7,016 B] 2023-09-30T01:30:22.815 INFO:teuthology.orchestra.run.smithi088.stdout:Get:93 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-attr all 19.3.0-2 [33.9 kB] 2023-09-30T01:30:22.825 INFO:teuthology.orchestra.run.smithi107.stdout:Get:82 http://archive.ubuntu.com/ubuntu focal/universe amd64 python-natsort-doc all 7.0.1-1 [103 kB] 2023-09-30T01:30:22.827 INFO:teuthology.orchestra.run.smithi107.stdout:Get:83 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-atomicwrites all 1.1.5-2build1 [7,016 B] 2023-09-30T01:30:22.827 INFO:teuthology.orchestra.run.smithi107.stdout:Get:84 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-attr all 19.3.0-2 [33.9 kB] 2023-09-30T01:30:22.981 INFO:teuthology.orchestra.run.smithi088.stdout:Get:94 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-webencodings all 0.5.1-1ubuntu1 [11.0 kB] 2023-09-30T01:30:22.992 INFO:teuthology.orchestra.run.smithi107.stdout:Get:85 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-webencodings all 0.5.1-1ubuntu1 [11.0 kB] 2023-09-30T01:30:23.064 INFO:teuthology.orchestra.run.smithi088.stdout:Get:95 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-html5lib all 1.0.1-2 [84.3 kB] 2023-09-30T01:30:23.075 INFO:teuthology.orchestra.run.smithi107.stdout:Get:86 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-html5lib all 1.0.1-2 [84.3 kB] 2023-09-30T01:30:23.246 INFO:teuthology.orchestra.run.smithi088.stdout:Get:96 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-lxml amd64 4.5.0-1ubuntu0.5 [1,384 kB] 2023-09-30T01:30:23.257 INFO:teuthology.orchestra.run.smithi107.stdout:Get:87 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-lxml amd64 4.5.0-1ubuntu0.5 [1,384 kB] 2023-09-30T01:30:23.576 INFO:teuthology.orchestra.run.smithi088.stdout:Get:97 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-packaging all 20.3-1 [26.8 kB] 2023-09-30T01:30:23.577 INFO:teuthology.orchestra.run.smithi088.stdout:Get:98 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2023-09-30T01:30:23.582 INFO:teuthology.orchestra.run.smithi088.stdout:Get:99 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-pluggy all 0.13.0-2 [18.4 kB] 2023-09-30T01:30:23.583 INFO:teuthology.orchestra.run.smithi088.stdout:Get:100 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-psutil amd64 5.5.1-1ubuntu4 [141 kB] 2023-09-30T01:30:23.587 INFO:teuthology.orchestra.run.smithi107.stdout:Get:88 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-packaging all 20.3-1 [26.8 kB] 2023-09-30T01:30:23.589 INFO:teuthology.orchestra.run.smithi107.stdout:Get:89 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2023-09-30T01:30:23.593 INFO:teuthology.orchestra.run.smithi107.stdout:Get:90 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-pluggy all 0.13.0-2 [18.4 kB] 2023-09-30T01:30:23.593 INFO:teuthology.orchestra.run.smithi088.stdout:Get:101 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 python3-py all 1.8.1-1ubuntu0.1 [65.6 kB] 2023-09-30T01:30:23.594 INFO:teuthology.orchestra.run.smithi107.stdout:Get:91 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-psutil amd64 5.5.1-1ubuntu4 [141 kB] 2023-09-30T01:30:23.598 INFO:teuthology.orchestra.run.smithi088.stdout:Get:102 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyinotify all 0.9.6-1.2ubuntu1 [24.8 kB] 2023-09-30T01:30:23.599 INFO:teuthology.orchestra.run.smithi088.stdout:Get:103 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-wcwidth all 0.1.8+dfsg1-3 [17.4 kB] 2023-09-30T01:30:23.600 INFO:teuthology.orchestra.run.smithi088.stdout:Get:104 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-pytest all 4.6.9-1 [178 kB] 2023-09-30T01:30:23.604 INFO:teuthology.orchestra.run.smithi107.stdout:Get:92 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 python3-py all 1.8.1-1ubuntu0.1 [65.6 kB] 2023-09-30T01:30:23.608 INFO:teuthology.orchestra.run.smithi107.stdout:Get:93 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyinotify all 0.9.6-1.2ubuntu1 [24.8 kB] 2023-09-30T01:30:23.610 INFO:teuthology.orchestra.run.smithi107.stdout:Get:94 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-wcwidth all 0.1.8+dfsg1-3 [17.4 kB] 2023-09-30T01:30:23.611 INFO:teuthology.orchestra.run.smithi107.stdout:Get:95 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-pytest all 4.6.9-1 [178 kB] 2023-09-30T01:30:23.775 INFO:teuthology.orchestra.run.smithi088.stdout:Get:105 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-simplejson amd64 3.16.0-2ubuntu2 [50.8 kB] 2023-09-30T01:30:23.786 INFO:teuthology.orchestra.run.smithi107.stdout:Get:96 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-simplejson amd64 3.16.0-2ubuntu2 [50.8 kB] 2023-09-30T01:30:24.033 INFO:teuthology.orchestra.run.smithi088.stdout:Get:106 http://archive.ubuntu.com/ubuntu focal/universe amd64 qttranslations5-l10n all 5.12.8-0ubuntu1 [1,486 kB] 2023-09-30T01:30:24.041 INFO:teuthology.orchestra.run.smithi107.stdout:Get:97 http://archive.ubuntu.com/ubuntu focal/universe amd64 qttranslations5-l10n all 5.12.8-0ubuntu1 [1,486 kB] 2023-09-30T01:30:25.489 INFO:teuthology.orchestra.run.smithi107.stdout:Get:98 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/ubuntu/focal/flavors/default focal/main amd64 ceph amd64 17.2.6-1165-g93dce5e1-1focal [3,928 B] 2023-09-30T01:30:25.489 INFO:teuthology.orchestra.run.smithi107.stdout:Get:99 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/ubuntu/focal/flavors/default focal/main amd64 ceph-fuse amd64 17.2.6-1165-g93dce5e1-1focal [868 kB] 2023-09-30T01:30:25.504 INFO:teuthology.orchestra.run.smithi107.stdout:Get:100 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/ubuntu/focal/flavors/default focal/main amd64 ceph-mds amd64 17.2.6-1165-g93dce5e1-1focal [2,404 kB] 2023-09-30T01:30:25.651 INFO:teuthology.orchestra.run.smithi107.stdout:Get:101 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/ubuntu/focal/flavors/default focal/main amd64 cephadm amd64 17.2.6-1165-g93dce5e1-1focal [78.9 kB] 2023-09-30T01:30:25.653 INFO:teuthology.orchestra.run.smithi107.stdout:Get:102 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-cephadm all 17.2.6-1165-g93dce5e1-1focal [117 kB] 2023-09-30T01:30:25.656 INFO:teuthology.orchestra.run.smithi107.stdout:Get:103 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-dashboard all 17.2.6-1165-g93dce5e1-1focal [1,760 kB] 2023-09-30T01:30:25.738 INFO:teuthology.orchestra.run.smithi107.stdout:Get:104 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-diskprediction-local all 17.2.6-1165-g93dce5e1-1focal [7,861 kB] 2023-09-30T01:30:25.762 INFO:teuthology.orchestra.run.smithi088.stdout:Get:107 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/ubuntu/focal/flavors/default focal/main amd64 ceph-volume all 17.2.6-1165-g93dce5e1-1focal [229 kB] 2023-09-30T01:30:25.767 INFO:teuthology.orchestra.run.smithi088.stdout:Get:108 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/ubuntu/focal/flavors/default focal/main amd64 libcephfs-dev amd64 17.2.6-1165-g93dce5e1-1focal [21.9 kB] 2023-09-30T01:30:25.767 INFO:teuthology.orchestra.run.smithi088.stdout:Get:109 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/ubuntu/focal/flavors/default focal/main amd64 radosgw amd64 17.2.6-1165-g93dce5e1-1focal [17.0 MB] 2023-09-30T01:30:26.440 INFO:teuthology.orchestra.run.smithi107.stdout:Get:105 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-k8sevents all 17.2.6-1165-g93dce5e1-1focal [4,048 B] 2023-09-30T01:30:26.441 INFO:teuthology.orchestra.run.smithi107.stdout:Get:106 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/ubuntu/focal/flavors/default focal/main amd64 ceph-test amd64 17.2.6-1165-g93dce5e1-1focal [56.2 MB] 2023-09-30T01:30:26.457 INFO:teuthology.orchestra.run.smithi088.stdout:Get:110 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/ubuntu/focal/flavors/default focal/main amd64 rbd-fuse amd64 17.2.6-1165-g93dce5e1-1focal [79.5 kB] 2023-09-30T01:30:27.086 INFO:teuthology.orchestra.run.smithi088.stdout:Fetched 209 MB in 9s (22.2 MB/s) 2023-09-30T01:30:27.266 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package liblttng-ust0:amd64. 2023-09-30T01:30:27.322 INFO:teuthology.orchestra.run.smithi088.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 141758 files and directories currently installed.) 2023-09-30T01:30:27.326 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../000-liblttng-ust0_2.11.0-1_amd64.deb ... 2023-09-30T01:30:27.423 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking liblttng-ust0:amd64 (2.11.0-1) ... 2023-09-30T01:30:27.789 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../001-librbd1_17.2.6-1165-g93dce5e1-1focal_amd64.deb ... 2023-09-30T01:30:27.938 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking librbd1 (17.2.6-1165-g93dce5e1-1focal) over (15.2.17-0ubuntu0.20.04.4) ... 2023-09-30T01:30:28.617 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../002-librados2_17.2.6-1165-g93dce5e1-1focal_amd64.deb ... 2023-09-30T01:30:28.743 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking librados2 (17.2.6-1165-g93dce5e1-1focal) over (15.2.17-0ubuntu0.20.04.4) ... 2023-09-30T01:30:29.036 INFO:teuthology.orchestra.run.smithi107.stdout:Get:107 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/ubuntu/focal/flavors/default focal/main amd64 ceph-volume all 17.2.6-1165-g93dce5e1-1focal [229 kB] 2023-09-30T01:30:29.040 INFO:teuthology.orchestra.run.smithi107.stdout:Get:108 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/ubuntu/focal/flavors/default focal/main amd64 libcephfs-dev amd64 17.2.6-1165-g93dce5e1-1focal [21.9 kB] 2023-09-30T01:30:29.041 INFO:teuthology.orchestra.run.smithi107.stdout:Get:109 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/ubuntu/focal/flavors/default focal/main amd64 radosgw amd64 17.2.6-1165-g93dce5e1-1focal [17.0 MB] 2023-09-30T01:30:29.447 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package libcephfs2. 2023-09-30T01:30:29.465 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../003-libcephfs2_17.2.6-1165-g93dce5e1-1focal_amd64.deb ... 2023-09-30T01:30:29.505 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking libcephfs2 (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T01:30:29.704 INFO:teuthology.orchestra.run.smithi107.stdout:Get:110 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/ubuntu/focal/flavors/default focal/main amd64 rbd-fuse amd64 17.2.6-1165-g93dce5e1-1focal [79.5 kB] 2023-09-30T01:30:29.858 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package python3-rados. 2023-09-30T01:30:29.874 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../004-python3-rados_17.2.6-1165-g93dce5e1-1focal_amd64.deb ... 2023-09-30T01:30:29.941 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking python3-rados (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T01:30:30.319 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package python3-ceph-argparse. 2023-09-30T01:30:30.335 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../005-python3-ceph-argparse_17.2.6-1165-g93dce5e1-1focal_all.deb ... 2023-09-30T01:30:30.344 INFO:teuthology.orchestra.run.smithi107.stdout:Fetched 209 MB in 13s (16.6 MB/s) 2023-09-30T01:30:30.385 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking python3-ceph-argparse (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T01:30:30.520 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package liblttng-ust0:amd64. 2023-09-30T01:30:30.576 INFO:teuthology.orchestra.run.smithi107.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 141758 files and directories currently installed.) 2023-09-30T01:30:30.581 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../000-liblttng-ust0_2.11.0-1_amd64.deb ... 2023-09-30T01:30:30.661 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking liblttng-ust0:amd64 (2.11.0-1) ... 2023-09-30T01:30:30.662 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package python3-cephfs. 2023-09-30T01:30:30.678 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../006-python3-cephfs_17.2.6-1165-g93dce5e1-1focal_amd64.deb ... 2023-09-30T01:30:30.721 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking python3-cephfs (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T01:30:30.985 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../001-librbd1_17.2.6-1165-g93dce5e1-1focal_amd64.deb ... 2023-09-30T01:30:31.082 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package python3-ceph-common. 2023-09-30T01:30:31.098 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../007-python3-ceph-common_17.2.6-1165-g93dce5e1-1focal_all.deb ... 2023-09-30T01:30:31.100 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking librbd1 (17.2.6-1165-g93dce5e1-1focal) over (15.2.17-0ubuntu0.20.04.4) ... 2023-09-30T01:30:31.140 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking python3-ceph-common (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T01:30:31.460 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package python3-prettytable. 2023-09-30T01:30:31.480 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../008-python3-prettytable_0.7.2-5_all.deb ... 2023-09-30T01:30:31.518 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking python3-prettytable (0.7.2-5) ... 2023-09-30T01:30:31.796 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package python3-rbd. 2023-09-30T01:30:31.813 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../009-python3-rbd_17.2.6-1165-g93dce5e1-1focal_amd64.deb ... 2023-09-30T01:30:31.854 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking python3-rbd (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T01:30:31.900 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../002-librados2_17.2.6-1165-g93dce5e1-1focal_amd64.deb ... 2023-09-30T01:30:32.055 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking librados2 (17.2.6-1165-g93dce5e1-1focal) over (15.2.17-0ubuntu0.20.04.4) ... 2023-09-30T01:30:32.490 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package liblua5.3-0:amd64. 2023-09-30T01:30:32.509 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../010-liblua5.3-0_5.3.3-1.1ubuntu2_amd64.deb ... 2023-09-30T01:30:32.557 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking liblua5.3-0:amd64 (5.3.3-1.1ubuntu2) ... 2023-09-30T01:30:32.703 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package libcephfs2. 2023-09-30T01:30:32.721 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../003-libcephfs2_17.2.6-1165-g93dce5e1-1focal_amd64.deb ... 2023-09-30T01:30:32.761 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking libcephfs2 (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T01:30:32.910 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package librabbitmq4:amd64. 2023-09-30T01:30:32.928 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../011-librabbitmq4_0.10.0-1_amd64.deb ... 2023-09-30T01:30:32.968 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking librabbitmq4:amd64 (0.10.0-1) ... 2023-09-30T01:30:33.097 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package python3-rados. 2023-09-30T01:30:33.115 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../004-python3-rados_17.2.6-1165-g93dce5e1-1focal_amd64.deb ... 2023-09-30T01:30:33.155 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking python3-rados (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T01:30:33.330 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package librdkafka1:amd64. 2023-09-30T01:30:33.346 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../012-librdkafka1_1.2.1-1ubuntu1_amd64.deb ... 2023-09-30T01:30:33.388 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking librdkafka1:amd64 (1.2.1-1ubuntu1) ... 2023-09-30T01:30:33.474 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package python3-ceph-argparse. 2023-09-30T01:30:33.492 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../005-python3-ceph-argparse_17.2.6-1165-g93dce5e1-1focal_all.deb ... 2023-09-30T01:30:33.524 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking python3-ceph-argparse (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T01:30:33.776 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package python3-cephfs. 2023-09-30T01:30:33.791 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2023-09-30T01:30:33.796 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../006-python3-cephfs_17.2.6-1165-g93dce5e1-1focal_amd64.deb ... 2023-09-30T01:30:33.811 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../013-libdouble-conversion3_3.1.5-4ubuntu1_amd64.deb ... 2023-09-30T01:30:33.834 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking python3-cephfs (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T01:30:33.873 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking libdouble-conversion3:amd64 (3.1.5-4ubuntu1) ... 2023-09-30T01:30:34.162 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package python3-ceph-common. 2023-09-30T01:30:34.178 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../007-python3-ceph-common_17.2.6-1165-g93dce5e1-1focal_all.deb ... 2023-09-30T01:30:34.237 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2023-09-30T01:30:34.245 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking python3-ceph-common (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T01:30:34.253 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../014-libpcre2-16-0_10.34-7ubuntu0.1_amd64.deb ... 2023-09-30T01:30:34.290 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking libpcre2-16-0:amd64 (10.34-7ubuntu0.1) ... 2023-09-30T01:30:34.540 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package python3-prettytable. 2023-09-30T01:30:34.560 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../008-python3-prettytable_0.7.2-5_all.deb ... 2023-09-30T01:30:34.598 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking python3-prettytable (0.7.2-5) ... 2023-09-30T01:30:34.757 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package libqt5core5a:amd64. 2023-09-30T01:30:34.777 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../015-libqt5core5a_5.12.8+dfsg-0ubuntu2.1_amd64.deb ... 2023-09-30T01:30:34.835 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking libqt5core5a:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2023-09-30T01:30:34.876 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package python3-rbd. 2023-09-30T01:30:34.893 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../009-python3-rbd_17.2.6-1165-g93dce5e1-1focal_amd64.deb ... 2023-09-30T01:30:34.934 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking python3-rbd (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T01:30:35.320 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package liblua5.3-0:amd64. 2023-09-30T01:30:35.337 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../010-liblua5.3-0_5.3.3-1.1ubuntu2_amd64.deb ... 2023-09-30T01:30:35.375 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2023-09-30T01:30:35.378 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking liblua5.3-0:amd64 (5.3.3-1.1ubuntu2) ... 2023-09-30T01:30:35.391 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../016-libqt5dbus5_5.12.8+dfsg-0ubuntu2.1_amd64.deb ... 2023-09-30T01:30:35.433 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking libqt5dbus5:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2023-09-30T01:30:35.747 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package librabbitmq4:amd64. 2023-09-30T01:30:35.763 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../011-librabbitmq4_0.10.0-1_amd64.deb ... 2023-09-30T01:30:35.805 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking librabbitmq4:amd64 (0.10.0-1) ... 2023-09-30T01:30:35.811 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package libqt5network5:amd64. 2023-09-30T01:30:35.827 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../017-libqt5network5_5.12.8+dfsg-0ubuntu2.1_amd64.deb ... 2023-09-30T01:30:35.870 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking libqt5network5:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2023-09-30T01:30:36.183 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package librdkafka1:amd64. 2023-09-30T01:30:36.204 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../012-librdkafka1_1.2.1-1ubuntu1_amd64.deb ... 2023-09-30T01:30:36.231 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package libthrift-0.13.0. 2023-09-30T01:30:36.241 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking librdkafka1:amd64 (1.2.1-1ubuntu1) ... 2023-09-30T01:30:36.250 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../018-libthrift-0.13.0_0.13.0-2build2_amd64.deb ... 2023-09-30T01:30:36.281 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking libthrift-0.13.0 (0.13.0-2build2) ... 2023-09-30T01:30:36.634 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2023-09-30T01:30:36.653 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../013-libdouble-conversion3_3.1.5-4ubuntu1_amd64.deb ... 2023-09-30T01:30:36.659 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package librgw2. 2023-09-30T01:30:36.677 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../019-librgw2_17.2.6-1165-g93dce5e1-1focal_amd64.deb ... 2023-09-30T01:30:36.694 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking libdouble-conversion3:amd64 (3.1.5-4ubuntu1) ... 2023-09-30T01:30:36.751 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking librgw2 (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T01:30:37.014 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2023-09-30T01:30:37.033 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../014-libpcre2-16-0_10.34-7ubuntu0.1_amd64.deb ... 2023-09-30T01:30:37.073 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking libpcre2-16-0:amd64 (10.34-7ubuntu0.1) ... 2023-09-30T01:30:37.528 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package python3-rgw. 2023-09-30T01:30:37.543 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package libqt5core5a:amd64. 2023-09-30T01:30:37.548 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../020-python3-rgw_17.2.6-1165-g93dce5e1-1focal_amd64.deb ... 2023-09-30T01:30:37.562 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../015-libqt5core5a_5.12.8+dfsg-0ubuntu2.1_amd64.deb ... 2023-09-30T01:30:37.586 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking python3-rgw (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T01:30:37.637 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking libqt5core5a:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2023-09-30T01:30:37.914 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package liboath0. 2023-09-30T01:30:37.930 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../021-liboath0_2.6.1-1.3_amd64.deb ... 2023-09-30T01:30:37.964 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking liboath0 (2.6.1-1.3) ... 2023-09-30T01:30:38.143 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2023-09-30T01:30:38.159 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../016-libqt5dbus5_5.12.8+dfsg-0ubuntu2.1_amd64.deb ... 2023-09-30T01:30:38.201 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking libqt5dbus5:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2023-09-30T01:30:38.286 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package libradosstriper1. 2023-09-30T01:30:38.302 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../022-libradosstriper1_17.2.6-1165-g93dce5e1-1focal_amd64.deb ... 2023-09-30T01:30:38.344 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking libradosstriper1 (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T01:30:38.596 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package libqt5network5:amd64. 2023-09-30T01:30:38.613 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../017-libqt5network5_5.12.8+dfsg-0ubuntu2.1_amd64.deb ... 2023-09-30T01:30:38.654 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking libqt5network5:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2023-09-30T01:30:38.773 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package ceph-common. 2023-09-30T01:30:38.789 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../023-ceph-common_17.2.6-1165-g93dce5e1-1focal_amd64.deb ... 2023-09-30T01:30:38.856 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking ceph-common (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T01:30:39.049 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package libthrift-0.13.0. 2023-09-30T01:30:39.065 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../018-libthrift-0.13.0_0.13.0-2build2_amd64.deb ... 2023-09-30T01:30:39.107 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking libthrift-0.13.0 (0.13.0-2build2) ... 2023-09-30T01:30:39.452 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package librgw2. 2023-09-30T01:30:39.469 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../019-librgw2_17.2.6-1165-g93dce5e1-1focal_amd64.deb ... 2023-09-30T01:30:39.544 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking librgw2 (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T01:30:40.346 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package python3-rgw. 2023-09-30T01:30:40.364 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../020-python3-rgw_17.2.6-1165-g93dce5e1-1focal_amd64.deb ... 2023-09-30T01:30:40.404 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking python3-rgw (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T01:30:40.765 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package liboath0. 2023-09-30T01:30:40.784 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../021-liboath0_2.6.1-1.3_amd64.deb ... 2023-09-30T01:30:40.823 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking liboath0 (2.6.1-1.3) ... 2023-09-30T01:30:41.160 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package libradosstriper1. 2023-09-30T01:30:41.175 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../022-libradosstriper1_17.2.6-1165-g93dce5e1-1focal_amd64.deb ... 2023-09-30T01:30:41.218 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking libradosstriper1 (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T01:30:41.544 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package ceph-base. 2023-09-30T01:30:41.561 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../024-ceph-base_17.2.6-1165-g93dce5e1-1focal_amd64.deb ... 2023-09-30T01:30:41.604 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking ceph-base (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T01:30:41.696 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package ceph-common. 2023-09-30T01:30:41.716 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../023-ceph-common_17.2.6-1165-g93dce5e1-1focal_amd64.deb ... 2023-09-30T01:30:41.754 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking ceph-common (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T01:30:42.590 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package python3-cryptography. 2023-09-30T01:30:42.606 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../025-python3-cryptography_2.8-3ubuntu0.1_amd64.deb ... 2023-09-30T01:30:42.640 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking python3-cryptography (2.8-3ubuntu0.1) ... 2023-09-30T01:30:43.060 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package python3-openssl. 2023-09-30T01:30:43.077 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../026-python3-openssl_19.0.0-1build1_all.deb ... 2023-09-30T01:30:43.118 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking python3-openssl (19.0.0-1build1) ... 2023-09-30T01:30:43.405 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2023-09-30T01:30:43.424 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../027-ceph-mgr-modules-core_17.2.6-1165-g93dce5e1-1focal_all.deb ... 2023-09-30T01:30:43.464 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking ceph-mgr-modules-core (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T01:30:43.894 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package python3-bcrypt. 2023-09-30T01:30:43.912 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../028-python3-bcrypt_3.1.7-2ubuntu1_amd64.deb ... 2023-09-30T01:30:43.951 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking python3-bcrypt (3.1.7-2ubuntu1) ... 2023-09-30T01:30:44.279 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package python3-cherrypy3. 2023-09-30T01:30:44.299 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../029-python3-cherrypy3_8.9.1-6_all.deb ... 2023-09-30T01:30:44.337 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking python3-cherrypy3 (8.9.1-6) ... 2023-09-30T01:30:44.467 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package ceph-base. 2023-09-30T01:30:44.487 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../024-ceph-base_17.2.6-1165-g93dce5e1-1focal_amd64.deb ... 2023-09-30T01:30:44.536 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking ceph-base (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T01:30:44.707 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package python3-natsort. 2023-09-30T01:30:44.723 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../030-python3-natsort_7.0.1-1_all.deb ... 2023-09-30T01:30:44.766 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking python3-natsort (7.0.1-1) ... 2023-09-30T01:30:45.119 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package python3-jwt. 2023-09-30T01:30:45.134 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../031-python3-jwt_1.7.1-2ubuntu2.1_all.deb ... 2023-09-30T01:30:45.177 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking python3-jwt (1.7.1-2ubuntu2.1) ... 2023-09-30T01:30:45.497 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package python3-logutils. 2023-09-30T01:30:45.516 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../032-python3-logutils_0.3.3-7_all.deb ... 2023-09-30T01:30:45.555 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking python3-logutils (0.3.3-7) ... 2023-09-30T01:30:45.580 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package python3-cryptography. 2023-09-30T01:30:45.597 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../025-python3-cryptography_2.8-3ubuntu0.1_amd64.deb ... 2023-09-30T01:30:45.638 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking python3-cryptography (2.8-3ubuntu0.1) ... 2023-09-30T01:30:45.899 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package python3-markupsafe. 2023-09-30T01:30:45.917 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../033-python3-markupsafe_1.1.0-1build2_amd64.deb ... 2023-09-30T01:30:45.982 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking python3-markupsafe (1.1.0-1build2) ... 2023-09-30T01:30:46.008 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package python3-openssl. 2023-09-30T01:30:46.028 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../026-python3-openssl_19.0.0-1build1_all.deb ... 2023-09-30T01:30:46.066 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking python3-openssl (19.0.0-1build1) ... 2023-09-30T01:30:46.319 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package python3-mako. 2023-09-30T01:30:46.337 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../034-python3-mako_1.1.0+ds1-1ubuntu2.1_all.deb ... 2023-09-30T01:30:46.370 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2023-09-30T01:30:46.386 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking python3-mako (1.1.0+ds1-1ubuntu2.1) ... 2023-09-30T01:30:46.388 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../027-ceph-mgr-modules-core_17.2.6-1165-g93dce5e1-1focal_all.deb ... 2023-09-30T01:30:46.428 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking ceph-mgr-modules-core (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T01:30:46.730 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package python3-simplegeneric. 2023-09-30T01:30:46.746 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../035-python3-simplegeneric_0.8.1-3_all.deb ... 2023-09-30T01:30:46.789 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2023-09-30T01:30:46.833 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package python3-bcrypt. 2023-09-30T01:30:46.852 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../028-python3-bcrypt_3.1.7-2ubuntu1_amd64.deb ... 2023-09-30T01:30:46.883 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking python3-bcrypt (3.1.7-2ubuntu1) ... 2023-09-30T01:30:47.116 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package python3-singledispatch. 2023-09-30T01:30:47.133 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../036-python3-singledispatch_3.4.0.3-2build1_all.deb ... 2023-09-30T01:30:47.174 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking python3-singledispatch (3.4.0.3-2build1) ... 2023-09-30T01:30:47.177 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package python3-cherrypy3. 2023-09-30T01:30:47.196 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../029-python3-cherrypy3_8.9.1-6_all.deb ... 2023-09-30T01:30:47.260 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking python3-cherrypy3 (8.9.1-6) ... 2023-09-30T01:30:47.511 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package python3-webob. 2023-09-30T01:30:47.527 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../037-python3-webob_1%3a1.8.5-2_all.deb ... 2023-09-30T01:30:47.569 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking python3-webob (1:1.8.5-2) ... 2023-09-30T01:30:48.473 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package python3-natsort. 2023-09-30T01:30:48.487 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../030-python3-natsort_7.0.1-1_all.deb ... 2023-09-30T01:30:48.530 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking python3-natsort (7.0.1-1) ... 2023-09-30T01:30:48.631 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package python3-soupsieve. 2023-09-30T01:30:48.649 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../038-python3-soupsieve_1.9.5+dfsg-1_all.deb ... 2023-09-30T01:30:48.689 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking python3-soupsieve (1.9.5+dfsg-1) ... 2023-09-30T01:30:48.875 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package python3-jwt. 2023-09-30T01:30:48.890 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../031-python3-jwt_1.7.1-2ubuntu2.1_all.deb ... 2023-09-30T01:30:48.933 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking python3-jwt (1.7.1-2ubuntu2.1) ... 2023-09-30T01:30:49.017 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package python3-bs4. 2023-09-30T01:30:49.036 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../039-python3-bs4_4.8.2-1_all.deb ... 2023-09-30T01:30:49.074 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking python3-bs4 (4.8.2-1) ... 2023-09-30T01:30:49.278 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package python3-logutils. 2023-09-30T01:30:49.294 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../032-python3-logutils_0.3.3-7_all.deb ... 2023-09-30T01:30:49.336 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking python3-logutils (0.3.3-7) ... 2023-09-30T01:30:49.453 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package python3-waitress. 2023-09-30T01:30:49.470 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../040-python3-waitress_1.4.1-1ubuntu0.1_all.deb ... 2023-09-30T01:30:49.514 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking python3-waitress (1.4.1-1ubuntu0.1) ... 2023-09-30T01:30:49.613 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package python3-markupsafe. 2023-09-30T01:30:49.632 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../033-python3-markupsafe_1.1.0-1build2_amd64.deb ... 2023-09-30T01:30:49.705 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking python3-markupsafe (1.1.0-1build2) ... 2023-09-30T01:30:49.856 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package python3-tempita. 2023-09-30T01:30:49.873 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../041-python3-tempita_0.5.2-6_all.deb ... 2023-09-30T01:30:49.933 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking python3-tempita (0.5.2-6) ... 2023-09-30T01:30:50.008 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package python3-mako. 2023-09-30T01:30:50.024 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../034-python3-mako_1.1.0+ds1-1ubuntu2.1_all.deb ... 2023-09-30T01:30:50.066 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking python3-mako (1.1.0+ds1-1ubuntu2.1) ... 2023-09-30T01:30:50.242 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package python3-paste. 2023-09-30T01:30:50.258 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../042-python3-paste_3.4.0+dfsg1-1_all.deb ... 2023-09-30T01:30:50.325 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking python3-paste (3.4.0+dfsg1-1) ... 2023-09-30T01:30:50.420 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package python3-simplegeneric. 2023-09-30T01:30:50.435 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../035-python3-simplegeneric_0.8.1-3_all.deb ... 2023-09-30T01:30:50.478 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2023-09-30T01:30:50.662 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2023-09-30T01:30:50.678 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../043-python-pastedeploy-tpl_2.1.0-1_all.deb ... 2023-09-30T01:30:50.720 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking python-pastedeploy-tpl (2.1.0-1) ... 2023-09-30T01:30:50.780 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package python3-singledispatch. 2023-09-30T01:30:50.800 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../036-python3-singledispatch_3.4.0.3-2build1_all.deb ... 2023-09-30T01:30:50.838 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking python3-singledispatch (3.4.0.3-2build1) ... 2023-09-30T01:30:51.056 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package python3-pastedeploy. 2023-09-30T01:30:51.072 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../044-python3-pastedeploy_2.1.0-1_all.deb ... 2023-09-30T01:30:51.114 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking python3-pastedeploy (2.1.0-1) ... 2023-09-30T01:30:51.133 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package python3-webob. 2023-09-30T01:30:51.150 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../037-python3-webob_1%3a1.8.5-2_all.deb ... 2023-09-30T01:30:51.191 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking python3-webob (1:1.8.5-2) ... 2023-09-30T01:30:51.433 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package python3-webtest. 2023-09-30T01:30:51.452 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../045-python3-webtest_2.0.32-2_all.deb ... 2023-09-30T01:30:51.491 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking python3-webtest (2.0.32-2) ... 2023-09-30T01:30:51.494 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package python3-soupsieve. 2023-09-30T01:30:51.510 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../038-python3-soupsieve_1.9.5+dfsg-1_all.deb ... 2023-09-30T01:30:51.552 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking python3-soupsieve (1.9.5+dfsg-1) ... 2023-09-30T01:30:51.836 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package python3-pecan. 2023-09-30T01:30:51.854 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../046-python3-pecan_1.3.3-3ubuntu1_all.deb ... 2023-09-30T01:30:51.888 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package python3-bs4. 2023-09-30T01:30:51.894 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking python3-pecan (1.3.3-3ubuntu1) ... 2023-09-30T01:30:51.905 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../039-python3-bs4_4.8.2-1_all.deb ... 2023-09-30T01:30:51.946 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking python3-bs4 (4.8.2-1) ... 2023-09-30T01:30:52.258 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package python3-werkzeug. 2023-09-30T01:30:52.277 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../047-python3-werkzeug_0.16.1+dfsg1-2ubuntu0.1_all.deb ... 2023-09-30T01:30:52.315 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking python3-werkzeug (0.16.1+dfsg1-2ubuntu0.1) ... 2023-09-30T01:30:52.341 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package python3-waitress. 2023-09-30T01:30:52.358 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../040-python3-waitress_1.4.1-1ubuntu0.1_all.deb ... 2023-09-30T01:30:52.402 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking python3-waitress (1.4.1-1ubuntu0.1) ... 2023-09-30T01:30:52.694 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2023-09-30T01:30:52.710 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../048-libsqlite3-mod-ceph_17.2.6-1165-g93dce5e1-1focal_amd64.deb ... 2023-09-30T01:30:52.761 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package python3-tempita. 2023-09-30T01:30:52.777 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking libsqlite3-mod-ceph (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T01:30:52.778 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../041-python3-tempita_0.5.2-6_all.deb ... 2023-09-30T01:30:52.819 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking python3-tempita (0.5.2-6) ... 2023-09-30T01:30:53.063 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package ceph-mgr. 2023-09-30T01:30:53.081 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../049-ceph-mgr_17.2.6-1165-g93dce5e1-1focal_amd64.deb ... 2023-09-30T01:30:53.138 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking ceph-mgr (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T01:30:53.147 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package python3-paste. 2023-09-30T01:30:53.163 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../042-python3-paste_3.4.0+dfsg1-1_all.deb ... 2023-09-30T01:30:53.205 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking python3-paste (3.4.0+dfsg1-1) ... 2023-09-30T01:30:53.531 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2023-09-30T01:30:53.549 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../043-python-pastedeploy-tpl_2.1.0-1_all.deb ... 2023-09-30T01:30:53.591 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking python-pastedeploy-tpl (2.1.0-1) ... 2023-09-30T01:30:53.641 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package ceph-mon. 2023-09-30T01:30:53.660 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../050-ceph-mon_17.2.6-1165-g93dce5e1-1focal_amd64.deb ... 2023-09-30T01:30:53.699 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking ceph-mon (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T01:30:53.885 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package python3-pastedeploy. 2023-09-30T01:30:53.904 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../044-python3-pastedeploy_2.1.0-1_all.deb ... 2023-09-30T01:30:53.943 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking python3-pastedeploy (2.1.0-1) ... 2023-09-30T01:30:54.280 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package python3-webtest. 2023-09-30T01:30:54.295 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../045-python3-webtest_2.0.32-2_all.deb ... 2023-09-30T01:30:54.338 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking python3-webtest (2.0.32-2) ... 2023-09-30T01:30:54.649 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package python3-pecan. 2023-09-30T01:30:54.665 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../046-python3-pecan_1.3.3-3ubuntu1_all.deb ... 2023-09-30T01:30:54.698 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking python3-pecan (1.3.3-3ubuntu1) ... 2023-09-30T01:30:54.911 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package ceph-osd. 2023-09-30T01:30:54.931 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../051-ceph-osd_17.2.6-1165-g93dce5e1-1focal_amd64.deb ... 2023-09-30T01:30:54.969 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking ceph-osd (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T01:30:55.037 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package python3-werkzeug. 2023-09-30T01:30:55.053 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../047-python3-werkzeug_0.16.1+dfsg1-2ubuntu0.1_all.deb ... 2023-09-30T01:30:55.086 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking python3-werkzeug (0.16.1+dfsg1-2ubuntu0.1) ... 2023-09-30T01:30:55.481 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2023-09-30T01:30:55.498 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../048-libsqlite3-mod-ceph_17.2.6-1165-g93dce5e1-1focal_amd64.deb ... 2023-09-30T01:30:55.539 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking libsqlite3-mod-ceph (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T01:30:55.892 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package ceph-mgr. 2023-09-30T01:30:55.911 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../049-ceph-mgr_17.2.6-1165-g93dce5e1-1focal_amd64.deb ... 2023-09-30T01:30:55.958 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking ceph-mgr (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T01:30:56.478 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package ceph-mon. 2023-09-30T01:30:56.495 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../050-ceph-mon_17.2.6-1165-g93dce5e1-1focal_amd64.deb ... 2023-09-30T01:30:56.536 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking ceph-mon (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T01:30:57.648 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package ceph-osd. 2023-09-30T01:30:57.666 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../051-ceph-osd_17.2.6-1165-g93dce5e1-1focal_amd64.deb ... 2023-09-30T01:30:57.706 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking ceph-osd (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T01:30:57.814 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package ceph. 2023-09-30T01:30:57.830 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../052-ceph_17.2.6-1165-g93dce5e1-1focal_amd64.deb ... 2023-09-30T01:30:57.872 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking ceph (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T01:30:58.216 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package ceph-fuse. 2023-09-30T01:30:58.233 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../053-ceph-fuse_17.2.6-1165-g93dce5e1-1focal_amd64.deb ... 2023-09-30T01:30:58.299 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking ceph-fuse (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T01:30:58.794 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package ceph-mds. 2023-09-30T01:30:58.810 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../054-ceph-mds_17.2.6-1165-g93dce5e1-1focal_amd64.deb ... 2023-09-30T01:30:58.853 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking ceph-mds (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T01:30:59.364 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package cephadm. 2023-09-30T01:30:59.383 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../055-cephadm_17.2.6-1165-g93dce5e1-1focal_amd64.deb ... 2023-09-30T01:30:59.413 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking cephadm (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T01:30:59.758 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package python3-jinja2. 2023-09-30T01:30:59.776 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../056-python3-jinja2_2.10.1-2_all.deb ... 2023-09-30T01:30:59.816 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking python3-jinja2 (2.10.1-2) ... 2023-09-30T01:31:00.161 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2023-09-30T01:31:00.181 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../057-ceph-mgr-cephadm_17.2.6-1165-g93dce5e1-1focal_all.deb ... 2023-09-30T01:31:00.218 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking ceph-mgr-cephadm (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T01:31:00.517 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package ceph. 2023-09-30T01:31:00.534 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../052-ceph_17.2.6-1165-g93dce5e1-1focal_amd64.deb ... 2023-09-30T01:31:00.575 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking ceph (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T01:31:00.580 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package python3-repoze.lru. 2023-09-30T01:31:00.600 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../058-python3-repoze.lru_0.7-2_all.deb ... 2023-09-30T01:31:00.638 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2023-09-30T01:31:00.890 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package libjs-underscore. 2023-09-30T01:31:00.903 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package ceph-fuse. 2023-09-30T01:31:00.906 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../059-libjs-underscore_1.9.1~dfsg-1ubuntu0.20.04.1_all.deb ... 2023-09-30T01:31:00.921 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../053-ceph-fuse_17.2.6-1165-g93dce5e1-1focal_amd64.deb ... 2023-09-30T01:31:00.961 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking ceph-fuse (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T01:31:00.973 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking libjs-underscore (1.9.1~dfsg-1ubuntu0.20.04.1) ... 2023-09-30T01:31:01.285 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package libjs-sphinxdoc. 2023-09-30T01:31:01.300 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../060-libjs-sphinxdoc_1.8.5-7ubuntu3_all.deb ... 2023-09-30T01:31:01.343 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking libjs-sphinxdoc (1.8.5-7ubuntu3) ... 2023-09-30T01:31:01.397 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package ceph-mds. 2023-09-30T01:31:01.414 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../054-ceph-mds_17.2.6-1165-g93dce5e1-1focal_amd64.deb ... 2023-09-30T01:31:01.447 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking ceph-mds (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T01:31:01.687 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package python3-routes. 2023-09-30T01:31:01.705 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../061-python3-routes_2.4.1-2_all.deb ... 2023-09-30T01:31:01.745 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking python3-routes (2.4.1-2) ... 2023-09-30T01:31:01.992 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package cephadm. 2023-09-30T01:31:02.012 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../055-cephadm_17.2.6-1165-g93dce5e1-1focal_amd64.deb ... 2023-09-30T01:31:02.049 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking cephadm (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T01:31:02.090 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2023-09-30T01:31:02.106 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../062-ceph-mgr-dashboard_17.2.6-1165-g93dce5e1-1focal_all.deb ... 2023-09-30T01:31:02.165 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking ceph-mgr-dashboard (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T01:31:02.411 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package python3-jinja2. 2023-09-30T01:31:02.427 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../056-python3-jinja2_2.10.1-2_all.deb ... 2023-09-30T01:31:02.469 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking python3-jinja2 (2.10.1-2) ... 2023-09-30T01:31:02.652 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package python3-decorator. 2023-09-30T01:31:02.668 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../063-python3-decorator_4.4.2-0ubuntu1_all.deb ... 2023-09-30T01:31:02.710 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking python3-decorator (4.4.2-0ubuntu1) ... 2023-09-30T01:31:02.813 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2023-09-30T01:31:02.830 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../057-ceph-mgr-cephadm_17.2.6-1165-g93dce5e1-1focal_all.deb ... 2023-09-30T01:31:02.871 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking ceph-mgr-cephadm (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T01:31:03.021 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package liblbfgsb0:amd64. 2023-09-30T01:31:03.041 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../064-liblbfgsb0_3.0+dfsg.3-7build1_amd64.deb ... 2023-09-30T01:31:03.078 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking liblbfgsb0:amd64 (3.0+dfsg.3-7build1) ... 2023-09-30T01:31:03.241 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package python3-repoze.lru. 2023-09-30T01:31:03.257 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../058-python3-repoze.lru_0.7-2_all.deb ... 2023-09-30T01:31:03.299 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2023-09-30T01:31:03.415 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package python3-scipy. 2023-09-30T01:31:03.433 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../065-python3-scipy_1.3.3-3build1_amd64.deb ... 2023-09-30T01:31:03.473 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking python3-scipy (1.3.3-3build1) ... 2023-09-30T01:31:03.852 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package libjs-underscore. 2023-09-30T01:31:03.868 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../059-libjs-underscore_1.9.1~dfsg-1ubuntu0.20.04.1_all.deb ... 2023-09-30T01:31:03.910 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking libjs-underscore (1.9.1~dfsg-1ubuntu0.20.04.1) ... 2023-09-30T01:31:04.162 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package libjs-sphinxdoc. 2023-09-30T01:31:04.181 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../060-libjs-sphinxdoc_1.8.5-7ubuntu3_all.deb ... 2023-09-30T01:31:04.220 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking libjs-sphinxdoc (1.8.5-7ubuntu3) ... 2023-09-30T01:31:04.531 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package python3-routes. 2023-09-30T01:31:04.552 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../061-python3-routes_2.4.1-2_all.deb ... 2023-09-30T01:31:04.589 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking python3-routes (2.4.1-2) ... 2023-09-30T01:31:04.934 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2023-09-30T01:31:04.951 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../062-ceph-mgr-dashboard_17.2.6-1165-g93dce5e1-1focal_all.deb ... 2023-09-30T01:31:04.992 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking ceph-mgr-dashboard (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T01:31:05.015 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package python3-sklearn-lib. 2023-09-30T01:31:05.033 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../066-python3-sklearn-lib_0.22.2.post1+dfsg-5_amd64.deb ... 2023-09-30T01:31:05.073 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking python3-sklearn-lib (0.22.2.post1+dfsg-5) ... 2023-09-30T01:31:05.512 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package python3-decorator. 2023-09-30T01:31:05.532 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../063-python3-decorator_4.4.2-0ubuntu1_all.deb ... 2023-09-30T01:31:05.570 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking python3-decorator (4.4.2-0ubuntu1) ... 2023-09-30T01:31:05.602 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package python3-joblib. 2023-09-30T01:31:05.618 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../067-python3-joblib_0.14.0-3_all.deb ... 2023-09-30T01:31:05.671 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking python3-joblib (0.14.0-3) ... 2023-09-30T01:31:05.898 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package liblbfgsb0:amd64. 2023-09-30T01:31:05.914 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../064-liblbfgsb0_3.0+dfsg.3-7build1_amd64.deb ... 2023-09-30T01:31:05.956 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking liblbfgsb0:amd64 (3.0+dfsg.3-7build1) ... 2023-09-30T01:31:06.055 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package python3-sklearn. 2023-09-30T01:31:06.071 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../068-python3-sklearn_0.22.2.post1+dfsg-5_all.deb ... 2023-09-30T01:31:06.113 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking python3-sklearn (0.22.2.post1+dfsg-5) ... 2023-09-30T01:31:06.250 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package python3-scipy. 2023-09-30T01:31:06.269 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../065-python3-scipy_1.3.3-3build1_amd64.deb ... 2023-09-30T01:31:06.316 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking python3-scipy (1.3.3-3build1) ... 2023-09-30T01:31:06.829 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2023-09-30T01:31:06.849 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../069-ceph-mgr-diskprediction-local_17.2.6-1165-g93dce5e1-1focal_all.deb ... 2023-09-30T01:31:06.887 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking ceph-mgr-diskprediction-local (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T01:31:07.617 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package python3-sklearn-lib. 2023-09-30T01:31:07.634 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../066-python3-sklearn-lib_0.22.2.post1+dfsg-5_amd64.deb ... 2023-09-30T01:31:07.675 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking python3-sklearn-lib (0.22.2.post1+dfsg-5) ... 2023-09-30T01:31:08.007 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package python3-cachetools. 2023-09-30T01:31:08.023 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../070-python3-cachetools_4.0.0-1_all.deb ... 2023-09-30T01:31:08.065 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking python3-cachetools (4.0.0-1) ... 2023-09-30T01:31:08.237 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package python3-joblib. 2023-09-30T01:31:08.253 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../067-python3-joblib_0.14.0-3_all.deb ... 2023-09-30T01:31:08.294 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking python3-joblib (0.14.0-3) ... 2023-09-30T01:31:08.384 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package python3-pyasn1. 2023-09-30T01:31:08.402 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../071-python3-pyasn1_0.4.2-3build1_all.deb ... 2023-09-30T01:31:08.442 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking python3-pyasn1 (0.4.2-3build1) ... 2023-09-30T01:31:08.665 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package python3-sklearn. 2023-09-30T01:31:08.684 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../068-python3-sklearn_0.22.2.post1+dfsg-5_all.deb ... 2023-09-30T01:31:08.739 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking python3-sklearn (0.22.2.post1+dfsg-5) ... 2023-09-30T01:31:08.796 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package python3-pyasn1-modules. 2023-09-30T01:31:08.822 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../072-python3-pyasn1-modules_0.2.1-0.2build1_all.deb ... 2023-09-30T01:31:08.879 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking python3-pyasn1-modules (0.2.1-0.2build1) ... 2023-09-30T01:31:09.215 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package python3-rsa. 2023-09-30T01:31:09.231 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../073-python3-rsa_4.0-3_all.deb ... 2023-09-30T01:31:09.273 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking python3-rsa (4.0-3) ... 2023-09-30T01:31:09.514 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2023-09-30T01:31:09.530 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../069-ceph-mgr-diskprediction-local_17.2.6-1165-g93dce5e1-1focal_all.deb ... 2023-09-30T01:31:09.572 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking ceph-mgr-diskprediction-local (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T01:31:09.602 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package python3-google-auth. 2023-09-30T01:31:09.621 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../074-python3-google-auth_1.5.1-2_all.deb ... 2023-09-30T01:31:09.659 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking python3-google-auth (1.5.1-2) ... 2023-09-30T01:31:10.013 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package python3-blinker. 2023-09-30T01:31:10.030 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../075-python3-blinker_1.4+dfsg1-0.3ubuntu1_all.deb ... 2023-09-30T01:31:10.070 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking python3-blinker (1.4+dfsg1-0.3ubuntu1) ... 2023-09-30T01:31:10.398 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package python3-oauthlib. 2023-09-30T01:31:10.417 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../076-python3-oauthlib_3.1.0-1ubuntu2_all.deb ... 2023-09-30T01:31:10.456 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking python3-oauthlib (3.1.0-1ubuntu2) ... 2023-09-30T01:31:10.658 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package python3-cachetools. 2023-09-30T01:31:10.675 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../070-python3-cachetools_4.0.0-1_all.deb ... 2023-09-30T01:31:10.716 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking python3-cachetools (4.0.0-1) ... 2023-09-30T01:31:10.785 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package python3-requests-oauthlib. 2023-09-30T01:31:10.804 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../077-python3-requests-oauthlib_1.0.0-1.1build1_all.deb ... 2023-09-30T01:31:10.843 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking python3-requests-oauthlib (1.0.0-1.1build1) ... 2023-09-30T01:31:11.027 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package python3-pyasn1. 2023-09-30T01:31:11.045 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../071-python3-pyasn1_0.4.2-3build1_all.deb ... 2023-09-30T01:31:11.118 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking python3-pyasn1 (0.4.2-3build1) ... 2023-09-30T01:31:11.179 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package python3-websocket. 2023-09-30T01:31:11.195 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../078-python3-websocket_0.53.0-2ubuntu1_all.deb ... 2023-09-30T01:31:11.237 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking python3-websocket (0.53.0-2ubuntu1) ... 2023-09-30T01:31:11.446 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package python3-pyasn1-modules. 2023-09-30T01:31:11.465 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../072-python3-pyasn1-modules_0.2.1-0.2build1_all.deb ... 2023-09-30T01:31:11.505 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking python3-pyasn1-modules (0.2.1-0.2build1) ... 2023-09-30T01:31:11.581 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package python3-kubernetes. 2023-09-30T01:31:11.598 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../079-python3-kubernetes_7.0.0~a1-2_all.deb ... 2023-09-30T01:31:11.666 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking python3-kubernetes (7.0.0~a1-2) ... 2023-09-30T01:31:11.841 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package python3-rsa. 2023-09-30T01:31:11.858 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../073-python3-rsa_4.0-3_all.deb ... 2023-09-30T01:31:11.899 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking python3-rsa (4.0-3) ... 2023-09-30T01:31:12.178 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2023-09-30T01:31:12.194 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../080-ceph-mgr-k8sevents_17.2.6-1165-g93dce5e1-1focal_all.deb ... 2023-09-30T01:31:12.236 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking ceph-mgr-k8sevents (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T01:31:12.243 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package python3-google-auth. 2023-09-30T01:31:12.262 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../074-python3-google-auth_1.5.1-2_all.deb ... 2023-09-30T01:31:12.302 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking python3-google-auth (1.5.1-2) ... 2023-09-30T01:31:12.630 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package python3-blinker. 2023-09-30T01:31:12.649 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../075-python3-blinker_1.4+dfsg1-0.3ubuntu1_all.deb ... 2023-09-30T01:31:12.665 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package libonig5:amd64. 2023-09-30T01:31:12.682 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../081-libonig5_6.9.4-1_amd64.deb ... 2023-09-30T01:31:12.687 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking python3-blinker (1.4+dfsg1-0.3ubuntu1) ... 2023-09-30T01:31:12.749 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking libonig5:amd64 (6.9.4-1) ... 2023-09-30T01:31:12.957 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package python3-oauthlib. 2023-09-30T01:31:12.976 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../076-python3-oauthlib_3.1.0-1ubuntu2_all.deb ... 2023-09-30T01:31:13.032 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking python3-oauthlib (3.1.0-1ubuntu2) ... 2023-09-30T01:31:13.102 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package libjq1:amd64. 2023-09-30T01:31:13.121 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../082-libjq1_1.6-1ubuntu0.20.04.1_amd64.deb ... 2023-09-30T01:31:13.160 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking libjq1:amd64 (1.6-1ubuntu0.20.04.1) ... 2023-09-30T01:31:13.352 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package python3-requests-oauthlib. 2023-09-30T01:31:13.370 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../077-python3-requests-oauthlib_1.0.0-1.1build1_all.deb ... 2023-09-30T01:31:13.401 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking python3-requests-oauthlib (1.0.0-1.1build1) ... 2023-09-30T01:31:13.446 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package jq. 2023-09-30T01:31:13.462 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../083-jq_1.6-1ubuntu0.20.04.1_amd64.deb ... 2023-09-30T01:31:13.504 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking jq (1.6-1ubuntu0.20.04.1) ... 2023-09-30T01:31:13.712 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package python3-websocket. 2023-09-30T01:31:13.732 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package socat. 2023-09-30T01:31:13.732 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../078-python3-websocket_0.53.0-2ubuntu1_all.deb ... 2023-09-30T01:31:13.752 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../084-socat_1.7.3.3-2_amd64.deb ... 2023-09-30T01:31:13.770 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking python3-websocket (0.53.0-2ubuntu1) ... 2023-09-30T01:31:13.806 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking socat (1.7.3.3-2) ... 2023-09-30T01:31:14.107 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package python3-kubernetes. 2023-09-30T01:31:14.123 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../079-python3-kubernetes_7.0.0~a1-2_all.deb ... 2023-09-30T01:31:14.135 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package xmlstarlet. 2023-09-30T01:31:14.156 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../085-xmlstarlet_1.6.1-2build1_amd64.deb ... 2023-09-30T01:31:14.164 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking python3-kubernetes (7.0.0~a1-2) ... 2023-09-30T01:31:14.193 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking xmlstarlet (1.6.1-2build1) ... 2023-09-30T01:31:14.533 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package ceph-test. 2023-09-30T01:31:14.552 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../086-ceph-test_17.2.6-1165-g93dce5e1-1focal_amd64.deb ... 2023-09-30T01:31:14.589 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking ceph-test (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T01:31:14.736 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2023-09-30T01:31:14.754 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../080-ceph-mgr-k8sevents_17.2.6-1165-g93dce5e1-1focal_all.deb ... 2023-09-30T01:31:14.794 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking ceph-mgr-k8sevents (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T01:31:15.215 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package libonig5:amd64. 2023-09-30T01:31:15.232 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../081-libonig5_6.9.4-1_amd64.deb ... 2023-09-30T01:31:15.274 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking libonig5:amd64 (6.9.4-1) ... 2023-09-30T01:31:15.677 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package libjq1:amd64. 2023-09-30T01:31:15.693 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../082-libjq1_1.6-1ubuntu0.20.04.1_amd64.deb ... 2023-09-30T01:31:15.735 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking libjq1:amd64 (1.6-1ubuntu0.20.04.1) ... 2023-09-30T01:31:16.037 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package jq. 2023-09-30T01:31:16.054 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../083-jq_1.6-1ubuntu0.20.04.1_amd64.deb ... 2023-09-30T01:31:16.096 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking jq (1.6-1ubuntu0.20.04.1) ... 2023-09-30T01:31:16.382 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package socat. 2023-09-30T01:31:16.401 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../084-socat_1.7.3.3-2_amd64.deb ... 2023-09-30T01:31:16.440 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking socat (1.7.3.3-2) ... 2023-09-30T01:31:16.727 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package xmlstarlet. 2023-09-30T01:31:16.745 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../085-xmlstarlet_1.6.1-2build1_amd64.deb ... 2023-09-30T01:31:16.785 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking xmlstarlet (1.6.1-2build1) ... 2023-09-30T01:31:17.148 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package ceph-test. 2023-09-30T01:31:17.167 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../086-ceph-test_17.2.6-1165-g93dce5e1-1focal_amd64.deb ... 2023-09-30T01:31:17.206 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking ceph-test (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T01:31:19.676 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package ceph-volume. 2023-09-30T01:31:19.694 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../087-ceph-volume_17.2.6-1165-g93dce5e1-1focal_all.deb ... 2023-09-30T01:31:19.734 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking ceph-volume (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T01:31:20.098 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package fonts-mathjax. 2023-09-30T01:31:20.117 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../088-fonts-mathjax_2.7.4+dfsg-1_all.deb ... 2023-09-30T01:31:20.156 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking fonts-mathjax (2.7.4+dfsg-1) ... 2023-09-30T01:31:20.701 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package libcephfs-dev. 2023-09-30T01:31:20.718 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../089-libcephfs-dev_17.2.6-1165-g93dce5e1-1focal_amd64.deb ... 2023-09-30T01:31:20.759 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking libcephfs-dev (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T01:31:21.170 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package nvme-cli. 2023-09-30T01:31:21.186 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../090-nvme-cli_1.9-1ubuntu0.1_amd64.deb ... 2023-09-30T01:31:21.246 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking nvme-cli (1.9-1ubuntu0.1) ... 2023-09-30T01:31:21.608 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package libjs-mathjax. 2023-09-30T01:31:21.625 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../091-libjs-mathjax_2.7.4+dfsg-1_all.deb ... 2023-09-30T01:31:21.682 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking libjs-mathjax (2.7.4+dfsg-1) ... 2023-09-30T01:31:22.328 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package ceph-volume. 2023-09-30T01:31:22.344 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../087-ceph-volume_17.2.6-1165-g93dce5e1-1focal_all.deb ... 2023-09-30T01:31:22.386 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking ceph-volume (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T01:31:22.699 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package fonts-mathjax. 2023-09-30T01:31:22.718 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../088-fonts-mathjax_2.7.4+dfsg-1_all.deb ... 2023-09-30T01:31:22.757 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking fonts-mathjax (2.7.4+dfsg-1) ... 2023-09-30T01:31:23.320 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package libcephfs-dev. 2023-09-30T01:31:23.340 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../089-libcephfs-dev_17.2.6-1165-g93dce5e1-1focal_amd64.deb ... 2023-09-30T01:31:23.378 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking libcephfs-dev (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T01:31:23.405 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package python-natsort-doc. 2023-09-30T01:31:23.426 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../092-python-natsort-doc_7.0.1-1_all.deb ... 2023-09-30T01:31:23.472 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking python-natsort-doc (7.0.1-1) ... 2023-09-30T01:31:23.798 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package nvme-cli. 2023-09-30T01:31:23.814 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../090-nvme-cli_1.9-1ubuntu0.1_amd64.deb ... 2023-09-30T01:31:23.841 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package python3-atomicwrites. 2023-09-30T01:31:23.859 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../093-python3-atomicwrites_1.1.5-2build1_all.deb ... 2023-09-30T01:31:23.880 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking nvme-cli (1.9-1ubuntu0.1) ... 2023-09-30T01:31:23.899 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking python3-atomicwrites (1.1.5-2build1) ... 2023-09-30T01:31:24.212 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package libjs-mathjax. 2023-09-30T01:31:24.219 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package python3-attr. 2023-09-30T01:31:24.232 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../091-libjs-mathjax_2.7.4+dfsg-1_all.deb ... 2023-09-30T01:31:24.235 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../094-python3-attr_19.3.0-2_all.deb ... 2023-09-30T01:31:24.277 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking python3-attr (19.3.0-2) ... 2023-09-30T01:31:24.278 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking libjs-mathjax (2.7.4+dfsg-1) ... 2023-09-30T01:31:24.638 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package python3-webencodings. 2023-09-30T01:31:24.655 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../095-python3-webencodings_0.5.1-1ubuntu1_all.deb ... 2023-09-30T01:31:24.696 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking python3-webencodings (0.5.1-1ubuntu1) ... 2023-09-30T01:31:25.024 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package python3-html5lib. 2023-09-30T01:31:25.040 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../096-python3-html5lib_1.0.1-2_all.deb ... 2023-09-30T01:31:25.082 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking python3-html5lib (1.0.1-2) ... 2023-09-30T01:31:25.452 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package python3-lxml:amd64. 2023-09-30T01:31:25.470 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../097-python3-lxml_4.5.0-1ubuntu0.5_amd64.deb ... 2023-09-30T01:31:25.543 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking python3-lxml:amd64 (4.5.0-1ubuntu0.5) ... 2023-09-30T01:31:25.974 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package python-natsort-doc. 2023-09-30T01:31:25.992 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../092-python-natsort-doc_7.0.1-1_all.deb ... 2023-09-30T01:31:26.022 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package python3-packaging. 2023-09-30T01:31:26.033 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking python-natsort-doc (7.0.1-1) ... 2023-09-30T01:31:26.038 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../098-python3-packaging_20.3-1_all.deb ... 2023-09-30T01:31:26.080 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking python3-packaging (20.3-1) ... 2023-09-30T01:31:26.369 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package python3-atomicwrites. 2023-09-30T01:31:26.390 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../093-python3-atomicwrites_1.1.5-2build1_all.deb ... 2023-09-30T01:31:26.415 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package python3-pastescript. 2023-09-30T01:31:26.427 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking python3-atomicwrites (1.1.5-2build1) ... 2023-09-30T01:31:26.432 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../099-python3-pastescript_2.0.2-4_all.deb ... 2023-09-30T01:31:26.474 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2023-09-30T01:31:26.731 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package python3-attr. 2023-09-30T01:31:26.747 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../094-python3-attr_19.3.0-2_all.deb ... 2023-09-30T01:31:26.814 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking python3-attr (19.3.0-2) ... 2023-09-30T01:31:26.818 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package python3-pluggy. 2023-09-30T01:31:26.836 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../100-python3-pluggy_0.13.0-2_all.deb ... 2023-09-30T01:31:26.877 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking python3-pluggy (0.13.0-2) ... 2023-09-30T01:31:27.117 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package python3-webencodings. 2023-09-30T01:31:27.137 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../095-python3-webencodings_0.5.1-1ubuntu1_all.deb ... 2023-09-30T01:31:27.183 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking python3-webencodings (0.5.1-1ubuntu1) ... 2023-09-30T01:31:27.196 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package python3-psutil. 2023-09-30T01:31:27.217 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../101-python3-psutil_5.5.1-1ubuntu4_amd64.deb ... 2023-09-30T01:31:27.274 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking python3-psutil (5.5.1-1ubuntu4) ... 2023-09-30T01:31:27.519 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package python3-html5lib. 2023-09-30T01:31:27.538 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../096-python3-html5lib_1.0.1-2_all.deb ... 2023-09-30T01:31:27.577 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking python3-html5lib (1.0.1-2) ... 2023-09-30T01:31:27.641 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package python3-py. 2023-09-30T01:31:27.661 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../102-python3-py_1.8.1-1ubuntu0.1_all.deb ... 2023-09-30T01:31:27.699 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking python3-py (1.8.1-1ubuntu0.1) ... 2023-09-30T01:31:27.948 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package python3-lxml:amd64. 2023-09-30T01:31:27.967 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../097-python3-lxml_4.5.0-1ubuntu0.5_amd64.deb ... 2023-09-30T01:31:28.014 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking python3-lxml:amd64 (4.5.0-1ubuntu0.5) ... 2023-09-30T01:31:28.052 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package python3-pyinotify. 2023-09-30T01:31:28.068 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../103-python3-pyinotify_0.9.6-1.2ubuntu1_all.deb ... 2023-09-30T01:31:28.110 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking python3-pyinotify (0.9.6-1.2ubuntu1) ... 2023-09-30T01:31:28.471 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package python3-wcwidth. 2023-09-30T01:31:28.490 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../104-python3-wcwidth_0.1.8+dfsg1-3_all.deb ... 2023-09-30T01:31:28.492 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package python3-packaging. 2023-09-30T01:31:28.509 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../098-python3-packaging_20.3-1_all.deb ... 2023-09-30T01:31:28.529 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking python3-wcwidth (0.1.8+dfsg1-3) ... 2023-09-30T01:31:28.550 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking python3-packaging (20.3-1) ... 2023-09-30T01:31:28.840 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package python3-pytest. 2023-09-30T01:31:28.861 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../105-python3-pytest_4.6.9-1_all.deb ... 2023-09-30T01:31:28.887 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package python3-pastescript. 2023-09-30T01:31:28.898 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking python3-pytest (4.6.9-1) ... 2023-09-30T01:31:28.903 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../099-python3-pastescript_2.0.2-4_all.deb ... 2023-09-30T01:31:28.945 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2023-09-30T01:31:29.268 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package python3-simplejson. 2023-09-30T01:31:29.273 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package python3-pluggy. 2023-09-30T01:31:29.289 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../106-python3-simplejson_3.16.0-2ubuntu2_amd64.deb ... 2023-09-30T01:31:29.294 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../100-python3-pluggy_0.13.0-2_all.deb ... 2023-09-30T01:31:29.326 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking python3-simplejson (3.16.0-2ubuntu2) ... 2023-09-30T01:31:29.332 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking python3-pluggy (0.13.0-2) ... 2023-09-30T01:31:29.629 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package qttranslations5-l10n. 2023-09-30T01:31:29.643 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package python3-psutil. 2023-09-30T01:31:29.649 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../107-qttranslations5-l10n_5.12.8-0ubuntu1_all.deb ... 2023-09-30T01:31:29.660 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../101-python3-psutil_5.5.1-1ubuntu4_amd64.deb ... 2023-09-30T01:31:29.706 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking qttranslations5-l10n (5.12.8-0ubuntu1) ... 2023-09-30T01:31:29.726 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking python3-psutil (5.5.1-1ubuntu4) ... 2023-09-30T01:31:30.046 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package python3-py. 2023-09-30T01:31:30.066 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../102-python3-py_1.8.1-1ubuntu0.1_all.deb ... 2023-09-30T01:31:30.104 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking python3-py (1.8.1-1ubuntu0.1) ... 2023-09-30T01:31:30.432 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package python3-pyinotify. 2023-09-30T01:31:30.453 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../103-python3-pyinotify_0.9.6-1.2ubuntu1_all.deb ... 2023-09-30T01:31:30.490 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking python3-pyinotify (0.9.6-1.2ubuntu1) ... 2023-09-30T01:31:30.542 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package radosgw. 2023-09-30T01:31:30.559 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../108-radosgw_17.2.6-1165-g93dce5e1-1focal_amd64.deb ... 2023-09-30T01:31:30.600 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking radosgw (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T01:31:30.818 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package python3-wcwidth. 2023-09-30T01:31:30.835 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../104-python3-wcwidth_0.1.8+dfsg1-3_all.deb ... 2023-09-30T01:31:30.876 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking python3-wcwidth (0.1.8+dfsg1-3) ... 2023-09-30T01:31:31.213 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package python3-pytest. 2023-09-30T01:31:31.231 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../105-python3-pytest_4.6.9-1_all.deb ... 2023-09-30T01:31:31.271 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking python3-pytest (4.6.9-1) ... 2023-09-30T01:31:31.641 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package python3-simplejson. 2023-09-30T01:31:31.659 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../106-python3-simplejson_3.16.0-2ubuntu2_amd64.deb ... 2023-09-30T01:31:31.699 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking python3-simplejson (3.16.0-2ubuntu2) ... 2023-09-30T01:31:31.961 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package qttranslations5-l10n. 2023-09-30T01:31:31.979 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../107-qttranslations5-l10n_5.12.8-0ubuntu1_all.deb ... 2023-09-30T01:31:32.010 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking qttranslations5-l10n (5.12.8-0ubuntu1) ... 2023-09-30T01:31:32.413 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package rbd-fuse. 2023-09-30T01:31:32.429 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../109-rbd-fuse_17.2.6-1165-g93dce5e1-1focal_amd64.deb ... 2023-09-30T01:31:32.478 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking rbd-fuse (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T01:31:32.791 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package radosgw. 2023-09-30T01:31:32.805 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up python3-attr (19.3.0-2) ... 2023-09-30T01:31:32.807 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../108-radosgw_17.2.6-1165-g93dce5e1-1focal_amd64.deb ... 2023-09-30T01:31:32.849 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking radosgw (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T01:31:33.108 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up librabbitmq4:amd64 (0.10.0-1) ... 2023-09-30T01:31:33.234 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up libdouble-conversion3:amd64 (3.1.5-4ubuntu1) ... 2023-09-30T01:31:33.385 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up nvme-cli (1.9-1ubuntu0.1) ... 2023-09-30T01:31:33.587 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up cephadm (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T01:31:34.253 INFO:teuthology.orchestra.run.smithi088.stdout:Adding system user cephadm....done 2023-09-30T01:31:34.447 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up fonts-mathjax (2.7.4+dfsg-1) ... 2023-09-30T01:31:34.573 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up python3-waitress (1.4.1-1ubuntu0.1) ... 2023-09-30T01:31:34.944 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package rbd-fuse. 2023-09-30T01:31:34.963 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../109-rbd-fuse_17.2.6-1165-g93dce5e1-1focal_amd64.deb ... 2023-09-30T01:31:35.002 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking rbd-fuse (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T01:31:35.166 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up python3-repoze.lru (0.7-2) ... 2023-09-30T01:31:35.330 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up python3-attr (19.3.0-2) ... 2023-09-30T01:31:35.459 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up libjs-mathjax (2.7.4+dfsg-1) ... 2023-09-30T01:31:35.610 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up liboath0 (2.6.1-1.3) ... 2023-09-30T01:31:35.625 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up librabbitmq4:amd64 (0.10.0-1) ... 2023-09-30T01:31:35.745 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up python3-py (1.8.1-1ubuntu0.1) ... 2023-09-30T01:31:35.751 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up libdouble-conversion3:amd64 (3.1.5-4ubuntu1) ... 2023-09-30T01:31:35.877 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up nvme-cli (1.9-1ubuntu0.1) ... 2023-09-30T01:31:36.080 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up python3-joblib (0.14.0-3) ... 2023-09-30T01:31:36.086 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up cephadm (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T01:31:36.473 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up python3-jwt (1.7.1-2ubuntu2.1) ... 2023-09-30T01:31:36.765 INFO:teuthology.orchestra.run.smithi107.stdout:Adding system user cephadm....done 2023-09-30T01:31:36.808 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up python3-cachetools (4.0.0-1) ... 2023-09-30T01:31:36.922 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up fonts-mathjax (2.7.4+dfsg-1) ... 2023-09-30T01:31:37.082 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up python3-waitress (1.4.1-1ubuntu0.1) ... 2023-09-30T01:31:37.117 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up python3-bcrypt (3.1.7-2ubuntu1) ... 2023-09-30T01:31:37.410 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up python3-pyinotify (0.9.6-1.2ubuntu1) ... 2023-09-30T01:31:37.425 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up python3-repoze.lru (0.7-2) ... 2023-09-30T01:31:37.719 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up python3-ceph-argparse (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T01:31:37.726 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up libjs-mathjax (2.7.4+dfsg-1) ... 2023-09-30T01:31:37.853 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up liboath0 (2.6.1-1.3) ... 2023-09-30T01:31:38.012 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up python3-py (1.8.1-1ubuntu0.1) ... 2023-09-30T01:31:38.020 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up python3-sklearn-lib (0.22.2.post1+dfsg-5) ... 2023-09-30T01:31:38.138 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up python3-markupsafe (1.1.0-1build2) ... 2023-09-30T01:31:38.355 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up python3-joblib (0.14.0-3) ... 2023-09-30T01:31:38.434 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up python3-webencodings (0.5.1-1ubuntu1) ... 2023-09-30T01:31:38.735 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up libpcre2-16-0:amd64 (10.34-7ubuntu0.1) ... 2023-09-30T01:31:38.783 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up python3-jwt (1.7.1-2ubuntu2.1) ... 2023-09-30T01:31:38.869 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up python3-psutil (5.5.1-1ubuntu4) ... 2023-09-30T01:31:39.075 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up python3-cachetools (4.0.0-1) ... 2023-09-30T01:31:39.238 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up python3-natsort (7.0.1-1) ... 2023-09-30T01:31:39.368 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up python3-bcrypt (3.1.7-2ubuntu1) ... 2023-09-30T01:31:39.522 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up python3-atomicwrites (1.1.5-2build1) ... 2023-09-30T01:31:39.679 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up python3-pyinotify (0.9.6-1.2ubuntu1) ... 2023-09-30T01:31:39.807 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up python3-simplejson (3.16.0-2ubuntu2) ... 2023-09-30T01:31:39.980 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up python3-ceph-argparse (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T01:31:40.158 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up python3-decorator (4.4.2-0ubuntu1) ... 2023-09-30T01:31:40.273 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up python3-sklearn-lib (0.22.2.post1+dfsg-5) ... 2023-09-30T01:31:40.432 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up python3-markupsafe (1.1.0-1build2) ... 2023-09-30T01:31:40.450 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up python3-jinja2 (2.10.1-2) ... 2023-09-30T01:31:40.725 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up python3-webencodings (0.5.1-1ubuntu1) ... 2023-09-30T01:31:40.804 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up python3-tempita (0.5.2-6) ... 2023-09-30T01:31:41.014 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up libpcre2-16-0:amd64 (10.34-7ubuntu0.1) ... 2023-09-30T01:31:41.096 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up python3-packaging (20.3-1) ... 2023-09-30T01:31:41.140 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up python3-psutil (5.5.1-1ubuntu4) ... 2023-09-30T01:31:41.406 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up python-pastedeploy-tpl (2.1.0-1) ... 2023-09-30T01:31:41.525 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up python3-natsort (7.0.1-1) ... 2023-09-30T01:31:41.556 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up qttranslations5-l10n (5.12.8-0ubuntu1) ... 2023-09-30T01:31:41.682 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up python3-wcwidth (0.1.8+dfsg1-3) ... 2023-09-30T01:31:41.842 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up python3-atomicwrites (1.1.5-2build1) ... 2023-09-30T01:31:41.958 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up python3-paste (3.4.0+dfsg1-1) ... 2023-09-30T01:31:42.102 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up python3-simplejson (3.16.0-2ubuntu2) ... 2023-09-30T01:31:42.343 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up python3-werkzeug (0.16.1+dfsg1-2ubuntu0.1) ... 2023-09-30T01:31:42.437 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up python3-decorator (4.4.2-0ubuntu1) ... 2023-09-30T01:31:42.712 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up python3-cryptography (2.8-3ubuntu0.1) ... 2023-09-30T01:31:42.715 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up python3-jinja2 (2.10.1-2) ... 2023-09-30T01:31:43.063 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up socat (1.7.3.3-2) ... 2023-09-30T01:31:43.083 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up python3-tempita (0.5.2-6) ... 2023-09-30T01:31:43.189 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up python3-ceph-common (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T01:31:43.376 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up python3-packaging (20.3-1) ... 2023-09-30T01:31:43.516 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up libqt5core5a:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2023-09-30T01:31:43.675 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up python3-html5lib (1.0.1-2) ... 2023-09-30T01:31:43.679 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up python-pastedeploy-tpl (2.1.0-1) ... 2023-09-30T01:31:43.839 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up qttranslations5-l10n (5.12.8-0ubuntu1) ... 2023-09-30T01:31:43.973 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up python3-wcwidth (0.1.8+dfsg1-3) ... 2023-09-30T01:31:44.009 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up librdkafka1:amd64 (1.2.1-1ubuntu1) ... 2023-09-30T01:31:44.119 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up xmlstarlet (1.6.1-2build1) ... 2023-09-30T01:31:44.258 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up python3-paste (3.4.0+dfsg1-1) ... 2023-09-30T01:31:44.270 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up python3-pluggy (0.13.0-2) ... 2023-09-30T01:31:44.579 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up python3-lxml:amd64 (4.5.0-1ubuntu0.5) ... 2023-09-30T01:31:44.643 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up python3-werkzeug (0.16.1+dfsg1-2ubuntu0.1) ... 2023-09-30T01:31:44.905 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up libqt5dbus5:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2023-09-30T01:31:45.003 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up python3-cryptography (2.8-3ubuntu0.1) ... 2023-09-30T01:31:45.040 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up liblua5.3-0:amd64 (5.3.3-1.1ubuntu2) ... 2023-09-30T01:31:45.166 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up python3-pyasn1 (0.4.2-3build1) ... 2023-09-30T01:31:45.363 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up socat (1.7.3.3-2) ... 2023-09-30T01:31:45.484 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up python3-singledispatch (3.4.0.3-2build1) ... 2023-09-30T01:31:45.489 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up python3-ceph-common (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T01:31:45.760 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up python3-logutils (0.3.3-7) ... 2023-09-30T01:31:45.807 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up libqt5core5a:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2023-09-30T01:31:45.942 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up python3-html5lib (1.0.1-2) ... 2023-09-30T01:31:46.095 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up liblttng-ust0:amd64 (2.11.0-1) ... 2023-09-30T01:31:46.221 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2023-09-30T01:31:46.285 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up librdkafka1:amd64 (1.2.1-1ubuntu1) ... 2023-09-30T01:31:46.402 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up xmlstarlet (1.6.1-2build1) ... 2023-09-30T01:31:46.513 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up python3-prettytable (0.7.2-5) ... 2023-09-30T01:31:46.528 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up python3-pluggy (0.13.0-2) ... 2023-09-30T01:31:46.808 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up python3-soupsieve (1.9.5+dfsg-1) ... 2023-09-30T01:31:46.829 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up python3-lxml:amd64 (4.5.0-1ubuntu0.5) ... 2023-09-30T01:31:47.139 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up libqt5dbus5:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2023-09-30T01:31:47.151 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up python3-websocket (0.53.0-2ubuntu1) ... 2023-09-30T01:31:47.274 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up liblua5.3-0:amd64 (5.3.3-1.1ubuntu2) ... 2023-09-30T01:31:47.400 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up python3-pyasn1 (0.4.2-3build1) ... 2023-09-30T01:31:47.550 INFO:teuthology.orchestra.run.smithi088.stdout:update-alternatives: using /usr/bin/python3-wsdump to provide /usr/bin/wsdump (wsdump) in auto mode 2023-09-30T01:31:47.701 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up python3-singledispatch (3.4.0.3-2build1) ... 2023-09-30T01:31:47.862 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up libonig5:amd64 (6.9.4-1) ... 2023-09-30T01:31:47.986 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up python3-logutils (0.3.3-7) ... 2023-09-30T01:31:47.988 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up python3-mako (1.1.0+ds1-1ubuntu2.1) ... 2023-09-30T01:31:48.295 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up liblttng-ust0:amd64 (2.11.0-1) ... 2023-09-30T01:31:48.298 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up python3-cherrypy3 (8.9.1-6) ... 2023-09-30T01:31:48.422 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2023-09-30T01:31:48.660 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up libjs-underscore (1.9.1~dfsg-1ubuntu0.20.04.1) ... 2023-09-30T01:31:48.732 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up python3-prettytable (0.7.2-5) ... 2023-09-30T01:31:48.786 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up liblbfgsb0:amd64 (3.0+dfsg.3-7build1) ... 2023-09-30T01:31:48.946 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up python3-webob (1:1.8.5-2) ... 2023-09-30T01:31:49.058 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up python3-soupsieve (1.9.5+dfsg-1) ... 2023-09-30T01:31:49.272 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up python3-blinker (1.4+dfsg1-0.3ubuntu1) ... 2023-09-30T01:31:49.370 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up python3-websocket (0.53.0-2ubuntu1) ... 2023-09-30T01:31:49.540 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up librados2 (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T01:31:49.699 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up python3-scipy (1.3.3-3build1) ... 2023-09-30T01:31:49.803 INFO:teuthology.orchestra.run.smithi107.stdout:update-alternatives: using /usr/bin/python3-wsdump to provide /usr/bin/wsdump (wsdump) in auto mode 2023-09-30T01:31:50.082 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up libonig5:amd64 (6.9.4-1) ... 2023-09-30T01:31:50.208 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up python3-mako (1.1.0+ds1-1ubuntu2.1) ... 2023-09-30T01:31:50.526 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up python3-cherrypy3 (8.9.1-6) ... 2023-09-30T01:31:50.869 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up libjs-underscore (1.9.1~dfsg-1ubuntu0.20.04.1) ... 2023-09-30T01:31:51.030 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up liblbfgsb0:amd64 (3.0+dfsg.3-7build1) ... 2023-09-30T01:31:51.158 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up python3-webob (1:1.8.5-2) ... 2023-09-30T01:31:51.158 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up libsqlite3-mod-ceph (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T01:31:51.285 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up libjq1:amd64 (1.6-1ubuntu0.20.04.1) ... 2023-09-30T01:31:51.444 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up python3-pytest (4.6.9-1) ... 2023-09-30T01:31:51.484 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up python3-blinker (1.4+dfsg1-0.3ubuntu1) ... 2023-09-30T01:31:51.785 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up librados2 (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T01:31:51.820 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up python3-openssl (19.0.0-1build1) ... 2023-09-30T01:31:51.945 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up python3-scipy (1.3.3-3build1) ... 2023-09-30T01:31:52.130 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up python3-pastedeploy (2.1.0-1) ... 2023-09-30T01:31:52.439 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up libqt5network5:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2023-09-30T01:31:52.565 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up python3-bs4 (4.8.2-1) ... 2023-09-30T01:31:52.900 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up libcephfs2 (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T01:31:53.026 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up python3-oauthlib (3.1.0-1ubuntu2) ... 2023-09-30T01:31:53.386 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up libradosstriper1 (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T01:31:53.420 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up libsqlite3-mod-ceph (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T01:31:53.537 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up python3-pyasn1-modules (0.2.1-0.2build1) ... 2023-09-30T01:31:53.546 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up libjq1:amd64 (1.6-1ubuntu0.20.04.1) ... 2023-09-30T01:31:53.664 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up python3-pytest (4.6.9-1) ... 2023-09-30T01:31:53.880 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up python3-sklearn (0.22.2.post1+dfsg-5) ... 2023-09-30T01:31:54.041 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up python3-openssl (19.0.0-1build1) ... 2023-09-30T01:31:54.342 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up python3-pastedeploy (2.1.0-1) ... 2023-09-30T01:31:54.626 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up libqt5network5:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2023-09-30T01:31:54.778 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up python3-bs4 (4.8.2-1) ... 2023-09-30T01:31:54.910 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up librbd1 (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T01:31:55.036 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up libjs-sphinxdoc (1.8.5-7ubuntu3) ... 2023-09-30T01:31:55.104 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up libcephfs2 (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T01:31:55.171 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up python-natsort-doc (7.0.1-1) ... 2023-09-30T01:31:55.255 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up python3-oauthlib (3.1.0-1ubuntu2) ... 2023-09-30T01:31:55.330 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up jq (1.6-1ubuntu0.20.04.1) ... 2023-09-30T01:31:55.456 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up python3-rsa (4.0-3) ... 2023-09-30T01:31:55.557 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up libradosstriper1 (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T01:31:55.683 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up python3-pyasn1-modules (0.2.1-0.2build1) ... 2023-09-30T01:31:55.765 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up libthrift-0.13.0 (0.13.0-2build2) ... 2023-09-30T01:31:55.913 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up python3-webtest (2.0.32-2) ... 2023-09-30T01:31:56.051 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up python3-sklearn (0.22.2.post1+dfsg-5) ... 2023-09-30T01:31:56.217 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up ceph-mgr-modules-core (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T01:31:56.368 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up ceph-fuse (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T01:31:56.595 INFO:teuthology.orchestra.run.smithi088.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2023-09-30T01:31:56.595 INFO:teuthology.orchestra.run.smithi088.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2023-09-30T01:31:57.053 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up libcephfs-dev (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T01:31:57.090 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up librbd1 (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T01:31:57.241 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up libjs-sphinxdoc (1.8.5-7ubuntu3) ... 2023-09-30T01:31:57.273 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up python3-requests-oauthlib (1.0.0-1.1build1) ... 2023-09-30T01:31:57.367 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up python-natsort-doc (7.0.1-1) ... 2023-09-30T01:31:57.493 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up jq (1.6-1ubuntu0.20.04.1) ... 2023-09-30T01:31:57.599 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up python3-pastescript (2.0.2-4) ... 2023-09-30T01:31:57.619 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up python3-rsa (4.0-3) ... 2023-09-30T01:31:57.920 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up libthrift-0.13.0 (0.13.0-2build2) ... 2023-09-30T01:31:57.942 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up python3-pecan (1.3.3-3ubuntu1) ... 2023-09-30T01:31:58.046 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up python3-webtest (2.0.32-2) ... 2023-09-30T01:31:58.320 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up python3-rados (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T01:31:58.381 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up ceph-mgr-modules-core (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T01:31:58.479 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up librgw2 (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T01:31:58.515 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up ceph-fuse (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T01:31:58.605 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up python3-rbd (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T01:31:58.756 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up rbd-fuse (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T01:31:58.764 INFO:teuthology.orchestra.run.smithi107.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2023-09-30T01:31:58.764 INFO:teuthology.orchestra.run.smithi107.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2023-09-30T01:31:58.899 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up python3-routes (2.4.1-2) ... 2023-09-30T01:31:59.192 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up python3-google-auth (1.5.1-2) ... 2023-09-30T01:31:59.225 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up libcephfs-dev (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T01:31:59.436 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up python3-requests-oauthlib (1.0.0-1.1build1) ... 2023-09-30T01:31:59.485 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up python3-rgw (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T01:31:59.619 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up python3-cephfs (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T01:31:59.730 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up python3-pastescript (2.0.2-4) ... 2023-09-30T01:31:59.754 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up python3-kubernetes (7.0.0~a1-2) ... 2023-09-30T01:32:00.065 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up python3-pecan (1.3.3-3ubuntu1) ... 2023-09-30T01:32:00.417 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up python3-rados (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T01:32:00.568 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up librgw2 (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T01:32:00.686 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up python3-rbd (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T01:32:00.690 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up ceph-common (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T01:32:00.804 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up rbd-fuse (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T01:32:00.930 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up python3-routes (2.4.1-2) ... 2023-09-30T01:32:01.248 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up python3-google-auth (1.5.1-2) ... 2023-09-30T01:32:01.292 INFO:teuthology.orchestra.run.smithi088.stdout:Adding group ceph....done 2023-09-30T01:32:01.550 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up python3-rgw (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T01:32:01.676 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up python3-cephfs (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T01:32:01.703 INFO:teuthology.orchestra.run.smithi088.stdout:Adding system user ceph....done 2023-09-30T01:32:01.802 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up python3-kubernetes (7.0.0~a1-2) ... 2023-09-30T01:32:01.880 INFO:teuthology.orchestra.run.smithi088.stdout:Setting system user ceph properties....done 2023-09-30T01:32:01.926 INFO:teuthology.orchestra.run.smithi088.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2023-09-30T01:32:02.068 INFO:teuthology.orchestra.run.smithi088.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2023-09-30T01:32:02.386 INFO:teuthology.orchestra.run.smithi088.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2023-09-30T01:32:02.722 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up ceph-common (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T01:32:02.842 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up ceph-test (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T01:32:02.993 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up radosgw (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T01:32:03.265 INFO:teuthology.orchestra.run.smithi088.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2023-09-30T01:32:03.265 INFO:teuthology.orchestra.run.smithi088.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2023-09-30T01:32:03.359 INFO:teuthology.orchestra.run.smithi107.stdout:Adding group ceph....done 2023-09-30T01:32:03.794 INFO:teuthology.orchestra.run.smithi107.stdout:Adding system user ceph....done 2023-09-30T01:32:03.920 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up ceph-base (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T01:32:04.029 INFO:teuthology.orchestra.run.smithi107.stdout:Setting system user ceph properties....done 2023-09-30T01:32:04.049 INFO:teuthology.orchestra.run.smithi107.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2023-09-30T01:32:04.188 INFO:teuthology.orchestra.run.smithi107.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2023-09-30T01:32:04.325 INFO:teuthology.orchestra.run.smithi088.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2023-09-30T01:32:04.525 INFO:teuthology.orchestra.run.smithi107.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2023-09-30T01:32:04.772 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up ceph-mds (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T01:32:04.983 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up ceph-test (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T01:32:04.992 INFO:teuthology.orchestra.run.smithi088.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2023-09-30T01:32:04.992 INFO:teuthology.orchestra.run.smithi088.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2023-09-30T01:32:05.109 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up radosgw (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T01:32:05.376 INFO:teuthology.orchestra.run.smithi107.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2023-09-30T01:32:05.376 INFO:teuthology.orchestra.run.smithi107.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2023-09-30T01:32:05.465 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up ceph-mgr (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T01:32:05.808 INFO:teuthology.orchestra.run.smithi088.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2023-09-30T01:32:05.808 INFO:teuthology.orchestra.run.smithi088.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2023-09-30T01:32:06.028 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up ceph-base (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T01:32:06.267 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up ceph-osd (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T01:32:06.377 INFO:teuthology.orchestra.run.smithi107.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2023-09-30T01:32:06.761 INFO:teuthology.orchestra.run.smithi088.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2023-09-30T01:32:06.761 INFO:teuthology.orchestra.run.smithi088.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2023-09-30T01:32:06.855 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up ceph-mds (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T01:32:07.092 INFO:teuthology.orchestra.run.smithi107.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2023-09-30T01:32:07.092 INFO:teuthology.orchestra.run.smithi107.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2023-09-30T01:32:07.230 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up ceph-mgr-k8sevents (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T01:32:07.361 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up ceph-mgr-diskprediction-local (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T01:32:07.504 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up ceph-mon (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T01:32:07.549 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up ceph-mgr (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T01:32:07.748 INFO:teuthology.orchestra.run.smithi088.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2023-09-30T01:32:07.748 INFO:teuthology.orchestra.run.smithi088.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2023-09-30T01:32:07.779 INFO:teuthology.orchestra.run.smithi107.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2023-09-30T01:32:07.779 INFO:teuthology.orchestra.run.smithi107.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2023-09-30T01:32:08.200 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up ceph-mgr-cephadm (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T01:32:08.225 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up ceph-osd (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T01:32:08.376 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up ceph (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T01:32:08.502 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up ceph-mgr-dashboard (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T01:32:08.670 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up ceph-volume (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T01:32:08.723 INFO:teuthology.orchestra.run.smithi107.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2023-09-30T01:32:08.723 INFO:teuthology.orchestra.run.smithi107.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2023-09-30T01:32:08.797 INFO:teuthology.orchestra.run.smithi088.stdout:Processing triggers for libc-bin (2.31-0ubuntu9.9) ... 2023-09-30T01:32:08.973 INFO:teuthology.orchestra.run.smithi088.stdout:Processing triggers for systemd (245.4-4ubuntu3.17) ... 2023-09-30T01:32:09.160 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up ceph-mgr-k8sevents (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T01:32:09.241 INFO:teuthology.orchestra.run.smithi088.stdout:Processing triggers for man-db (2.9.1-1) ... 2023-09-30T01:32:09.311 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up ceph-mgr-diskprediction-local (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T01:32:09.446 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up ceph-mon (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T01:32:09.669 INFO:teuthology.orchestra.run.smithi107.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2023-09-30T01:32:09.669 INFO:teuthology.orchestra.run.smithi107.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2023-09-30T01:32:10.117 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up ceph-mgr-cephadm (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T01:32:10.285 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up ceph (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T01:32:10.402 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up ceph-mgr-dashboard (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T01:32:10.545 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up ceph-volume (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T01:32:10.705 INFO:teuthology.orchestra.run.smithi107.stdout:Processing triggers for libc-bin (2.31-0ubuntu9.9) ... 2023-09-30T01:32:11.070 INFO:teuthology.orchestra.run.smithi107.stdout:Processing triggers for systemd (245.4-4ubuntu3.17) ... 2023-09-30T01:32:11.460 INFO:teuthology.orchestra.run.smithi107.stdout:Processing triggers for man-db (2.9.1-1) ... 2023-09-30T01:32:12.052 INFO:teuthology.orchestra.run.smithi088.stdout:Processing triggers for fontconfig (2.13.1-2ubuntu3) ... 2023-09-30T01:32:12.715 INFO:teuthology.orchestra.run.smithi088.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2023-09-30T01:32:12.717 DEBUG:teuthology.parallel:result is None 2023-09-30T01:32:14.354 INFO:teuthology.orchestra.run.smithi107.stdout:Processing triggers for fontconfig (2.13.1-2ubuntu3) ... 2023-09-30T01:32:15.002 INFO:teuthology.orchestra.run.smithi107.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2023-09-30T01:32:15.004 DEBUG:teuthology.parallel:result is None 2023-09-30T01:32:15.004 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&sha1=93dce5e17918f8aca65993a7a2dc40233a47b481 2023-09-30T01:32:15.235 DEBUG:teuthology.orchestra.run.smithi088:> dpkg-query -W -f '${Version}' ceph 2023-09-30T01:32:15.260 INFO:teuthology.orchestra.run.smithi088.stdout:17.2.6-1165-g93dce5e1-1focal 2023-09-30T01:32:15.260 INFO:teuthology.packaging:The installed version of ceph is 17.2.6-1165-g93dce5e1-1focal 2023-09-30T01:32:15.260 INFO:teuthology.task.install:The correct ceph version 17.2.6-1165-g93dce5e1-1focal is installed. 2023-09-30T01:32:15.261 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&sha1=93dce5e17918f8aca65993a7a2dc40233a47b481 2023-09-30T01:32:15.489 DEBUG:teuthology.orchestra.run.smithi107:> dpkg-query -W -f '${Version}' ceph 2023-09-30T01:32:15.512 INFO:teuthology.orchestra.run.smithi107.stdout:17.2.6-1165-g93dce5e1-1focal 2023-09-30T01:32:15.513 INFO:teuthology.packaging:The installed version of ceph is 17.2.6-1165-g93dce5e1-1focal 2023-09-30T01:32:15.513 INFO:teuthology.task.install:The correct ceph version 17.2.6-1165-g93dce5e1-1focal is installed. 2023-09-30T01:32:15.513 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2023-09-30T01:32:15.513 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2023-09-30T01:32:15.513 DEBUG:teuthology.orchestra.run.smithi088:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-09-30T01:32:15.530 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-09-30T01:32:15.530 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-09-30T01:32:15.571 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2023-09-30T01:32:15.571 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2023-09-30T01:32:15.571 DEBUG:teuthology.orchestra.run.smithi088:> sudo dd of=/usr/bin/daemon-helper 2023-09-30T01:32:15.587 DEBUG:teuthology.orchestra.run.smithi088:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-09-30T01:32:15.601 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-09-30T01:32:15.601 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd of=/usr/bin/daemon-helper 2023-09-30T01:32:15.617 DEBUG:teuthology.orchestra.run.smithi107:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-09-30T01:32:15.674 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2023-09-30T01:32:15.675 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2023-09-30T01:32:15.675 DEBUG:teuthology.orchestra.run.smithi088:> sudo dd of=/usr/bin/adjust-ulimits 2023-09-30T01:32:15.691 DEBUG:teuthology.orchestra.run.smithi088:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-09-30T01:32:15.702 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-09-30T01:32:15.702 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd of=/usr/bin/adjust-ulimits 2023-09-30T01:32:15.718 DEBUG:teuthology.orchestra.run.smithi107:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-09-30T01:32:15.729 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2023-09-30T01:32:15.730 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2023-09-30T01:32:15.730 DEBUG:teuthology.orchestra.run.smithi088:> sudo dd of=/usr/bin/stdin-killer 2023-09-30T01:32:15.747 DEBUG:teuthology.orchestra.run.smithi088:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-09-30T01:32:15.805 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-09-30T01:32:15.806 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd of=/usr/bin/stdin-killer 2023-09-30T01:32:15.821 DEBUG:teuthology.orchestra.run.smithi107:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-09-30T01:32:15.878 INFO:teuthology.run_tasks:Running task ceph... 2023-09-30T01:32:15.940 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2023-09-30T01:32:15.941 DEBUG:teuthology.orchestra.run.smithi088:> sudo chmod 777 /var/log/ceph 2023-09-30T01:32:15.943 DEBUG:teuthology.orchestra.run.smithi107:> sudo chmod 777 /var/log/ceph 2023-09-30T01:32:15.958 INFO:tasks.ceph:Disabling ceph logrotate... 2023-09-30T01:32:15.958 DEBUG:teuthology.orchestra.run.smithi088:> sudo rm -f -- /etc/logrotate.d/ceph 2023-09-30T01:32:15.960 DEBUG:teuthology.orchestra.run.smithi107:> sudo rm -f -- /etc/logrotate.d/ceph 2023-09-30T01:32:15.969 INFO:tasks.ceph:Creating extra log directories... 2023-09-30T01:32:15.970 DEBUG:teuthology.orchestra.run.smithi088:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2023-09-30T01:32:16.012 DEBUG:teuthology.orchestra.run.smithi107:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2023-09-30T01:32:16.039 INFO:tasks.ceph:Creating ceph cluster ceph... 2023-09-30T01:32:16.039 INFO:tasks.ceph:config {'conf': {'client': {'debug monc': 20, 'debug ms': 1}, 'global': {'mon client directed command retry': 5, 'mon election default strategy': 3, 'ms bind msgr1': False, 'ms bind msgr2': True, 'ms inject socket failures': 5000, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon debug extra checks': True, 'mon min osdmap epochs': 100, 'mon osdmap full prune enabled': True, 'mon osdmap full prune interval': 10, 'mon osdmap full prune min': 200, 'mon osdmap full prune txsize': 100, 'mon scrub inject crc mismatch': 0.01, 'mon scrub inject missing keys': 0.05, 'mon scrub interval': 300, 'paxos service trim min': 5}, 'osd': {'bluefs allocator': 'hybrid', 'bluestore allocator': 'hybrid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon client ping interval': 4, 'mon client ping timeout': 12, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd beacon report interval': 10, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random'}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'ScrubResult', 'scrub mismatch', 'overall HEALTH_', '\\(MGR_DOWN\\)', '\\(PG_AVAILABILITY\\)', '\\(SLOW_OPS\\)', 'slow request', '\\(OSD_SLOW_PING_TIME', 'overall HEALTH_', '\\(MON_DOWN\\)', '\\(OSDMAP_FLAGS\\)'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2023-09-30T01:32:16.039 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi/7406999', 'branch': 'wip-yuri4-testing-2023-09-29-0928-quincy', 'description': 'rados/monthrash/{ceph clusters/9-mons mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-hybrid rados supported-random-distro$/{ubuntu_latest} thrashers/many workloads/rados_mon_osdmap_prune}', 'email': 'yweinste@redhat.com', 'first_in_suite': False, 'job_id': '7406999', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'ktype': 'distro', 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 3, 'size': 10}}], 'os_type': 'ubuntu', 'os_version': '20.04', 'overrides': {'admin_socket': {'branch': 'wip-yuri4-testing-2023-09-29-0928-quincy'}, 'ceph': {'conf': {'client': {'debug monc': 20, 'debug ms': 1}, 'global': {'mon client directed command retry': 5, 'mon election default strategy': 3, 'ms bind msgr1': False, 'ms bind msgr2': True, 'ms inject socket failures': 5000, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon debug extra checks': True, 'mon min osdmap epochs': 100, 'mon osdmap full prune enabled': True, 'mon osdmap full prune interval': 10, 'mon osdmap full prune min': 200, 'mon osdmap full prune txsize': 100, 'mon scrub inject crc mismatch': 0.01, 'mon scrub inject missing keys': 0.05, 'mon scrub interval': 300, 'paxos service trim min': 5}, 'osd': {'bluefs allocator': 'hybrid', 'bluestore allocator': 'hybrid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon client ping interval': 4, 'mon client ping timeout': 12, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd beacon report interval': 10, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'ScrubResult', 'scrub mismatch', 'overall HEALTH_', '\\(MGR_DOWN\\)', '\\(PG_AVAILABILITY\\)', '\\(SLOW_OPS\\)', 'slow request', '\\(OSD_SLOW_PING_TIME', 'overall HEALTH_', '\\(MON_DOWN\\)', '\\(OSDMAP_FLAGS\\)'], 'sha1': '93dce5e17918f8aca65993a7a2dc40233a47b481'}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}, 'osd': {'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'install': {'ceph': {'flavor': 'default', 'sha1': '93dce5e17918f8aca65993a7a2dc40233a47b481'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-yuri4-testing-2023-09-29-0928-quincy', 'sha1': '93dce5e17918f8aca65993a7a2dc40233a47b481'}}, 'owner': 'scheduled_yuriw@teuthology', 'priority': 75, 'repo': 'https://github.com/ceph/ceph-ci.git', 'roles': [['mon.a', 'mon.b', 'mon.c', 'mon.d', 'mon.e', 'osd.0', 'osd.1', 'osd.2'], ['mon.f', 'mon.g', 'mon.h', 'mon.i', 'mgr.x', 'osd.3', 'osd.4', 'osd.5', 'client.0']], 'seed': 9810, 'sha1': '93dce5e17918f8aca65993a7a2dc40233a47b481', 'sleep_before_teardown': 0, 'subset': '111/120000', 'suite': 'rados', 'suite_branch': 'wip-yuri4-testing-2023-09-29-0928-quincy', 'suite_path': '/home/teuthworker/src/github.com_ceph_ceph-c_93dce5e17918f8aca65993a7a2dc40233a47b481/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/ceph/ceph-ci.git', 'suite_sha1': '93dce5e17918f8aca65993a7a2dc40233a47b481', 'targets': {'smithi088.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFyT/r7xuKfmg50w0OdQebr4A/ptJo1Zk6DyZNZNZM/q5rW+sznh+wvHqeT3e4YtCdZLaXyvMms8o8u8H5nXOYM=', 'smithi107.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFyT/r7xuKfmg50w0OdQebr4A/ptJo1Zk6DyZNZNZM/q5rW+sznh+wvHqeT3e4YtCdZLaXyvMms8o8u8H5nXOYM='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': None}, {'mon_thrash': {'freeze_mon_duration': 20, 'freeze_mon_probability': 10, 'revive_delay': 20, 'thrash_delay': 1, 'thrash_many': True}}, {'workunit': {'clients': {'client.0': ['mon/test_mon_osdmap_prune.sh']}}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '54e62bcbac4e53d9685e08328b790d3b20d71cae', 'timestamp': '2023-09-29_19:38:56', 'tube': 'smithi', 'user': 'yuriw', 'verbose': True, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2830'} 2023-09-30T01:32:16.039 DEBUG:teuthology.orchestra.run.smithi088:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2023-09-30T01:32:16.042 DEBUG:teuthology.orchestra.run.smithi107:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2023-09-30T01:32:16.049 DEBUG:teuthology.orchestra.run.smithi088:> sudo install -d -m0777 -- /var/run/ceph 2023-09-30T01:32:16.094 DEBUG:teuthology.orchestra.run.smithi107:> sudo install -d -m0777 -- /var/run/ceph 2023-09-30T01:32:16.111 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2023-09-30T01:32:16.111 DEBUG:teuthology.orchestra.run.smithi088:> dd if=/scratch_devs of=/dev/stdout 2023-09-30T01:32:16.116 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-09-30T01:32:16.117 DEBUG:teuthology.orchestra.run.smithi088:> stat /dev/vg_nvme/lv_1 2023-09-30T01:32:16.165 INFO:teuthology.orchestra.run.smithi088.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-09-30T01:32:16.165 INFO:teuthology.orchestra.run.smithi088.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-09-30T01:32:16.166 INFO:teuthology.orchestra.run.smithi088.stdout:Device: 6h/6d Inode: 358 Links: 1 2023-09-30T01:32:16.166 INFO:teuthology.orchestra.run.smithi088.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-09-30T01:32:16.166 INFO:teuthology.orchestra.run.smithi088.stdout:Access: 2023-09-30 01:28:37.031639872 +0000 2023-09-30T01:32:16.166 INFO:teuthology.orchestra.run.smithi088.stdout:Modify: 2023-09-30 01:28:36.739646601 +0000 2023-09-30T01:32:16.166 INFO:teuthology.orchestra.run.smithi088.stdout:Change: 2023-09-30 01:28:36.739646601 +0000 2023-09-30T01:32:16.166 INFO:teuthology.orchestra.run.smithi088.stdout: Birth: - 2023-09-30T01:32:16.166 DEBUG:teuthology.orchestra.run.smithi088:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-09-30T01:32:16.177 INFO:teuthology.orchestra.run.smithi088.stderr:1+0 records in 2023-09-30T01:32:16.177 INFO:teuthology.orchestra.run.smithi088.stderr:1+0 records out 2023-09-30T01:32:16.177 INFO:teuthology.orchestra.run.smithi088.stderr:512 bytes copied, 0.000396731 s, 1.3 MB/s 2023-09-30T01:32:16.178 DEBUG:teuthology.orchestra.run.smithi088:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-09-30T01:32:16.185 DEBUG:teuthology.orchestra.run.smithi088:> stat /dev/vg_nvme/lv_2 2023-09-30T01:32:16.233 INFO:teuthology.orchestra.run.smithi088.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-09-30T01:32:16.233 INFO:teuthology.orchestra.run.smithi088.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-09-30T01:32:16.233 INFO:teuthology.orchestra.run.smithi088.stdout:Device: 6h/6d Inode: 364 Links: 1 2023-09-30T01:32:16.233 INFO:teuthology.orchestra.run.smithi088.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-09-30T01:32:16.233 INFO:teuthology.orchestra.run.smithi088.stdout:Access: 2023-09-30 01:28:37.567627517 +0000 2023-09-30T01:32:16.234 INFO:teuthology.orchestra.run.smithi088.stdout:Modify: 2023-09-30 01:28:37.291633879 +0000 2023-09-30T01:32:16.234 INFO:teuthology.orchestra.run.smithi088.stdout:Change: 2023-09-30 01:28:37.291633879 +0000 2023-09-30T01:32:16.234 INFO:teuthology.orchestra.run.smithi088.stdout: Birth: - 2023-09-30T01:32:16.234 DEBUG:teuthology.orchestra.run.smithi088:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-09-30T01:32:16.244 INFO:teuthology.orchestra.run.smithi088.stderr:1+0 records in 2023-09-30T01:32:16.244 INFO:teuthology.orchestra.run.smithi088.stderr:1+0 records out 2023-09-30T01:32:16.245 INFO:teuthology.orchestra.run.smithi088.stderr:512 bytes copied, 0.000362431 s, 1.4 MB/s 2023-09-30T01:32:16.245 DEBUG:teuthology.orchestra.run.smithi088:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-09-30T01:32:16.252 DEBUG:teuthology.orchestra.run.smithi088:> stat /dev/vg_nvme/lv_3 2023-09-30T01:32:16.258 INFO:teuthology.orchestra.run.smithi088.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-09-30T01:32:16.258 INFO:teuthology.orchestra.run.smithi088.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-09-30T01:32:16.258 INFO:teuthology.orchestra.run.smithi088.stdout:Device: 6h/6d Inode: 374 Links: 1 2023-09-30T01:32:16.258 INFO:teuthology.orchestra.run.smithi088.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-09-30T01:32:16.258 INFO:teuthology.orchestra.run.smithi088.stdout:Access: 2023-09-30 01:28:38.099615255 +0000 2023-09-30T01:32:16.258 INFO:teuthology.orchestra.run.smithi088.stdout:Modify: 2023-09-30 01:28:37.807621985 +0000 2023-09-30T01:32:16.258 INFO:teuthology.orchestra.run.smithi088.stdout:Change: 2023-09-30 01:28:37.807621985 +0000 2023-09-30T01:32:16.258 INFO:teuthology.orchestra.run.smithi088.stdout: Birth: - 2023-09-30T01:32:16.259 DEBUG:teuthology.orchestra.run.smithi088:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-09-30T01:32:16.269 INFO:teuthology.orchestra.run.smithi088.stderr:1+0 records in 2023-09-30T01:32:16.269 INFO:teuthology.orchestra.run.smithi088.stderr:1+0 records out 2023-09-30T01:32:16.269 INFO:teuthology.orchestra.run.smithi088.stderr:512 bytes copied, 0.000400011 s, 1.3 MB/s 2023-09-30T01:32:16.270 DEBUG:teuthology.orchestra.run.smithi088:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-09-30T01:32:16.323 DEBUG:teuthology.orchestra.run.smithi088:> stat /dev/vg_nvme/lv_4 2023-09-30T01:32:16.330 INFO:teuthology.orchestra.run.smithi088.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-09-30T01:32:16.330 INFO:teuthology.orchestra.run.smithi088.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-09-30T01:32:16.330 INFO:teuthology.orchestra.run.smithi088.stdout:Device: 6h/6d Inode: 378 Links: 1 2023-09-30T01:32:16.330 INFO:teuthology.orchestra.run.smithi088.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-09-30T01:32:16.330 INFO:teuthology.orchestra.run.smithi088.stdout:Access: 2023-09-30 01:28:38.651602531 +0000 2023-09-30T01:32:16.330 INFO:teuthology.orchestra.run.smithi088.stdout:Modify: 2023-09-30 01:28:38.371608985 +0000 2023-09-30T01:32:16.330 INFO:teuthology.orchestra.run.smithi088.stdout:Change: 2023-09-30 01:28:38.371608985 +0000 2023-09-30T01:32:16.331 INFO:teuthology.orchestra.run.smithi088.stdout: Birth: - 2023-09-30T01:32:16.331 DEBUG:teuthology.orchestra.run.smithi088:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-09-30T01:32:16.342 INFO:teuthology.orchestra.run.smithi088.stderr:1+0 records in 2023-09-30T01:32:16.342 INFO:teuthology.orchestra.run.smithi088.stderr:1+0 records out 2023-09-30T01:32:16.342 INFO:teuthology.orchestra.run.smithi088.stderr:512 bytes copied, 0.000342628 s, 1.5 MB/s 2023-09-30T01:32:16.343 DEBUG:teuthology.orchestra.run.smithi088:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-09-30T01:32:16.395 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2023-09-30T01:32:16.395 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-09-30T01:32:16.395 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/scratch_devs of=/dev/stdout 2023-09-30T01:32:16.402 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-09-30T01:32:16.402 DEBUG:teuthology.orchestra.run.smithi107:> stat /dev/vg_nvme/lv_1 2023-09-30T01:32:16.410 INFO:teuthology.orchestra.run.smithi107.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-09-30T01:32:16.410 INFO:teuthology.orchestra.run.smithi107.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-09-30T01:32:16.410 INFO:teuthology.orchestra.run.smithi107.stdout:Device: 6h/6d Inode: 358 Links: 1 2023-09-30T01:32:16.410 INFO:teuthology.orchestra.run.smithi107.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-09-30T01:32:16.410 INFO:teuthology.orchestra.run.smithi107.stdout:Access: 2023-09-30 01:28:36.365869154 +0000 2023-09-30T01:32:16.410 INFO:teuthology.orchestra.run.smithi107.stdout:Modify: 2023-09-30 01:28:36.085875607 +0000 2023-09-30T01:32:16.411 INFO:teuthology.orchestra.run.smithi107.stdout:Change: 2023-09-30 01:28:36.085875607 +0000 2023-09-30T01:32:16.411 INFO:teuthology.orchestra.run.smithi107.stdout: Birth: - 2023-09-30T01:32:16.411 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-09-30T01:32:16.423 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-09-30T01:32:16.423 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-09-30T01:32:16.423 INFO:teuthology.orchestra.run.smithi107.stderr:512 bytes copied, 0.000398494 s, 1.3 MB/s 2023-09-30T01:32:16.424 DEBUG:teuthology.orchestra.run.smithi107:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-09-30T01:32:16.475 DEBUG:teuthology.orchestra.run.smithi107:> stat /dev/vg_nvme/lv_2 2023-09-30T01:32:16.483 INFO:teuthology.orchestra.run.smithi107.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-09-30T01:32:16.483 INFO:teuthology.orchestra.run.smithi107.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-09-30T01:32:16.483 INFO:teuthology.orchestra.run.smithi107.stdout:Device: 6h/6d Inode: 368 Links: 1 2023-09-30T01:32:16.483 INFO:teuthology.orchestra.run.smithi107.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-09-30T01:32:16.483 INFO:teuthology.orchestra.run.smithi107.stdout:Access: 2023-09-30 01:28:36.901856798 +0000 2023-09-30T01:32:16.483 INFO:teuthology.orchestra.run.smithi107.stdout:Modify: 2023-09-30 01:28:36.605863621 +0000 2023-09-30T01:32:16.483 INFO:teuthology.orchestra.run.smithi107.stdout:Change: 2023-09-30 01:28:36.605863621 +0000 2023-09-30T01:32:16.484 INFO:teuthology.orchestra.run.smithi107.stdout: Birth: - 2023-09-30T01:32:16.484 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-09-30T01:32:16.496 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-09-30T01:32:16.496 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-09-30T01:32:16.496 INFO:teuthology.orchestra.run.smithi107.stderr:512 bytes copied, 0.000356227 s, 1.4 MB/s 2023-09-30T01:32:16.497 DEBUG:teuthology.orchestra.run.smithi107:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-09-30T01:32:16.505 DEBUG:teuthology.orchestra.run.smithi107:> stat /dev/vg_nvme/lv_3 2023-09-30T01:32:16.510 INFO:teuthology.orchestra.run.smithi107.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-09-30T01:32:16.510 INFO:teuthology.orchestra.run.smithi107.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-09-30T01:32:16.510 INFO:teuthology.orchestra.run.smithi107.stdout:Device: 6h/6d Inode: 374 Links: 1 2023-09-30T01:32:16.510 INFO:teuthology.orchestra.run.smithi107.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-09-30T01:32:16.510 INFO:teuthology.orchestra.run.smithi107.stdout:Access: 2023-09-30 01:28:37.381845735 +0000 2023-09-30T01:32:16.510 INFO:teuthology.orchestra.run.smithi107.stdout:Modify: 2023-09-30 01:28:37.109852004 +0000 2023-09-30T01:32:16.510 INFO:teuthology.orchestra.run.smithi107.stdout:Change: 2023-09-30 01:28:37.109852004 +0000 2023-09-30T01:32:16.511 INFO:teuthology.orchestra.run.smithi107.stdout: Birth: - 2023-09-30T01:32:16.511 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-09-30T01:32:16.565 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-09-30T01:32:16.566 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-09-30T01:32:16.566 INFO:teuthology.orchestra.run.smithi107.stderr:512 bytes copied, 0.000392301 s, 1.3 MB/s 2023-09-30T01:32:16.567 DEBUG:teuthology.orchestra.run.smithi107:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-09-30T01:32:16.576 DEBUG:teuthology.orchestra.run.smithi107:> stat /dev/vg_nvme/lv_4 2023-09-30T01:32:16.581 INFO:teuthology.orchestra.run.smithi107.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-09-30T01:32:16.581 INFO:teuthology.orchestra.run.smithi107.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-09-30T01:32:16.581 INFO:teuthology.orchestra.run.smithi107.stdout:Device: 6h/6d Inode: 379 Links: 1 2023-09-30T01:32:16.581 INFO:teuthology.orchestra.run.smithi107.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-09-30T01:32:16.581 INFO:teuthology.orchestra.run.smithi107.stdout:Access: 2023-09-30 01:28:37.873834394 +0000 2023-09-30T01:32:16.582 INFO:teuthology.orchestra.run.smithi107.stdout:Modify: 2023-09-30 01:28:37.601840664 +0000 2023-09-30T01:32:16.582 INFO:teuthology.orchestra.run.smithi107.stdout:Change: 2023-09-30 01:28:37.601840664 +0000 2023-09-30T01:32:16.582 INFO:teuthology.orchestra.run.smithi107.stdout: Birth: - 2023-09-30T01:32:16.582 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-09-30T01:32:16.592 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-09-30T01:32:16.592 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-09-30T01:32:16.592 INFO:teuthology.orchestra.run.smithi107.stderr:512 bytes copied, 0.00036804 s, 1.4 MB/s 2023-09-30T01:32:16.593 DEBUG:teuthology.orchestra.run.smithi107:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-09-30T01:32:16.600 INFO:tasks.ceph:osd dev map: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'} 2023-09-30T01:32:16.600 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi088.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}, Remote(name='ubuntu@smithi107.front.sepia.ceph.com'): {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'}} 2023-09-30T01:32:16.600 INFO:tasks.ceph:Generating config... 2023-09-30T01:32:16.601 INFO:tasks.ceph:[client] debug monc = 20 2023-09-30T01:32:16.602 INFO:tasks.ceph:[client] debug ms = 1 2023-09-30T01:32:16.602 INFO:tasks.ceph:[global] mon client directed command retry = 5 2023-09-30T01:32:16.602 INFO:tasks.ceph:[global] mon election default strategy = 3 2023-09-30T01:32:16.602 INFO:tasks.ceph:[global] ms bind msgr1 = False 2023-09-30T01:32:16.602 INFO:tasks.ceph:[global] ms bind msgr2 = True 2023-09-30T01:32:16.602 INFO:tasks.ceph:[global] ms inject socket failures = 5000 2023-09-30T01:32:16.602 INFO:tasks.ceph:[global] ms type = async 2023-09-30T01:32:16.602 INFO:tasks.ceph:[mgr] debug mgr = 20 2023-09-30T01:32:16.602 INFO:tasks.ceph:[mgr] debug ms = 1 2023-09-30T01:32:16.602 INFO:tasks.ceph:[mon] debug mon = 20 2023-09-30T01:32:16.602 INFO:tasks.ceph:[mon] debug ms = 1 2023-09-30T01:32:16.603 INFO:tasks.ceph:[mon] debug paxos = 20 2023-09-30T01:32:16.603 INFO:tasks.ceph:[mon] mon debug extra checks = True 2023-09-30T01:32:16.603 INFO:tasks.ceph:[mon] mon min osdmap epochs = 100 2023-09-30T01:32:16.603 INFO:tasks.ceph:[mon] mon osdmap full prune enabled = True 2023-09-30T01:32:16.603 INFO:tasks.ceph:[mon] mon osdmap full prune interval = 10 2023-09-30T01:32:16.603 INFO:tasks.ceph:[mon] mon osdmap full prune min = 200 2023-09-30T01:32:16.603 INFO:tasks.ceph:[mon] mon osdmap full prune txsize = 100 2023-09-30T01:32:16.603 INFO:tasks.ceph:[mon] mon scrub inject crc mismatch = 0.01 2023-09-30T01:32:16.603 INFO:tasks.ceph:[mon] mon scrub inject missing keys = 0.05 2023-09-30T01:32:16.603 INFO:tasks.ceph:[mon] mon scrub interval = 300 2023-09-30T01:32:16.603 INFO:tasks.ceph:[mon] paxos service trim min = 5 2023-09-30T01:32:16.603 INFO:tasks.ceph:[osd] bluefs allocator = hybrid 2023-09-30T01:32:16.603 INFO:tasks.ceph:[osd] bluestore allocator = hybrid 2023-09-30T01:32:16.603 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2023-09-30T01:32:16.604 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2023-09-30T01:32:16.604 INFO:tasks.ceph:[osd] bluestore zero block detection = True 2023-09-30T01:32:16.604 INFO:tasks.ceph:[osd] debug bluefs = 20 2023-09-30T01:32:16.604 INFO:tasks.ceph:[osd] debug bluestore = 20 2023-09-30T01:32:16.604 INFO:tasks.ceph:[osd] debug ms = 1 2023-09-30T01:32:16.604 INFO:tasks.ceph:[osd] debug osd = 20 2023-09-30T01:32:16.604 INFO:tasks.ceph:[osd] debug rocksdb = 10 2023-09-30T01:32:16.604 INFO:tasks.ceph:[osd] mon client ping interval = 4 2023-09-30T01:32:16.604 INFO:tasks.ceph:[osd] mon client ping timeout = 12 2023-09-30T01:32:16.604 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2023-09-30T01:32:16.604 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2023-09-30T01:32:16.604 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2023-09-30T01:32:16.604 INFO:tasks.ceph:[osd] osd beacon report interval = 10 2023-09-30T01:32:16.605 INFO:tasks.ceph:[osd] osd debug verify cached snaps = True 2023-09-30T01:32:16.605 INFO:tasks.ceph:[osd] osd debug verify missing on start = True 2023-09-30T01:32:16.605 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2023-09-30T01:32:16.605 INFO:tasks.ceph:[osd] osd mclock override recovery settings = True 2023-09-30T01:32:16.605 INFO:tasks.ceph:[osd] osd mclock profile = high_recovery_ops 2023-09-30T01:32:16.605 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2023-09-30T01:32:16.605 INFO:tasks.ceph:[osd] osd op queue = debug_random 2023-09-30T01:32:16.605 INFO:tasks.ceph:[osd] osd op queue cut off = debug_random 2023-09-30T01:32:16.605 INFO:tasks.ceph:Setting up mon.a... 2023-09-30T01:32:16.605 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2023-09-30T01:32:16.640 INFO:teuthology.orchestra.run.smithi088.stdout:creating /etc/ceph/ceph.keyring 2023-09-30T01:32:16.642 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2023-09-30T01:32:16.675 DEBUG:teuthology.orchestra.run.smithi088:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-09-30T01:32:16.733 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.88'), ('mon.b', '[v2:172.21.15.88:3301,v1:172.21.15.88:6790]'), ('mon.c', '[v2:172.21.15.88:3302,v1:172.21.15.88:6791]'), ('mon.d', '[v2:172.21.15.88:3303,v1:172.21.15.88:6792]'), ('mon.e', '[v2:172.21.15.88:3304,v1:172.21.15.88:6793]'), ('mon.f', '172.21.15.107'), ('mon.g', '[v2:172.21.15.107:3301,v1:172.21.15.107:6790]'), ('mon.h', '[v2:172.21.15.107:3302,v1:172.21.15.107:6791]'), ('mon.i', '[v2:172.21.15.107:3303,v1:172.21.15.107:6792]')] 2023-09-30T01:32:16.734 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.88,[v2:172.21.15.88:3301,v1:172.21.15.88:6790],[v2:172.21.15.88:3302,v1:172.21.15.88:6791],[v2:172.21.15.88:3303,v1:172.21.15.88:6792],[v2:172.21.15.88:3304,v1:172.21.15.88:6793],172.21.15.107,[v2:172.21.15.107:3301,v1:172.21.15.107:6790],[v2:172.21.15.107:3302,v1:172.21.15.107:6791],[v2:172.21.15.107:3303,v1:172.21.15.107:6792]', 'mon client directed command retry': 5, 'mon election default strategy': 3, 'ms bind msgr1': False, 'ms bind msgr2': True, 'ms inject socket failures': 5000, 'ms type': 'async'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bluefs allocator': 'hybrid', 'bluestore allocator': 'hybrid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon client ping interval': 4, 'mon client ping timeout': 12, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd beacon report interval': 10, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random'}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon debug extra checks': True, 'mon min osdmap epochs': 100, 'mon osdmap full prune enabled': True, 'mon osdmap full prune interval': 10, 'mon osdmap full prune min': 200, 'mon osdmap full prune txsize': 100, 'mon scrub inject crc mismatch': 0.01, 'mon scrub inject missing keys': 0.05, 'mon scrub interval': 300, 'paxos service trim min': 5}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug monc': 20, 'debug ms': 1}, 'mon.a': {}, 'mon.b': {}, 'mon.c': {}, 'mon.d': {}, 'mon.e': {}, 'mon.f': {}, 'mon.g': {}, 'mon.h': {}, 'mon.i': {}} 2023-09-30T01:32:16.735 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2023-09-30T01:32:16.735 DEBUG:teuthology.orchestra.run.smithi088:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2023-09-30T01:32:16.739 DEBUG:teuthology.orchestra.run.smithi088:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.88 --addv b '[v2:172.21.15.88:3301,v1:172.21.15.88:6790]' --addv c '[v2:172.21.15.88:3302,v1:172.21.15.88:6791]' --addv d '[v2:172.21.15.88:3303,v1:172.21.15.88:6792]' --addv e '[v2:172.21.15.88:3304,v1:172.21.15.88:6793]' --add f 172.21.15.107 --addv g '[v2:172.21.15.107:3301,v1:172.21.15.107:6790]' --addv h '[v2:172.21.15.107:3302,v1:172.21.15.107:6791]' --addv i '[v2:172.21.15.107:3303,v1:172.21.15.107:6792]' --print /home/ubuntu/cephtest/ceph.monmap 2023-09-30T01:32:16.799 INFO:teuthology.orchestra.run.smithi088.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2023-09-30T01:32:16.799 INFO:teuthology.orchestra.run.smithi088.stdout:monmaptool: generated fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:32:16.799 INFO:teuthology.orchestra.run.smithi088.stdout:setting min_mon_release = octopus 2023-09-30T01:32:16.799 INFO:teuthology.orchestra.run.smithi088.stdout:epoch 0 2023-09-30T01:32:16.799 INFO:teuthology.orchestra.run.smithi088.stdout:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:32:16.800 INFO:teuthology.orchestra.run.smithi088.stdout:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:32:16.800 INFO:teuthology.orchestra.run.smithi088.stdout:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:32:16.800 INFO:teuthology.orchestra.run.smithi088.stdout:min_mon_release 15 (octopus) 2023-09-30T01:32:16.800 INFO:teuthology.orchestra.run.smithi088.stdout:election_strategy: 3 2023-09-30T01:32:16.800 INFO:teuthology.orchestra.run.smithi088.stdout:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:32:16.800 INFO:teuthology.orchestra.run.smithi088.stdout:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:32:16.800 INFO:teuthology.orchestra.run.smithi088.stdout:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:32:16.800 INFO:teuthology.orchestra.run.smithi088.stdout:3: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:32:16.800 INFO:teuthology.orchestra.run.smithi088.stdout:4: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:32:16.800 INFO:teuthology.orchestra.run.smithi088.stdout:5: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:32:16.800 INFO:teuthology.orchestra.run.smithi088.stdout:6: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:32:16.800 INFO:teuthology.orchestra.run.smithi088.stdout:7: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:32:16.801 INFO:teuthology.orchestra.run.smithi088.stdout:8: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:32:16.801 INFO:teuthology.orchestra.run.smithi088.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (9 monitors) 2023-09-30T01:32:16.802 DEBUG:teuthology.orchestra.run.smithi088:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2023-09-30T01:32:16.808 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 650badb5-899b-4472-88a6-cfdccee24d52... 2023-09-30T01:32:16.809 DEBUG:teuthology.orchestra.run.smithi088:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2023-09-30T01:32:16.852 DEBUG:teuthology.orchestra.run.smithi107:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2023-09-30T01:32:16.884 INFO:teuthology.orchestra.run.smithi088.stdout:[global] 2023-09-30T01:32:16.885 INFO:teuthology.orchestra.run.smithi088.stdout: chdir = "" 2023-09-30T01:32:16.885 INFO:teuthology.orchestra.run.smithi088.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-09-30T01:32:16.885 INFO:teuthology.orchestra.run.smithi088.stdout: auth supported = cephx 2023-09-30T01:32:16.885 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-09-30T01:32:16.885 INFO:teuthology.orchestra.run.smithi088.stdout: filestore xattr use omap = true 2023-09-30T01:32:16.885 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-09-30T01:32:16.885 INFO:teuthology.orchestra.run.smithi088.stdout: mon clock drift allowed = 1.000 2023-09-30T01:32:16.885 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-09-30T01:32:16.885 INFO:teuthology.orchestra.run.smithi088.stdout: osd crush chooseleaf type = 0 2023-09-30T01:32:16.885 INFO:teuthology.orchestra.run.smithi088.stdout: auth debug = true 2023-09-30T01:32:16.885 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-09-30T01:32:16.885 INFO:teuthology.orchestra.run.smithi088.stdout: ms die on old message = true 2023-09-30T01:32:16.886 INFO:teuthology.orchestra.run.smithi088.stdout: ms die on bug = true 2023-09-30T01:32:16.886 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-09-30T01:32:16.886 INFO:teuthology.orchestra.run.smithi088.stdout: mon max pg per osd = 10000 # >= luminous 2023-09-30T01:32:16.886 INFO:teuthology.orchestra.run.smithi088.stdout: mon pg warn max object skew = 0 2023-09-30T01:32:16.886 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-09-30T01:32:16.886 INFO:teuthology.orchestra.run.smithi088.stdout: # disable pg_autoscaler by default for new pools 2023-09-30T01:32:16.886 INFO:teuthology.orchestra.run.smithi088.stdout: osd_pool_default_pg_autoscale_mode = off 2023-09-30T01:32:16.886 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-09-30T01:32:16.886 INFO:teuthology.orchestra.run.smithi088.stdout: osd pool default size = 2 2023-09-30T01:32:16.886 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-09-30T01:32:16.886 INFO:teuthology.orchestra.run.smithi088.stdout: mon osd allow primary affinity = true 2023-09-30T01:32:16.887 INFO:teuthology.orchestra.run.smithi088.stdout: mon osd allow pg remap = true 2023-09-30T01:32:16.887 INFO:teuthology.orchestra.run.smithi088.stdout: mon warn on legacy crush tunables = false 2023-09-30T01:32:16.887 INFO:teuthology.orchestra.run.smithi088.stdout: mon warn on crush straw calc version zero = false 2023-09-30T01:32:16.887 INFO:teuthology.orchestra.run.smithi088.stdout: mon warn on no sortbitwise = false 2023-09-30T01:32:16.887 INFO:teuthology.orchestra.run.smithi088.stdout: mon warn on osd down out interval zero = false 2023-09-30T01:32:16.887 INFO:teuthology.orchestra.run.smithi088.stdout: mon warn on too few osds = false 2023-09-30T01:32:16.887 INFO:teuthology.orchestra.run.smithi088.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-09-30T01:32:16.887 INFO:teuthology.orchestra.run.smithi088.stdout: mon_warn_on_pool_no_redundancy = false 2023-09-30T01:32:16.887 INFO:teuthology.orchestra.run.smithi088.stdout: mon_allow_pool_size_one = true 2023-09-30T01:32:16.887 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-09-30T01:32:16.887 INFO:teuthology.orchestra.run.smithi088.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-09-30T01:32:16.888 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-09-30T01:32:16.888 INFO:teuthology.orchestra.run.smithi088.stdout: osd default data pool replay window = 5 2023-09-30T01:32:16.888 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-09-30T01:32:16.888 INFO:teuthology.orchestra.run.smithi088.stdout: mon allow pool delete = true 2023-09-30T01:32:16.888 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-09-30T01:32:16.888 INFO:teuthology.orchestra.run.smithi088.stdout: mon cluster log file level = debug 2023-09-30T01:32:16.888 INFO:teuthology.orchestra.run.smithi088.stdout: debug asserts on shutdown = true 2023-09-30T01:32:16.888 INFO:teuthology.orchestra.run.smithi088.stdout: mon health detail to clog = false 2023-09-30T01:32:16.888 INFO:teuthology.orchestra.run.smithi088.stdout: mon host = "172.21.15.88,[v2:172.21.15.88:3301,v1:172.21.15.88:6790],[v2:172.21.15.88:3302,v1:172.21.15.88:6791],[v2:172.21.15.88:3303,v1:172.21.15.88:6792],[v2:172.21.15.88:3304,v1:172.21.15.88:6793],172.21.15.107,[v2:172.21.15.107:3301,v1:172.21.15.107:6790],[v2:172.21.15.107:3302,v1:172.21.15.107:6791],[v2:172.21.15.107:3303,v1:172.21.15.107:6792]" 2023-09-30T01:32:16.888 INFO:teuthology.orchestra.run.smithi088.stdout: mon client directed command retry = 5 2023-09-30T01:32:16.888 INFO:teuthology.orchestra.run.smithi088.stdout: mon election default strategy = 3 2023-09-30T01:32:16.888 INFO:teuthology.orchestra.run.smithi088.stdout: ms bind msgr1 = False 2023-09-30T01:32:16.889 INFO:teuthology.orchestra.run.smithi088.stdout: ms bind msgr2 = True 2023-09-30T01:32:16.889 INFO:teuthology.orchestra.run.smithi088.stdout: ms inject socket failures = 5000 2023-09-30T01:32:16.889 INFO:teuthology.orchestra.run.smithi088.stdout: ms type = async 2023-09-30T01:32:16.889 INFO:teuthology.orchestra.run.smithi088.stdout: fsid = 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:32:16.889 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-09-30T01:32:16.889 INFO:teuthology.orchestra.run.smithi088.stdout:[osd] 2023-09-30T01:32:16.889 INFO:teuthology.orchestra.run.smithi088.stdout: osd journal size = 100 2023-09-30T01:32:16.889 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-09-30T01:32:16.889 INFO:teuthology.orchestra.run.smithi088.stdout: osd scrub load threshold = 5.0 2023-09-30T01:32:16.889 INFO:teuthology.orchestra.run.smithi088.stdout: osd scrub max interval = 600 2023-09-30T01:32:16.889 INFO:teuthology.orchestra.run.smithi088.stdout: osd mclock profile = high_recovery_ops 2023-09-30T01:32:16.889 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-09-30T01:32:16.890 INFO:teuthology.orchestra.run.smithi088.stdout: osd recover clone overlap = true 2023-09-30T01:32:16.890 INFO:teuthology.orchestra.run.smithi088.stdout: osd recovery max chunk = 1048576 2023-09-30T01:32:16.890 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-09-30T01:32:16.890 INFO:teuthology.orchestra.run.smithi088.stdout: osd debug shutdown = true 2023-09-30T01:32:16.890 INFO:teuthology.orchestra.run.smithi088.stdout: osd debug op order = true 2023-09-30T01:32:16.890 INFO:teuthology.orchestra.run.smithi088.stdout: osd debug verify stray on activate = true 2023-09-30T01:32:16.890 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-09-30T01:32:16.890 INFO:teuthology.orchestra.run.smithi088.stdout: osd open classes on start = true 2023-09-30T01:32:16.890 INFO:teuthology.orchestra.run.smithi088.stdout: osd debug pg log writeout = true 2023-09-30T01:32:16.890 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-09-30T01:32:16.890 INFO:teuthology.orchestra.run.smithi088.stdout: osd deep scrub update digest min age = 30 2023-09-30T01:32:16.890 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-09-30T01:32:16.891 INFO:teuthology.orchestra.run.smithi088.stdout: osd map max advance = 10 2023-09-30T01:32:16.891 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-09-30T01:32:16.891 INFO:teuthology.orchestra.run.smithi088.stdout: journal zero on create = true 2023-09-30T01:32:16.891 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-09-30T01:32:16.891 INFO:teuthology.orchestra.run.smithi088.stdout: filestore ondisk finisher threads = 3 2023-09-30T01:32:16.891 INFO:teuthology.orchestra.run.smithi088.stdout: filestore apply finisher threads = 3 2023-09-30T01:32:16.891 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-09-30T01:32:16.891 INFO:teuthology.orchestra.run.smithi088.stdout: bdev debug aio = true 2023-09-30T01:32:16.891 INFO:teuthology.orchestra.run.smithi088.stdout: osd debug misdirected ops = true 2023-09-30T01:32:16.891 INFO:teuthology.orchestra.run.smithi088.stdout: bluefs allocator = hybrid 2023-09-30T01:32:16.891 INFO:teuthology.orchestra.run.smithi088.stdout: bluestore allocator = hybrid 2023-09-30T01:32:16.892 INFO:teuthology.orchestra.run.smithi088.stdout: bluestore block size = 96636764160 2023-09-30T01:32:16.892 INFO:teuthology.orchestra.run.smithi088.stdout: bluestore fsck on mount = True 2023-09-30T01:32:16.892 INFO:teuthology.orchestra.run.smithi088.stdout: bluestore zero block detection = True 2023-09-30T01:32:16.892 INFO:teuthology.orchestra.run.smithi088.stdout: debug bluefs = 20 2023-09-30T01:32:16.892 INFO:teuthology.orchestra.run.smithi088.stdout: debug bluestore = 20 2023-09-30T01:32:16.892 INFO:teuthology.orchestra.run.smithi088.stdout: debug ms = 1 2023-09-30T01:32:16.892 INFO:teuthology.orchestra.run.smithi088.stdout: debug osd = 20 2023-09-30T01:32:16.892 INFO:teuthology.orchestra.run.smithi088.stdout: debug rocksdb = 10 2023-09-30T01:32:16.892 INFO:teuthology.orchestra.run.smithi088.stdout: mon client ping interval = 4 2023-09-30T01:32:16.892 INFO:teuthology.orchestra.run.smithi088.stdout: mon client ping timeout = 12 2023-09-30T01:32:16.892 INFO:teuthology.orchestra.run.smithi088.stdout: mon osd backfillfull_ratio = 0.85 2023-09-30T01:32:16.892 INFO:teuthology.orchestra.run.smithi088.stdout: mon osd full ratio = 0.9 2023-09-30T01:32:16.893 INFO:teuthology.orchestra.run.smithi088.stdout: mon osd nearfull ratio = 0.8 2023-09-30T01:32:16.893 INFO:teuthology.orchestra.run.smithi088.stdout: osd beacon report interval = 10 2023-09-30T01:32:16.893 INFO:teuthology.orchestra.run.smithi088.stdout: osd debug verify cached snaps = True 2023-09-30T01:32:16.893 INFO:teuthology.orchestra.run.smithi088.stdout: osd debug verify missing on start = True 2023-09-30T01:32:16.893 INFO:teuthology.orchestra.run.smithi088.stdout: osd failsafe full ratio = 0.95 2023-09-30T01:32:16.893 INFO:teuthology.orchestra.run.smithi088.stdout: osd mclock override recovery settings = True 2023-09-30T01:32:16.893 INFO:teuthology.orchestra.run.smithi088.stdout: osd objectstore = bluestore 2023-09-30T01:32:16.893 INFO:teuthology.orchestra.run.smithi088.stdout: osd op queue = debug_random 2023-09-30T01:32:16.893 INFO:teuthology.orchestra.run.smithi088.stdout: osd op queue cut off = debug_random 2023-09-30T01:32:16.893 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-09-30T01:32:16.893 INFO:teuthology.orchestra.run.smithi088.stdout:[mgr] 2023-09-30T01:32:16.894 INFO:teuthology.orchestra.run.smithi088.stdout: debug ms = 1 2023-09-30T01:32:16.894 INFO:teuthology.orchestra.run.smithi088.stdout: debug mgr = 20 2023-09-30T01:32:16.894 INFO:teuthology.orchestra.run.smithi088.stdout: debug mon = 20 2023-09-30T01:32:16.894 INFO:teuthology.orchestra.run.smithi088.stdout: debug auth = 20 2023-09-30T01:32:16.894 INFO:teuthology.orchestra.run.smithi088.stdout: mon reweight min pgs per osd = 4 2023-09-30T01:32:16.894 INFO:teuthology.orchestra.run.smithi088.stdout: mon reweight min bytes per osd = 10 2023-09-30T01:32:16.894 INFO:teuthology.orchestra.run.smithi088.stdout: mgr/telemetry/nag = false 2023-09-30T01:32:16.894 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-09-30T01:32:16.895 INFO:teuthology.orchestra.run.smithi088.stdout:[mon] 2023-09-30T01:32:16.895 INFO:teuthology.orchestra.run.smithi088.stdout: debug ms = 1 2023-09-30T01:32:16.895 INFO:teuthology.orchestra.run.smithi088.stdout: debug mon = 20 2023-09-30T01:32:16.895 INFO:teuthology.orchestra.run.smithi088.stdout: debug paxos = 20 2023-09-30T01:32:16.895 INFO:teuthology.orchestra.run.smithi088.stdout: debug auth = 20 2023-09-30T01:32:16.895 INFO:teuthology.orchestra.run.smithi088.stdout: mon data avail warn = 5 2023-09-30T01:32:16.895 INFO:teuthology.orchestra.run.smithi088.stdout: mon mgr mkfs grace = 240 2023-09-30T01:32:16.895 INFO:teuthology.orchestra.run.smithi088.stdout: mon reweight min pgs per osd = 4 2023-09-30T01:32:16.896 INFO:teuthology.orchestra.run.smithi088.stdout: mon osd reporter subtree level = osd 2023-09-30T01:32:16.896 INFO:teuthology.orchestra.run.smithi088.stdout: mon osd prime pg temp = true 2023-09-30T01:32:16.896 INFO:teuthology.orchestra.run.smithi088.stdout: mon reweight min bytes per osd = 10 2023-09-30T01:32:16.896 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-09-30T01:32:16.896 INFO:teuthology.orchestra.run.smithi088.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-09-30T01:32:16.896 INFO:teuthology.orchestra.run.smithi088.stdout: auth mon ticket ttl = 660 # 11m 2023-09-30T01:32:16.896 INFO:teuthology.orchestra.run.smithi088.stdout: auth service ticket ttl = 240 # 4m 2023-09-30T01:32:16.896 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-09-30T01:32:16.896 INFO:teuthology.orchestra.run.smithi088.stdout: # don't complain about insecure global_id in the test suite 2023-09-30T01:32:16.896 INFO:teuthology.orchestra.run.smithi088.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-09-30T01:32:16.896 INFO:teuthology.orchestra.run.smithi088.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-09-30T01:32:16.897 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-09-30T01:32:16.897 INFO:teuthology.orchestra.run.smithi088.stdout: # 1m isn't quite enough 2023-09-30T01:32:16.897 INFO:teuthology.orchestra.run.smithi088.stdout: mon_down_mkfs_grace = 2m 2023-09-30T01:32:16.897 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-09-30T01:32:16.897 INFO:teuthology.orchestra.run.smithi088.stdout: mon_warn_on_filestore_osds = false 2023-09-30T01:32:16.897 INFO:teuthology.orchestra.run.smithi088.stdout: mon debug extra checks = True 2023-09-30T01:32:16.897 INFO:teuthology.orchestra.run.smithi088.stdout: mon min osdmap epochs = 100 2023-09-30T01:32:16.897 INFO:teuthology.orchestra.run.smithi088.stdout: mon osdmap full prune enabled = True 2023-09-30T01:32:16.897 INFO:teuthology.orchestra.run.smithi088.stdout: mon osdmap full prune interval = 10 2023-09-30T01:32:16.897 INFO:teuthology.orchestra.run.smithi088.stdout: mon osdmap full prune min = 200 2023-09-30T01:32:16.898 INFO:teuthology.orchestra.run.smithi088.stdout: mon osdmap full prune txsize = 100 2023-09-30T01:32:16.898 INFO:teuthology.orchestra.run.smithi088.stdout: mon scrub inject crc mismatch = 0.01 2023-09-30T01:32:16.898 INFO:teuthology.orchestra.run.smithi088.stdout: mon scrub inject missing keys = 0.05 2023-09-30T01:32:16.898 INFO:teuthology.orchestra.run.smithi088.stdout: mon scrub interval = 300 2023-09-30T01:32:16.898 INFO:teuthology.orchestra.run.smithi088.stdout: paxos service trim min = 5 2023-09-30T01:32:16.898 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-09-30T01:32:16.898 INFO:teuthology.orchestra.run.smithi088.stdout:[client] 2023-09-30T01:32:16.898 INFO:teuthology.orchestra.run.smithi088.stdout: rgw cache enabled = true 2023-09-30T01:32:16.899 INFO:teuthology.orchestra.run.smithi088.stdout: rgw enable ops log = true 2023-09-30T01:32:16.899 INFO:teuthology.orchestra.run.smithi088.stdout: rgw enable usage log = true 2023-09-30T01:32:16.899 INFO:teuthology.orchestra.run.smithi088.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-09-30T01:32:16.899 INFO:teuthology.orchestra.run.smithi088.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-09-30T01:32:16.899 INFO:teuthology.orchestra.run.smithi088.stdout: debug monc = 20 2023-09-30T01:32:16.899 INFO:teuthology.orchestra.run.smithi088.stdout: debug ms = 1 2023-09-30T01:32:16.899 INFO:teuthology.orchestra.run.smithi088.stdout:[mon.a] 2023-09-30T01:32:16.899 INFO:teuthology.orchestra.run.smithi088.stdout:[mon.b] 2023-09-30T01:32:16.899 INFO:teuthology.orchestra.run.smithi088.stdout:[mon.c] 2023-09-30T01:32:16.899 INFO:teuthology.orchestra.run.smithi088.stdout:[mon.d] 2023-09-30T01:32:16.900 INFO:teuthology.orchestra.run.smithi088.stdout:[mon.e] 2023-09-30T01:32:16.900 INFO:teuthology.orchestra.run.smithi088.stdout:[mon.f] 2023-09-30T01:32:16.900 INFO:teuthology.orchestra.run.smithi088.stdout:[mon.g] 2023-09-30T01:32:16.900 INFO:teuthology.orchestra.run.smithi088.stdout:[mon.h] 2023-09-30T01:32:16.900 INFO:teuthology.orchestra.run.smithi088.stdout:[mon.i] 2023-09-30T01:32:16.901 INFO:teuthology.orchestra.run.smithi107.stdout:[global] 2023-09-30T01:32:16.902 INFO:teuthology.orchestra.run.smithi107.stdout: chdir = "" 2023-09-30T01:32:16.902 INFO:teuthology.orchestra.run.smithi107.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-09-30T01:32:16.902 INFO:teuthology.orchestra.run.smithi107.stdout: auth supported = cephx 2023-09-30T01:32:16.902 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-09-30T01:32:16.902 INFO:teuthology.orchestra.run.smithi107.stdout: filestore xattr use omap = true 2023-09-30T01:32:16.902 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-09-30T01:32:16.902 INFO:teuthology.orchestra.run.smithi107.stdout: mon clock drift allowed = 1.000 2023-09-30T01:32:16.902 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-09-30T01:32:16.902 INFO:teuthology.orchestra.run.smithi107.stdout: osd crush chooseleaf type = 0 2023-09-30T01:32:16.902 INFO:teuthology.orchestra.run.smithi107.stdout: auth debug = true 2023-09-30T01:32:16.903 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-09-30T01:32:16.903 INFO:teuthology.orchestra.run.smithi107.stdout: ms die on old message = true 2023-09-30T01:32:16.903 INFO:teuthology.orchestra.run.smithi107.stdout: ms die on bug = true 2023-09-30T01:32:16.903 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-09-30T01:32:16.903 INFO:teuthology.orchestra.run.smithi107.stdout: mon max pg per osd = 10000 # >= luminous 2023-09-30T01:32:16.903 INFO:teuthology.orchestra.run.smithi107.stdout: mon pg warn max object skew = 0 2023-09-30T01:32:16.903 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-09-30T01:32:16.903 INFO:teuthology.orchestra.run.smithi107.stdout: # disable pg_autoscaler by default for new pools 2023-09-30T01:32:16.903 INFO:teuthology.orchestra.run.smithi107.stdout: osd_pool_default_pg_autoscale_mode = off 2023-09-30T01:32:16.903 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-09-30T01:32:16.903 INFO:teuthology.orchestra.run.smithi107.stdout: osd pool default size = 2 2023-09-30T01:32:16.903 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-09-30T01:32:16.904 INFO:teuthology.orchestra.run.smithi107.stdout: mon osd allow primary affinity = true 2023-09-30T01:32:16.904 INFO:teuthology.orchestra.run.smithi107.stdout: mon osd allow pg remap = true 2023-09-30T01:32:16.904 INFO:teuthology.orchestra.run.smithi107.stdout: mon warn on legacy crush tunables = false 2023-09-30T01:32:16.904 INFO:teuthology.orchestra.run.smithi107.stdout: mon warn on crush straw calc version zero = false 2023-09-30T01:32:16.904 INFO:teuthology.orchestra.run.smithi107.stdout: mon warn on no sortbitwise = false 2023-09-30T01:32:16.904 INFO:teuthology.orchestra.run.smithi107.stdout: mon warn on osd down out interval zero = false 2023-09-30T01:32:16.904 INFO:teuthology.orchestra.run.smithi107.stdout: mon warn on too few osds = false 2023-09-30T01:32:16.904 INFO:teuthology.orchestra.run.smithi107.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-09-30T01:32:16.904 INFO:teuthology.orchestra.run.smithi107.stdout: mon_warn_on_pool_no_redundancy = false 2023-09-30T01:32:16.904 INFO:teuthology.orchestra.run.smithi107.stdout: mon_allow_pool_size_one = true 2023-09-30T01:32:16.904 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-09-30T01:32:16.905 INFO:teuthology.orchestra.run.smithi107.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-09-30T01:32:16.905 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-09-30T01:32:16.905 INFO:teuthology.orchestra.run.smithi107.stdout: osd default data pool replay window = 5 2023-09-30T01:32:16.905 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-09-30T01:32:16.905 INFO:teuthology.orchestra.run.smithi107.stdout: mon allow pool delete = true 2023-09-30T01:32:16.905 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-09-30T01:32:16.905 INFO:teuthology.orchestra.run.smithi107.stdout: mon cluster log file level = debug 2023-09-30T01:32:16.905 INFO:teuthology.orchestra.run.smithi107.stdout: debug asserts on shutdown = true 2023-09-30T01:32:16.905 INFO:teuthology.orchestra.run.smithi107.stdout: mon health detail to clog = false 2023-09-30T01:32:16.905 INFO:teuthology.orchestra.run.smithi107.stdout: mon host = "172.21.15.88,[v2:172.21.15.88:3301,v1:172.21.15.88:6790],[v2:172.21.15.88:3302,v1:172.21.15.88:6791],[v2:172.21.15.88:3303,v1:172.21.15.88:6792],[v2:172.21.15.88:3304,v1:172.21.15.88:6793],172.21.15.107,[v2:172.21.15.107:3301,v1:172.21.15.107:6790],[v2:172.21.15.107:3302,v1:172.21.15.107:6791],[v2:172.21.15.107:3303,v1:172.21.15.107:6792]" 2023-09-30T01:32:16.905 INFO:teuthology.orchestra.run.smithi107.stdout: mon client directed command retry = 5 2023-09-30T01:32:16.906 INFO:teuthology.orchestra.run.smithi107.stdout: mon election default strategy = 3 2023-09-30T01:32:16.906 INFO:teuthology.orchestra.run.smithi107.stdout: ms bind msgr1 = False 2023-09-30T01:32:16.906 INFO:teuthology.orchestra.run.smithi107.stdout: ms bind msgr2 = True 2023-09-30T01:32:16.906 INFO:teuthology.orchestra.run.smithi107.stdout: ms inject socket failures = 5000 2023-09-30T01:32:16.906 INFO:teuthology.orchestra.run.smithi107.stdout: ms type = async 2023-09-30T01:32:16.906 INFO:teuthology.orchestra.run.smithi107.stdout: fsid = 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:32:16.906 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-09-30T01:32:16.906 INFO:teuthology.orchestra.run.smithi107.stdout:[osd] 2023-09-30T01:32:16.906 INFO:teuthology.orchestra.run.smithi107.stdout: osd journal size = 100 2023-09-30T01:32:16.906 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-09-30T01:32:16.906 INFO:teuthology.orchestra.run.smithi107.stdout: osd scrub load threshold = 5.0 2023-09-30T01:32:16.906 INFO:teuthology.orchestra.run.smithi107.stdout: osd scrub max interval = 600 2023-09-30T01:32:16.906 INFO:teuthology.orchestra.run.smithi107.stdout: osd mclock profile = high_recovery_ops 2023-09-30T01:32:16.906 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-09-30T01:32:16.907 INFO:teuthology.orchestra.run.smithi107.stdout: osd recover clone overlap = true 2023-09-30T01:32:16.907 INFO:teuthology.orchestra.run.smithi107.stdout: osd recovery max chunk = 1048576 2023-09-30T01:32:16.907 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-09-30T01:32:16.907 INFO:teuthology.orchestra.run.smithi107.stdout: osd debug shutdown = true 2023-09-30T01:32:16.907 INFO:teuthology.orchestra.run.smithi107.stdout: osd debug op order = true 2023-09-30T01:32:16.907 INFO:teuthology.orchestra.run.smithi107.stdout: osd debug verify stray on activate = true 2023-09-30T01:32:16.907 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-09-30T01:32:16.907 INFO:teuthology.orchestra.run.smithi107.stdout: osd open classes on start = true 2023-09-30T01:32:16.907 INFO:teuthology.orchestra.run.smithi107.stdout: osd debug pg log writeout = true 2023-09-30T01:32:16.907 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-09-30T01:32:16.907 INFO:teuthology.orchestra.run.smithi107.stdout: osd deep scrub update digest min age = 30 2023-09-30T01:32:16.907 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-09-30T01:32:16.907 INFO:teuthology.orchestra.run.smithi107.stdout: osd map max advance = 10 2023-09-30T01:32:16.908 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-09-30T01:32:16.908 INFO:teuthology.orchestra.run.smithi107.stdout: journal zero on create = true 2023-09-30T01:32:16.908 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-09-30T01:32:16.908 INFO:teuthology.orchestra.run.smithi107.stdout: filestore ondisk finisher threads = 3 2023-09-30T01:32:16.908 INFO:teuthology.orchestra.run.smithi107.stdout: filestore apply finisher threads = 3 2023-09-30T01:32:16.908 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-09-30T01:32:16.908 INFO:teuthology.orchestra.run.smithi107.stdout: bdev debug aio = true 2023-09-30T01:32:16.908 INFO:teuthology.orchestra.run.smithi107.stdout: osd debug misdirected ops = true 2023-09-30T01:32:16.908 INFO:teuthology.orchestra.run.smithi107.stdout: bluefs allocator = hybrid 2023-09-30T01:32:16.908 INFO:teuthology.orchestra.run.smithi107.stdout: bluestore allocator = hybrid 2023-09-30T01:32:16.908 INFO:teuthology.orchestra.run.smithi107.stdout: bluestore block size = 96636764160 2023-09-30T01:32:16.908 INFO:teuthology.orchestra.run.smithi107.stdout: bluestore fsck on mount = True 2023-09-30T01:32:16.908 INFO:teuthology.orchestra.run.smithi107.stdout: bluestore zero block detection = True 2023-09-30T01:32:16.908 INFO:teuthology.orchestra.run.smithi107.stdout: debug bluefs = 20 2023-09-30T01:32:16.909 INFO:teuthology.orchestra.run.smithi107.stdout: debug bluestore = 20 2023-09-30T01:32:16.909 INFO:teuthology.orchestra.run.smithi107.stdout: debug ms = 1 2023-09-30T01:32:16.909 INFO:teuthology.orchestra.run.smithi107.stdout: debug osd = 20 2023-09-30T01:32:16.909 INFO:teuthology.orchestra.run.smithi107.stdout: debug rocksdb = 10 2023-09-30T01:32:16.909 INFO:teuthology.orchestra.run.smithi107.stdout: mon client ping interval = 4 2023-09-30T01:32:16.909 INFO:teuthology.orchestra.run.smithi107.stdout: mon client ping timeout = 12 2023-09-30T01:32:16.909 INFO:teuthology.orchestra.run.smithi107.stdout: mon osd backfillfull_ratio = 0.85 2023-09-30T01:32:16.909 INFO:teuthology.orchestra.run.smithi107.stdout: mon osd full ratio = 0.9 2023-09-30T01:32:16.909 INFO:teuthology.orchestra.run.smithi107.stdout: mon osd nearfull ratio = 0.8 2023-09-30T01:32:16.909 INFO:teuthology.orchestra.run.smithi107.stdout: osd beacon report interval = 10 2023-09-30T01:32:16.909 INFO:teuthology.orchestra.run.smithi107.stdout: osd debug verify cached snaps = True 2023-09-30T01:32:16.909 INFO:teuthology.orchestra.run.smithi107.stdout: osd debug verify missing on start = True 2023-09-30T01:32:16.909 INFO:teuthology.orchestra.run.smithi107.stdout: osd failsafe full ratio = 0.95 2023-09-30T01:32:16.910 INFO:teuthology.orchestra.run.smithi107.stdout: osd mclock override recovery settings = True 2023-09-30T01:32:16.910 INFO:teuthology.orchestra.run.smithi107.stdout: osd objectstore = bluestore 2023-09-30T01:32:16.910 INFO:teuthology.orchestra.run.smithi107.stdout: osd op queue = debug_random 2023-09-30T01:32:16.910 INFO:teuthology.orchestra.run.smithi107.stdout: osd op queue cut off = debug_random 2023-09-30T01:32:16.910 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-09-30T01:32:16.910 INFO:teuthology.orchestra.run.smithi107.stdout:[mgr] 2023-09-30T01:32:16.910 INFO:teuthology.orchestra.run.smithi107.stdout: debug ms = 1 2023-09-30T01:32:16.910 INFO:teuthology.orchestra.run.smithi107.stdout: debug mgr = 20 2023-09-30T01:32:16.910 INFO:teuthology.orchestra.run.smithi107.stdout: debug mon = 20 2023-09-30T01:32:16.910 INFO:teuthology.orchestra.run.smithi107.stdout: debug auth = 20 2023-09-30T01:32:16.910 INFO:teuthology.orchestra.run.smithi107.stdout: mon reweight min pgs per osd = 4 2023-09-30T01:32:16.911 INFO:teuthology.orchestra.run.smithi107.stdout: mon reweight min bytes per osd = 10 2023-09-30T01:32:16.911 INFO:teuthology.orchestra.run.smithi107.stdout: mgr/telemetry/nag = false 2023-09-30T01:32:16.911 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-09-30T01:32:16.911 INFO:teuthology.orchestra.run.smithi107.stdout:[mon] 2023-09-30T01:32:16.911 INFO:teuthology.orchestra.run.smithi107.stdout: debug ms = 1 2023-09-30T01:32:16.911 INFO:teuthology.orchestra.run.smithi107.stdout: debug mon = 20 2023-09-30T01:32:16.911 INFO:teuthology.orchestra.run.smithi107.stdout: debug paxos = 20 2023-09-30T01:32:16.911 INFO:teuthology.orchestra.run.smithi107.stdout: debug auth = 20 2023-09-30T01:32:16.911 INFO:teuthology.orchestra.run.smithi107.stdout: mon data avail warn = 5 2023-09-30T01:32:16.911 INFO:teuthology.orchestra.run.smithi107.stdout: mon mgr mkfs grace = 240 2023-09-30T01:32:16.911 INFO:teuthology.orchestra.run.smithi107.stdout: mon reweight min pgs per osd = 4 2023-09-30T01:32:16.911 INFO:teuthology.orchestra.run.smithi107.stdout: mon osd reporter subtree level = osd 2023-09-30T01:32:16.912 INFO:teuthology.orchestra.run.smithi107.stdout: mon osd prime pg temp = true 2023-09-30T01:32:16.912 INFO:teuthology.orchestra.run.smithi107.stdout: mon reweight min bytes per osd = 10 2023-09-30T01:32:16.912 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-09-30T01:32:16.912 INFO:teuthology.orchestra.run.smithi107.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-09-30T01:32:16.912 INFO:teuthology.orchestra.run.smithi107.stdout: auth mon ticket ttl = 660 # 11m 2023-09-30T01:32:16.912 INFO:teuthology.orchestra.run.smithi107.stdout: auth service ticket ttl = 240 # 4m 2023-09-30T01:32:16.912 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-09-30T01:32:16.912 INFO:teuthology.orchestra.run.smithi107.stdout: # don't complain about insecure global_id in the test suite 2023-09-30T01:32:16.912 INFO:teuthology.orchestra.run.smithi107.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-09-30T01:32:16.912 INFO:teuthology.orchestra.run.smithi107.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-09-30T01:32:16.912 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-09-30T01:32:16.913 INFO:teuthology.orchestra.run.smithi107.stdout: # 1m isn't quite enough 2023-09-30T01:32:16.913 INFO:teuthology.orchestra.run.smithi107.stdout: mon_down_mkfs_grace = 2m 2023-09-30T01:32:16.913 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-09-30T01:32:16.913 INFO:teuthology.orchestra.run.smithi107.stdout: mon_warn_on_filestore_osds = false 2023-09-30T01:32:16.913 INFO:teuthology.orchestra.run.smithi107.stdout: mon debug extra checks = True 2023-09-30T01:32:16.913 INFO:teuthology.orchestra.run.smithi107.stdout: mon min osdmap epochs = 100 2023-09-30T01:32:16.913 INFO:teuthology.orchestra.run.smithi107.stdout: mon osdmap full prune enabled = True 2023-09-30T01:32:16.913 INFO:teuthology.orchestra.run.smithi107.stdout: mon osdmap full prune interval = 10 2023-09-30T01:32:16.913 INFO:teuthology.orchestra.run.smithi107.stdout: mon osdmap full prune min = 200 2023-09-30T01:32:16.913 INFO:teuthology.orchestra.run.smithi107.stdout: mon osdmap full prune txsize = 100 2023-09-30T01:32:16.913 INFO:teuthology.orchestra.run.smithi107.stdout: mon scrub inject crc mismatch = 0.01 2023-09-30T01:32:16.913 INFO:teuthology.orchestra.run.smithi107.stdout: mon scrub inject missing keys = 0.05 2023-09-30T01:32:16.914 INFO:teuthology.orchestra.run.smithi107.stdout: mon scrub interval = 300 2023-09-30T01:32:16.914 INFO:teuthology.orchestra.run.smithi107.stdout: paxos service trim min = 5 2023-09-30T01:32:16.914 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-09-30T01:32:16.914 INFO:teuthology.orchestra.run.smithi107.stdout:[client] 2023-09-30T01:32:16.914 INFO:teuthology.orchestra.run.smithi107.stdout: rgw cache enabled = true 2023-09-30T01:32:16.914 INFO:teuthology.orchestra.run.smithi107.stdout: rgw enable ops log = true 2023-09-30T01:32:16.914 INFO:teuthology.orchestra.run.smithi107.stdout: rgw enable usage log = true 2023-09-30T01:32:16.914 INFO:teuthology.orchestra.run.smithi107.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-09-30T01:32:16.914 INFO:teuthology.orchestra.run.smithi107.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-09-30T01:32:16.914 INFO:teuthology.orchestra.run.smithi107.stdout: debug monc = 20 2023-09-30T01:32:16.914 INFO:teuthology.orchestra.run.smithi107.stdout: debug ms = 1 2023-09-30T01:32:16.914 INFO:teuthology.orchestra.run.smithi107.stdout:[mon.a] 2023-09-30T01:32:16.915 INFO:teuthology.orchestra.run.smithi107.stdout:[mon.b] 2023-09-30T01:32:16.915 INFO:teuthology.orchestra.run.smithi107.stdout:[mon.c] 2023-09-30T01:32:16.915 INFO:teuthology.orchestra.run.smithi107.stdout:[mon.d] 2023-09-30T01:32:16.915 INFO:teuthology.orchestra.run.smithi107.stdout:[mon.e] 2023-09-30T01:32:16.915 INFO:teuthology.orchestra.run.smithi107.stdout:[mon.f] 2023-09-30T01:32:16.915 INFO:teuthology.orchestra.run.smithi107.stdout:[mon.g] 2023-09-30T01:32:16.915 INFO:teuthology.orchestra.run.smithi107.stdout:[mon.h] 2023-09-30T01:32:16.915 INFO:teuthology.orchestra.run.smithi107.stdout:[mon.i] 2023-09-30T01:32:16.915 INFO:tasks.ceph:Creating admin key on mon.a... 2023-09-30T01:32:16.916 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2023-09-30T01:32:16.965 INFO:tasks.ceph:Copying monmap to all nodes... 2023-09-30T01:32:16.965 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2023-09-30T01:32:16.965 DEBUG:teuthology.orchestra.run.smithi088:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2023-09-30T01:32:16.972 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2023-09-30T01:32:16.972 DEBUG:teuthology.orchestra.run.smithi088:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2023-09-30T01:32:16.976 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi088.front.sepia.ceph.com 2023-09-30T01:32:16.976 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2023-09-30T01:32:16.976 DEBUG:teuthology.orchestra.run.smithi088:> sudo dd of=/etc/ceph/ceph.keyring 2023-09-30T01:32:16.976 DEBUG:teuthology.orchestra.run.smithi088:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-09-30T01:32:16.997 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2023-09-30T01:32:16.998 DEBUG:teuthology.orchestra.run.smithi088:> dd of=/home/ubuntu/cephtest/ceph.monmap 2023-09-30T01:32:17.003 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi107.front.sepia.ceph.com 2023-09-30T01:32:17.003 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-09-30T01:32:17.003 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd of=/etc/ceph/ceph.keyring 2023-09-30T01:32:17.003 DEBUG:teuthology.orchestra.run.smithi107:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-09-30T01:32:17.030 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-09-30T01:32:17.030 DEBUG:teuthology.orchestra.run.smithi107:> dd of=/home/ubuntu/cephtest/ceph.monmap 2023-09-30T01:32:17.035 INFO:tasks.ceph:Setting up mon nodes... 2023-09-30T01:32:17.035 INFO:tasks.ceph:Setting up mgr nodes... 2023-09-30T01:32:17.035 DEBUG:teuthology.orchestra.run.smithi107:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2023-09-30T01:32:17.074 INFO:teuthology.orchestra.run.smithi107.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2023-09-30T01:32:17.076 INFO:tasks.ceph:Setting up mds nodes... 2023-09-30T01:32:17.076 INFO:tasks.ceph_client:Setting up client nodes... 2023-09-30T01:32:17.077 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2023-09-30T01:32:17.153 INFO:teuthology.orchestra.run.smithi107.stdout:creating /etc/ceph/ceph.client.0.keyring 2023-09-30T01:32:17.166 INFO:tasks.ceph:Running mkfs on osd nodes... 2023-09-30T01:32:17.167 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi088.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}, Remote(name='ubuntu@smithi107.front.sepia.ceph.com'): {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'}} 2023-09-30T01:32:17.167 DEBUG:teuthology.orchestra.run.smithi088:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2023-09-30T01:32:17.183 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2023-09-30T01:32:17.183 INFO:tasks.ceph:role: osd.0 2023-09-30T01:32:17.184 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi088.front.sepia.ceph.com 2023-09-30T01:32:17.184 DEBUG:teuthology.orchestra.run.smithi088:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2023-09-30T01:32:17.734 INFO:teuthology.orchestra.run.smithi088.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2023-09-30T01:32:17.734 INFO:teuthology.orchestra.run.smithi088.stdout: = sectsz=512 attr=2, projid32bit=1 2023-09-30T01:32:17.734 INFO:teuthology.orchestra.run.smithi088.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-09-30T01:32:17.734 INFO:teuthology.orchestra.run.smithi088.stdout: = reflink=1 2023-09-30T01:32:17.734 INFO:teuthology.orchestra.run.smithi088.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-09-30T01:32:17.734 INFO:teuthology.orchestra.run.smithi088.stdout: = sunit=0 swidth=0 blks 2023-09-30T01:32:17.734 INFO:teuthology.orchestra.run.smithi088.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-09-30T01:32:17.734 INFO:teuthology.orchestra.run.smithi088.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-09-30T01:32:17.734 INFO:teuthology.orchestra.run.smithi088.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-09-30T01:32:17.735 INFO:teuthology.orchestra.run.smithi088.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-09-30T01:32:17.736 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi088.front.sepia.ceph.com -o noatime 2023-09-30T01:32:17.736 DEBUG:teuthology.orchestra.run.smithi088:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2023-09-30T01:32:17.822 DEBUG:teuthology.orchestra.run.smithi088:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2023-09-30T01:32:17.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-09-30T01:32:17.833 INFO:teuthology.orchestra.run.smithi088.stderr:sudo: /sbin/restorecon: command not found 2023-09-30T01:32:17.834 DEBUG:teuthology.orchestra.run.smithi088:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2023-09-30T01:32:17.845 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2023-09-30T01:32:17.845 INFO:tasks.ceph:role: osd.1 2023-09-30T01:32:17.845 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi088.front.sepia.ceph.com 2023-09-30T01:32:17.845 DEBUG:teuthology.orchestra.run.smithi088:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2023-09-30T01:32:18.409 INFO:teuthology.orchestra.run.smithi088.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2023-09-30T01:32:18.410 INFO:teuthology.orchestra.run.smithi088.stdout: = sectsz=512 attr=2, projid32bit=1 2023-09-30T01:32:18.410 INFO:teuthology.orchestra.run.smithi088.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-09-30T01:32:18.410 INFO:teuthology.orchestra.run.smithi088.stdout: = reflink=1 2023-09-30T01:32:18.410 INFO:teuthology.orchestra.run.smithi088.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-09-30T01:32:18.410 INFO:teuthology.orchestra.run.smithi088.stdout: = sunit=0 swidth=0 blks 2023-09-30T01:32:18.410 INFO:teuthology.orchestra.run.smithi088.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-09-30T01:32:18.410 INFO:teuthology.orchestra.run.smithi088.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-09-30T01:32:18.410 INFO:teuthology.orchestra.run.smithi088.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-09-30T01:32:18.410 INFO:teuthology.orchestra.run.smithi088.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-09-30T01:32:18.411 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi088.front.sepia.ceph.com -o noatime 2023-09-30T01:32:18.411 DEBUG:teuthology.orchestra.run.smithi088:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2023-09-30T01:32:18.445 DEBUG:teuthology.orchestra.run.smithi088:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2023-09-30T01:32:18.453 INFO:teuthology.orchestra.run.smithi088.stderr:sudo: /sbin/restorecon: command not found 2023-09-30T01:32:18.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-09-30T01:32:18.455 DEBUG:teuthology.orchestra.run.smithi088:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2023-09-30T01:32:18.512 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2023-09-30T01:32:18.512 INFO:tasks.ceph:role: osd.2 2023-09-30T01:32:18.512 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi088.front.sepia.ceph.com 2023-09-30T01:32:18.512 DEBUG:teuthology.orchestra.run.smithi088:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2023-09-30T01:32:18.980 INFO:teuthology.orchestra.run.smithi088.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2023-09-30T01:32:18.981 INFO:teuthology.orchestra.run.smithi088.stdout: = sectsz=512 attr=2, projid32bit=1 2023-09-30T01:32:18.981 INFO:teuthology.orchestra.run.smithi088.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-09-30T01:32:18.981 INFO:teuthology.orchestra.run.smithi088.stdout: = reflink=1 2023-09-30T01:32:18.981 INFO:teuthology.orchestra.run.smithi088.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-09-30T01:32:18.981 INFO:teuthology.orchestra.run.smithi088.stdout: = sunit=0 swidth=0 blks 2023-09-30T01:32:18.981 INFO:teuthology.orchestra.run.smithi088.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-09-30T01:32:18.981 INFO:teuthology.orchestra.run.smithi088.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-09-30T01:32:18.981 INFO:teuthology.orchestra.run.smithi088.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-09-30T01:32:18.981 INFO:teuthology.orchestra.run.smithi088.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-09-30T01:32:18.982 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi088.front.sepia.ceph.com -o noatime 2023-09-30T01:32:18.983 DEBUG:teuthology.orchestra.run.smithi088:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2023-09-30T01:32:19.035 DEBUG:teuthology.orchestra.run.smithi088:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2023-09-30T01:32:19.048 INFO:teuthology.orchestra.run.smithi088.stderr:sudo: /sbin/restorecon: command not found 2023-09-30T01:32:19.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-09-30T01:32:19.049 DEBUG:teuthology.orchestra.run.smithi088:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-09-30T01:32:19.180 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:19.179+0000 7fd736c2d3c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2023-09-30T01:32:19.181 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:19.179+0000 7fd736c2d3c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2023-09-30T01:32:19.181 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:19.179+0000 7fd736c2d3c0 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-09-30T01:32:19.181 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:19.179+0000 7fd736c2d3c0 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-09-30T01:32:19.181 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:19.179+0000 7fd736c2d3c0 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-09-30T01:32:19.183 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:19.183+0000 7fd736c2d3c0 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2023-09-30T01:32:19.747 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:19.747+0000 7fd736c2d3c0 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-09-30T01:32:20.533 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:20.531+0000 7fd736c2d3c0 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-09-30T01:32:21.315 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:21.315+0000 7fd736c2d3c0 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-09-30T01:32:22.195 DEBUG:teuthology.orchestra.run.smithi088:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2023-09-30T01:32:22.211 DEBUG:teuthology.orchestra.run.smithi088:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-09-30T01:32:22.246 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:22.243+0000 7f2500d243c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2023-09-30T01:32:22.247 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:22.243+0000 7f2500d243c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2023-09-30T01:32:22.247 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:22.243+0000 7f2500d243c0 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-09-30T01:32:22.247 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:22.243+0000 7f2500d243c0 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-09-30T01:32:22.247 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:22.243+0000 7f2500d243c0 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-09-30T01:32:22.247 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:22.247+0000 7f2500d243c0 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2023-09-30T01:32:22.812 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:22.811+0000 7f2500d243c0 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-09-30T01:32:23.596 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:23.595+0000 7f2500d243c0 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-09-30T01:32:24.382 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:24.379+0000 7f2500d243c0 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-09-30T01:32:25.235 DEBUG:teuthology.orchestra.run.smithi088:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2023-09-30T01:32:25.249 DEBUG:teuthology.orchestra.run.smithi088:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-09-30T01:32:25.287 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:25.286+0000 7fb94a1183c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2023-09-30T01:32:25.287 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:25.286+0000 7fb94a1183c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2023-09-30T01:32:25.288 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:25.286+0000 7fb94a1183c0 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-09-30T01:32:25.288 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:25.286+0000 7fb94a1183c0 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-09-30T01:32:25.288 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:25.286+0000 7fb94a1183c0 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-09-30T01:32:25.288 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:25.286+0000 7fb94a1183c0 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2023-09-30T01:32:25.857 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:25.854+0000 7fb94a1183c0 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-09-30T01:32:26.641 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:26.638+0000 7fb94a1183c0 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-09-30T01:32:27.424 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:27.422+0000 7fb94a1183c0 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-09-30T01:32:28.298 DEBUG:teuthology.orchestra.run.smithi088:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2023-09-30T01:32:28.359 DEBUG:teuthology.orchestra.run.smithi107:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2023-09-30T01:32:28.376 INFO:tasks.ceph:roles_to_devs: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'} 2023-09-30T01:32:28.376 INFO:tasks.ceph:role: osd.3 2023-09-30T01:32:28.376 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi107.front.sepia.ceph.com 2023-09-30T01:32:28.376 DEBUG:teuthology.orchestra.run.smithi107:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2023-09-30T01:32:28.896 INFO:teuthology.orchestra.run.smithi107.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2023-09-30T01:32:28.896 INFO:teuthology.orchestra.run.smithi107.stdout: = sectsz=512 attr=2, projid32bit=1 2023-09-30T01:32:28.896 INFO:teuthology.orchestra.run.smithi107.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-09-30T01:32:28.896 INFO:teuthology.orchestra.run.smithi107.stdout: = reflink=1 2023-09-30T01:32:28.897 INFO:teuthology.orchestra.run.smithi107.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-09-30T01:32:28.897 INFO:teuthology.orchestra.run.smithi107.stdout: = sunit=0 swidth=0 blks 2023-09-30T01:32:28.897 INFO:teuthology.orchestra.run.smithi107.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-09-30T01:32:28.897 INFO:teuthology.orchestra.run.smithi107.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-09-30T01:32:28.897 INFO:teuthology.orchestra.run.smithi107.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-09-30T01:32:28.897 INFO:teuthology.orchestra.run.smithi107.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-09-30T01:32:28.898 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi107.front.sepia.ceph.com -o noatime 2023-09-30T01:32:28.898 DEBUG:teuthology.orchestra.run.smithi107:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-3 2023-09-30T01:32:28.967 DEBUG:teuthology.orchestra.run.smithi107:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2023-09-30T01:32:28.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-09-30T01:32:28.979 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: /sbin/restorecon: command not found 2023-09-30T01:32:28.980 DEBUG:teuthology.orchestra.run.smithi107:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2023-09-30T01:32:28.991 INFO:tasks.ceph:roles_to_devs: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'} 2023-09-30T01:32:28.991 INFO:tasks.ceph:role: osd.4 2023-09-30T01:32:28.991 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi107.front.sepia.ceph.com 2023-09-30T01:32:28.991 DEBUG:teuthology.orchestra.run.smithi107:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2023-09-30T01:32:29.515 INFO:teuthology.orchestra.run.smithi107.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2023-09-30T01:32:29.515 INFO:teuthology.orchestra.run.smithi107.stdout: = sectsz=512 attr=2, projid32bit=1 2023-09-30T01:32:29.516 INFO:teuthology.orchestra.run.smithi107.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-09-30T01:32:29.516 INFO:teuthology.orchestra.run.smithi107.stdout: = reflink=1 2023-09-30T01:32:29.516 INFO:teuthology.orchestra.run.smithi107.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-09-30T01:32:29.516 INFO:teuthology.orchestra.run.smithi107.stdout: = sunit=0 swidth=0 blks 2023-09-30T01:32:29.516 INFO:teuthology.orchestra.run.smithi107.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-09-30T01:32:29.516 INFO:teuthology.orchestra.run.smithi107.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-09-30T01:32:29.516 INFO:teuthology.orchestra.run.smithi107.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-09-30T01:32:29.516 INFO:teuthology.orchestra.run.smithi107.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-09-30T01:32:29.518 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi107.front.sepia.ceph.com -o noatime 2023-09-30T01:32:29.518 DEBUG:teuthology.orchestra.run.smithi107:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-4 2023-09-30T01:32:29.592 DEBUG:teuthology.orchestra.run.smithi107:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2023-09-30T01:32:29.604 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: /sbin/restorecon: command not found 2023-09-30T01:32:29.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-09-30T01:32:29.606 DEBUG:teuthology.orchestra.run.smithi107:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2023-09-30T01:32:29.615 INFO:tasks.ceph:roles_to_devs: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'} 2023-09-30T01:32:29.615 INFO:tasks.ceph:role: osd.5 2023-09-30T01:32:29.615 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi107.front.sepia.ceph.com 2023-09-30T01:32:29.616 DEBUG:teuthology.orchestra.run.smithi107:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2023-09-30T01:32:30.067 INFO:teuthology.orchestra.run.smithi107.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2023-09-30T01:32:30.067 INFO:teuthology.orchestra.run.smithi107.stdout: = sectsz=512 attr=2, projid32bit=1 2023-09-30T01:32:30.067 INFO:teuthology.orchestra.run.smithi107.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-09-30T01:32:30.067 INFO:teuthology.orchestra.run.smithi107.stdout: = reflink=1 2023-09-30T01:32:30.067 INFO:teuthology.orchestra.run.smithi107.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-09-30T01:32:30.067 INFO:teuthology.orchestra.run.smithi107.stdout: = sunit=0 swidth=0 blks 2023-09-30T01:32:30.067 INFO:teuthology.orchestra.run.smithi107.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-09-30T01:32:30.067 INFO:teuthology.orchestra.run.smithi107.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-09-30T01:32:30.067 INFO:teuthology.orchestra.run.smithi107.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-09-30T01:32:30.068 INFO:teuthology.orchestra.run.smithi107.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-09-30T01:32:30.068 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi107.front.sepia.ceph.com -o noatime 2023-09-30T01:32:30.069 DEBUG:teuthology.orchestra.run.smithi107:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-5 2023-09-30T01:32:30.125 DEBUG:teuthology.orchestra.run.smithi107:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2023-09-30T01:32:30.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-09-30T01:32:30.137 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: /sbin/restorecon: command not found 2023-09-30T01:32:30.138 DEBUG:teuthology.orchestra.run.smithi107:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-09-30T01:32:30.223 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:30.221+0000 7f307a5383c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2023-09-30T01:32:30.225 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:30.221+0000 7f307a5383c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2023-09-30T01:32:30.225 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:30.221+0000 7f307a5383c0 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2023-09-30T01:32:30.225 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:30.221+0000 7f307a5383c0 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2023-09-30T01:32:30.225 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:30.221+0000 7f307a5383c0 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2023-09-30T01:32:30.229 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:30.225+0000 7f307a5383c0 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2023-09-30T01:32:30.788 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:30.785+0000 7f307a5383c0 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-09-30T01:32:31.582 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:31.577+0000 7f307a5383c0 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-09-30T01:32:32.368 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:32.361+0000 7f307a5383c0 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-09-30T01:32:33.256 DEBUG:teuthology.orchestra.run.smithi107:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2023-09-30T01:32:33.273 DEBUG:teuthology.orchestra.run.smithi107:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-09-30T01:32:33.314 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:33.309+0000 7f20a7a2e3c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2023-09-30T01:32:33.315 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:33.309+0000 7f20a7a2e3c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2023-09-30T01:32:33.315 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:33.309+0000 7f20a7a2e3c0 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2023-09-30T01:32:33.315 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:33.309+0000 7f20a7a2e3c0 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2023-09-30T01:32:33.315 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:33.309+0000 7f20a7a2e3c0 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2023-09-30T01:32:33.316 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:33.313+0000 7f20a7a2e3c0 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2023-09-30T01:32:33.871 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:33.865+0000 7f20a7a2e3c0 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-09-30T01:32:34.668 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:34.665+0000 7f20a7a2e3c0 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-09-30T01:32:35.458 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:35.453+0000 7f20a7a2e3c0 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-09-30T01:32:36.315 DEBUG:teuthology.orchestra.run.smithi107:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2023-09-30T01:32:36.331 DEBUG:teuthology.orchestra.run.smithi107:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-09-30T01:32:36.374 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:36.369+0000 7f6c64ef13c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2023-09-30T01:32:36.375 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:36.369+0000 7f6c64ef13c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2023-09-30T01:32:36.375 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:36.369+0000 7f6c64ef13c0 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2023-09-30T01:32:36.375 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:36.369+0000 7f6c64ef13c0 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2023-09-30T01:32:36.375 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:36.369+0000 7f6c64ef13c0 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2023-09-30T01:32:36.375 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:36.373+0000 7f6c64ef13c0 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2023-09-30T01:32:36.932 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:36.929+0000 7f6c64ef13c0 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-09-30T01:32:37.714 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:37.709+0000 7f6c64ef13c0 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-09-30T01:32:38.503 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:38.497+0000 7f6c64ef13c0 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-09-30T01:32:39.351 DEBUG:teuthology.orchestra.run.smithi107:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2023-09-30T01:32:39.411 INFO:tasks.ceph:Reading keys from all nodes... 2023-09-30T01:32:39.412 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2023-09-30T01:32:39.412 DEBUG:teuthology.orchestra.run.smithi088:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2023-09-30T01:32:39.429 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2023-09-30T01:32:39.429 DEBUG:teuthology.orchestra.run.smithi088:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2023-09-30T01:32:39.487 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2023-09-30T01:32:39.487 DEBUG:teuthology.orchestra.run.smithi088:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2023-09-30T01:32:39.498 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-09-30T01:32:39.499 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2023-09-30T01:32:39.514 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-09-30T01:32:39.514 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2023-09-30T01:32:39.526 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-09-30T01:32:39.526 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2023-09-30T01:32:39.581 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-09-30T01:32:39.581 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2023-09-30T01:32:39.596 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-09-30T01:32:39.596 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2023-09-30T01:32:39.603 INFO:tasks.ceph:Adding keys to all mons... 2023-09-30T01:32:39.603 DEBUG:teuthology.orchestra.run.smithi088:> sudo tee -a /etc/ceph/ceph.keyring 2023-09-30T01:32:39.605 DEBUG:teuthology.orchestra.run.smithi107:> sudo tee -a /etc/ceph/ceph.keyring 2023-09-30T01:32:39.650 INFO:teuthology.orchestra.run.smithi088.stdout:[osd.0] 2023-09-30T01:32:39.650 INFO:teuthology.orchestra.run.smithi088.stdout: key = AQCjehdlaerQChAAFrSoLyFLY/3eIduQnJHIhA== 2023-09-30T01:32:39.650 INFO:teuthology.orchestra.run.smithi088.stdout:[osd.1] 2023-09-30T01:32:39.650 INFO:teuthology.orchestra.run.smithi088.stdout: key = AQCmehdldNm8DhAASPGMuHjGY169T89a9YJWMw== 2023-09-30T01:32:39.650 INFO:teuthology.orchestra.run.smithi088.stdout:[osd.2] 2023-09-30T01:32:39.650 INFO:teuthology.orchestra.run.smithi088.stdout: key = AQCpehdllOQpERAA5q/1b+M2vhvLozT1hgtQpQ== 2023-09-30T01:32:39.650 INFO:teuthology.orchestra.run.smithi088.stdout:[mgr.x] 2023-09-30T01:32:39.650 INFO:teuthology.orchestra.run.smithi088.stdout: key = AQChehdlThluBBAA6VDvqcksoDd1L2ovFI9v1w== 2023-09-30T01:32:39.650 INFO:teuthology.orchestra.run.smithi088.stdout:[osd.3] 2023-09-30T01:32:39.650 INFO:teuthology.orchestra.run.smithi088.stdout: key = AQCuehdl0ztZDRAAzYrnqfpJItMmJODMaKmINQ== 2023-09-30T01:32:39.651 INFO:teuthology.orchestra.run.smithi088.stdout:[osd.4] 2023-09-30T01:32:39.651 INFO:teuthology.orchestra.run.smithi088.stdout: key = AQCxehdl+HLJEhAAIJ7sN2+jct4HOLj1XIQBpQ== 2023-09-30T01:32:39.651 INFO:teuthology.orchestra.run.smithi088.stdout:[osd.5] 2023-09-30T01:32:39.651 INFO:teuthology.orchestra.run.smithi088.stdout: key = AQC0ehdlP/hQFhAALR4ujqZG4iGy79bUywHBNA== 2023-09-30T01:32:39.651 INFO:teuthology.orchestra.run.smithi088.stdout:[client.0] 2023-09-30T01:32:39.651 INFO:teuthology.orchestra.run.smithi088.stdout: key = AQChehdlv8cpCRAAfZKPAUShaXVZKbb18VlfXA== 2023-09-30T01:32:39.659 INFO:teuthology.orchestra.run.smithi107.stdout:[osd.0] 2023-09-30T01:32:39.659 INFO:teuthology.orchestra.run.smithi107.stdout: key = AQCjehdlaerQChAAFrSoLyFLY/3eIduQnJHIhA== 2023-09-30T01:32:39.659 INFO:teuthology.orchestra.run.smithi107.stdout:[osd.1] 2023-09-30T01:32:39.659 INFO:teuthology.orchestra.run.smithi107.stdout: key = AQCmehdldNm8DhAASPGMuHjGY169T89a9YJWMw== 2023-09-30T01:32:39.659 INFO:teuthology.orchestra.run.smithi107.stdout:[osd.2] 2023-09-30T01:32:39.660 INFO:teuthology.orchestra.run.smithi107.stdout: key = AQCpehdllOQpERAA5q/1b+M2vhvLozT1hgtQpQ== 2023-09-30T01:32:39.660 INFO:teuthology.orchestra.run.smithi107.stdout:[mgr.x] 2023-09-30T01:32:39.660 INFO:teuthology.orchestra.run.smithi107.stdout: key = AQChehdlThluBBAA6VDvqcksoDd1L2ovFI9v1w== 2023-09-30T01:32:39.660 INFO:teuthology.orchestra.run.smithi107.stdout:[osd.3] 2023-09-30T01:32:39.660 INFO:teuthology.orchestra.run.smithi107.stdout: key = AQCuehdl0ztZDRAAzYrnqfpJItMmJODMaKmINQ== 2023-09-30T01:32:39.660 INFO:teuthology.orchestra.run.smithi107.stdout:[osd.4] 2023-09-30T01:32:39.660 INFO:teuthology.orchestra.run.smithi107.stdout: key = AQCxehdl+HLJEhAAIJ7sN2+jct4HOLj1XIQBpQ== 2023-09-30T01:32:39.660 INFO:teuthology.orchestra.run.smithi107.stdout:[osd.5] 2023-09-30T01:32:39.660 INFO:teuthology.orchestra.run.smithi107.stdout: key = AQC0ehdlP/hQFhAALR4ujqZG4iGy79bUywHBNA== 2023-09-30T01:32:39.660 INFO:teuthology.orchestra.run.smithi107.stdout:[client.0] 2023-09-30T01:32:39.660 INFO:teuthology.orchestra.run.smithi107.stdout: key = AQChehdlv8cpCRAAfZKPAUShaXVZKbb18VlfXA== 2023-09-30T01:32:39.661 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-09-30T01:32:39.663 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-09-30T01:32:39.707 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-09-30T01:32:39.752 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-09-30T01:32:39.800 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-09-30T01:32:39.803 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-09-30T01:32:39.846 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-09-30T01:32:39.848 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-09-30T01:32:39.893 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-09-30T01:32:39.932 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-09-30T01:32:39.976 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-09-30T01:32:39.979 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-09-30T01:32:40.024 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-09-30T01:32:40.026 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-09-30T01:32:40.066 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-09-30T01:32:40.116 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-09-30T01:32:40.164 INFO:tasks.ceph:Running mkfs on mon nodes... 2023-09-30T01:32:40.164 DEBUG:teuthology.orchestra.run.smithi088:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2023-09-30T01:32:40.179 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-09-30T01:32:40.234 DEBUG:teuthology.orchestra.run.smithi088:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2023-09-30T01:32:40.249 DEBUG:teuthology.orchestra.run.smithi088:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2023-09-30T01:32:40.259 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-09-30T01:32:40.364 DEBUG:teuthology.orchestra.run.smithi088:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2023-09-30T01:32:40.380 DEBUG:teuthology.orchestra.run.smithi088:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2023-09-30T01:32:40.390 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-09-30T01:32:40.442 DEBUG:teuthology.orchestra.run.smithi088:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2023-09-30T01:32:40.455 DEBUG:teuthology.orchestra.run.smithi088:> sudo mkdir -p /var/lib/ceph/mon/ceph-d 2023-09-30T01:32:40.465 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i d --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-09-30T01:32:40.516 DEBUG:teuthology.orchestra.run.smithi088:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-d 2023-09-30T01:32:40.530 DEBUG:teuthology.orchestra.run.smithi088:> sudo mkdir -p /var/lib/ceph/mon/ceph-e 2023-09-30T01:32:40.585 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i e --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-09-30T01:32:40.639 DEBUG:teuthology.orchestra.run.smithi088:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-e 2023-09-30T01:32:40.652 DEBUG:teuthology.orchestra.run.smithi107:> sudo mkdir -p /var/lib/ceph/mon/ceph-f 2023-09-30T01:32:40.669 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i f --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-09-30T01:32:40.726 DEBUG:teuthology.orchestra.run.smithi107:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-f 2023-09-30T01:32:40.738 DEBUG:teuthology.orchestra.run.smithi107:> sudo mkdir -p /var/lib/ceph/mon/ceph-g 2023-09-30T01:32:40.752 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i g --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-09-30T01:32:40.854 DEBUG:teuthology.orchestra.run.smithi107:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-g 2023-09-30T01:32:40.868 DEBUG:teuthology.orchestra.run.smithi107:> sudo mkdir -p /var/lib/ceph/mon/ceph-h 2023-09-30T01:32:40.881 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i h --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-09-30T01:32:40.937 DEBUG:teuthology.orchestra.run.smithi107:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-h 2023-09-30T01:32:40.950 DEBUG:teuthology.orchestra.run.smithi107:> sudo mkdir -p /var/lib/ceph/mon/ceph-i 2023-09-30T01:32:41.007 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i i --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-09-30T01:32:41.060 DEBUG:teuthology.orchestra.run.smithi107:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-i 2023-09-30T01:32:41.072 DEBUG:teuthology.orchestra.run.smithi088:> rm -- /home/ubuntu/cephtest/ceph.monmap 2023-09-30T01:32:41.074 DEBUG:teuthology.orchestra.run.smithi107:> rm -- /home/ubuntu/cephtest/ceph.monmap 2023-09-30T01:32:41.121 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2023-09-30T01:32:41.121 INFO:tasks.ceph.mon.a:Restarting daemon 2023-09-30T01:32:41.122 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2023-09-30T01:32:41.124 INFO:tasks.ceph.mon.a:Started 2023-09-30T01:32:41.124 INFO:tasks.ceph.mon.b:Restarting daemon 2023-09-30T01:32:41.124 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2023-09-30T01:32:41.126 INFO:tasks.ceph.mon.b:Started 2023-09-30T01:32:41.127 INFO:tasks.ceph.mon.c:Restarting daemon 2023-09-30T01:32:41.127 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2023-09-30T01:32:41.129 INFO:tasks.ceph.mon.c:Started 2023-09-30T01:32:41.129 INFO:tasks.ceph.mon.d:Restarting daemon 2023-09-30T01:32:41.129 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i d 2023-09-30T01:32:41.131 INFO:tasks.ceph.mon.d:Started 2023-09-30T01:32:41.131 INFO:tasks.ceph.mon.e:Restarting daemon 2023-09-30T01:32:41.132 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i e 2023-09-30T01:32:41.134 INFO:tasks.ceph.mon.e:Started 2023-09-30T01:32:41.134 INFO:tasks.ceph.mon.f:Restarting daemon 2023-09-30T01:32:41.134 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i f 2023-09-30T01:32:41.139 INFO:tasks.ceph.mon.f:Started 2023-09-30T01:32:41.139 INFO:tasks.ceph.mon.g:Restarting daemon 2023-09-30T01:32:41.139 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i g 2023-09-30T01:32:41.141 INFO:tasks.ceph.mon.g:Started 2023-09-30T01:32:41.141 INFO:tasks.ceph.mon.h:Restarting daemon 2023-09-30T01:32:41.142 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i h 2023-09-30T01:32:41.144 INFO:tasks.ceph.mon.h:Started 2023-09-30T01:32:41.144 INFO:tasks.ceph.mon.i:Restarting daemon 2023-09-30T01:32:41.144 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i i 2023-09-30T01:32:41.147 INFO:tasks.ceph.mon.i:Started 2023-09-30T01:32:41.147 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2023-09-30T01:32:41.147 INFO:tasks.ceph.mgr.x:Restarting daemon 2023-09-30T01:32:41.147 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2023-09-30T01:32:41.149 INFO:tasks.ceph.mgr.x:Started 2023-09-30T01:32:41.150 INFO:tasks.ceph:Setting crush tunables to default 2023-09-30T01:32:41.150 DEBUG:teuthology.orchestra.run.smithi088:> sudo ceph --cluster ceph osd crush tunables default 2023-09-30T01:32:41.259 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:41.258+0000 7fb69130c700 10 monclient: get_monmap_and_config 2023-09-30T01:32:41.260 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:41.258+0000 7fb69130c700 10 monclient: build_initial_monmap 2023-09-30T01:32:41.260 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:41.258+0000 7fb69130c700 10 monclient: monmap: 2023-09-30T01:32:41.260 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2023-09-30T01:32:41.260 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:32:41.260 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:41.260585+0000 2023-09-30T01:32:41.261 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:41.260585+0000 2023-09-30T01:32:41.261 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2023-09-30T01:32:41.261 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:32:41.261 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a-legacy 2023-09-30T01:32:41.261 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.107:6789/0 mon.noname-f-legacy 2023-09-30T01:32:41.261 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.noname-b 2023-09-30T01:32:41.261 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.noname-g 2023-09-30T01:32:41.261 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.noname-c 2023-09-30T01:32:41.262 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.noname-h 2023-09-30T01:32:41.262 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.noname-d 2023-09-30T01:32:41.262 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.noname-i 2023-09-30T01:32:41.262 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.noname-e 2023-09-30T01:32:41.262 INFO:teuthology.orchestra.run.smithi088.stderr:9: v2:172.21.15.88:3300/0 mon.noname-a 2023-09-30T01:32:41.262 INFO:teuthology.orchestra.run.smithi088.stderr:10: v2:172.21.15.107:3300/0 mon.noname-f 2023-09-30T01:32:41.262 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:32:41.266 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:41.258+0000 7fb69130c700 1 Processor -- start 2023-09-30T01:32:41.266 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:41.258+0000 7fb69130c700 1 -- start start 2023-09-30T01:32:41.267 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:41.258+0000 7fb69130c700 10 monclient: init 2023-09-30T01:32:41.267 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:41.258+0000 7fb69130c700 10 monclient: _reopen_session rank -1 2023-09-30T01:32:41.267 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:41.258+0000 7fb69130c700 10 monclient: _add_conns ranks=[4,5,9,3,10,7,0,6,2,1,8] 2023-09-30T01:32:41.267 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:41.258+0000 7fb69130c700 1 --2- >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7fb68c097a40 0x7fb68c132700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:32:41.267 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:41.258+0000 7fb69130c700 10 monclient(hunting): picked mon.noname-c con 0x7fb68c097a40 addr [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] 2023-09-30T01:32:41.267 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:41.258+0000 7fb69130c700 1 --2- >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7fb68c1311c0 0x7fb68c132c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:32:41.267 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:41.258+0000 7fb69130c700 10 monclient(hunting): picked mon.noname-h con 0x7fb68c1311c0 addr [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] 2023-09-30T01:32:41.267 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:41.258+0000 7fb69130c700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fb68c1332e0 0x7fb68c133710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:32:41.267 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:41.258+0000 7fb69130c700 10 monclient(hunting): picked mon.noname-a con 0x7fb68c1332e0 addr v2:172.21.15.88:3300/0 2023-09-30T01:32:41.267 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:41.258+0000 7fb69130c700 10 monclient(hunting): start opening mon connection 2023-09-30T01:32:41.268 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:41.258+0000 7fb69130c700 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb68c131590 con 0x7fb68c1332e0 2023-09-30T01:32:41.268 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:41.258+0000 7fb69130c700 10 monclient(hunting): start opening mon connection 2023-09-30T01:32:41.268 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:41.258+0000 7fb69130c700 1 -- --> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] -- mon_getmap magic: 0 v1 -- 0x7fb68c09a5d0 con 0x7fb68c097a40 2023-09-30T01:32:41.268 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:41.258+0000 7fb69130c700 10 monclient(hunting): start opening mon connection 2023-09-30T01:32:41.268 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:41.258+0000 7fb69130c700 1 -- --> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] -- mon_getmap magic: 0 v1 -- 0x7fb68c097e10 con 0x7fb68c1311c0 2023-09-30T01:32:41.268 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:41.258+0000 7fb69130c700 10 monclient(hunting): _renew_subs 2023-09-30T01:32:41.268 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:41.258+0000 7fb69130c700 10 monclient(hunting): authenticate will time out at 833.298377s 2023-09-30T01:32:41.268 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:41.258+0000 7fb68bfff700 1 --2- >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7fb68c097a40 0x7fb68c132700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:32:41.268 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:41.258+0000 7fb68bfff700 1 --2- >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7fb68c097a40 0x7fb68c132700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3302/0 says I am v2:172.21.15.88:42446/0 (socket says 172.21.15.88:42446) 2023-09-30T01:32:41.268 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:41.258+0000 7fb690b0b700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fb68c1332e0 0x7fb68c133710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:32:41.269 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:41.258+0000 7fb68bfff700 1 -- 172.21.15.88:0/3779817799 learned_addr learned my addr 172.21.15.88:0/3779817799 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-09-30T01:32:41.269 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:41.258+0000 7fb68bfff700 10 monclient(hunting): get_auth_request con 0x7fb68c097a40 auth_method 0 2023-09-30T01:32:41.269 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:41.258+0000 7fb68bfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:32:41.269 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:41.258+0000 7fb68bfff700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:32:41.269 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:41.258+0000 7fb68bfff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:32:41.269 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:41.258+0000 7fb690b0b700 10 monclient(hunting): get_auth_request con 0x7fb68c1332e0 auth_method 0 2023-09-30T01:32:41.269 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:41.258+0000 7fb690b0b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:32:41.269 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:41.258+0000 7fb690b0b700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:32:41.269 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:41.258+0000 7fb690b0b700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:32:41.269 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:41.258+0000 7fb68bfff700 1 -- 172.21.15.88:0/3779817799 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7fb68c097a40 msgr2=0x7fb68c132700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-09-30T01:32:41.270 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:41.258+0000 7fb68bfff700 1 -- 172.21.15.88:0/3779817799 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7fb68c097a40 msgr2=0x7fb68c132700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-09-30T01:32:41.270 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:41.258+0000 7fb68bfff700 1 --2- 172.21.15.88:0/3779817799 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7fb68c097a40 0x7fb68c132700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2023-09-30T01:32:41.270 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:41.258+0000 7fb68b7fe700 1 --2- 172.21.15.88:0/3779817799 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7fb68c1311c0 0x7fb68c132c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:32:41.270 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:41.258+0000 7fb68bfff700 1 --2- 172.21.15.88:0/3779817799 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7fb68c097a40 0x7fb68c132700 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-09-30T01:32:41.270 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:41.258+0000 7fb690b0b700 1 -- 172.21.15.88:0/3779817799 >> v2:172.21.15.88:3300/0 conn(0x7fb68c1332e0 msgr2=0x7fb68c133710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2023-09-30T01:32:41.270 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:41.258+0000 7fb68b7fe700 10 monclient(hunting): get_auth_request con 0x7fb68c1311c0 auth_method 0 2023-09-30T01:32:41.270 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:41.258+0000 7fb68b7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:32:41.271 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:41.258+0000 7fb68b7fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:32:41.271 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:41.258+0000 7fb690b0b700 1 -- 172.21.15.88:0/3779817799 >> v2:172.21.15.88:3300/0 conn(0x7fb68c1332e0 msgr2=0x7fb68c133710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-09-30T01:32:41.271 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:41.258+0000 7fb68b7fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:32:41.271 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:41.258+0000 7fb690b0b700 1 --2- 172.21.15.88:0/3779817799 >> v2:172.21.15.88:3300/0 conn(0x7fb68c1332e0 0x7fb68c133710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2023-09-30T01:32:41.271 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:41.258+0000 7fb690b0b700 1 --2- 172.21.15.88:0/3779817799 >> v2:172.21.15.88:3300/0 conn(0x7fb68c1332e0 0x7fb68c133710 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-09-30T01:32:41.271 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:41.258+0000 7fb68b7fe700 1 -- 172.21.15.88:0/3779817799 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7fb68c1311c0 msgr2=0x7fb68c132c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-09-30T01:32:41.272 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:41.258+0000 7fb68b7fe700 1 -- 172.21.15.88:0/3779817799 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7fb68c1311c0 msgr2=0x7fb68c132c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-09-30T01:32:41.272 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:41.258+0000 7fb68b7fe700 1 --2- 172.21.15.88:0/3779817799 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7fb68c1311c0 0x7fb68c132c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2023-09-30T01:32:41.272 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:41.258+0000 7fb68b7fe700 1 --2- 172.21.15.88:0/3779817799 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7fb68c1311c0 0x7fb68c132c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-09-30T01:32:41.464 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:41.462+0000 7fb690b0b700 1 --2- 172.21.15.88:0/3779817799 >> v2:172.21.15.88:3300/0 conn(0x7fb68c1332e0 0x7fb68c133710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:32:41.464 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:41.462+0000 7fb68bfff700 1 --2- 172.21.15.88:0/3779817799 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7fb68c097a40 0x7fb68c132700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:32:41.464 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:41.462+0000 7fb690b0b700 10 monclient(hunting): get_auth_request con 0x7fb68c1332e0 auth_method 0 2023-09-30T01:32:41.464 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:41.462+0000 7fb690b0b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:32:41.465 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:41.462+0000 7fb690b0b700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:32:41.465 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:41.462+0000 7fb690b0b700 10 monclient(hunting): _init_auth already have auth, reseting 2023-09-30T01:32:41.465 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:41.462+0000 7fb68bfff700 10 monclient(hunting): get_auth_request con 0x7fb68c097a40 auth_method 0 2023-09-30T01:32:41.465 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:41.462+0000 7fb68bfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:32:41.465 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:41.462+0000 7fb68bfff700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:32:41.465 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:41.462+0000 7fb68bfff700 10 monclient(hunting): _init_auth already have auth, reseting 2023-09-30T01:32:41.465 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:41.462+0000 7fb68b7fe700 1 --2- 172.21.15.88:0/3779817799 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7fb68c1311c0 0x7fb68c132c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:32:41.465 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:41.462+0000 7fb68b7fe700 10 monclient(hunting): get_auth_request con 0x7fb68c1311c0 auth_method 0 2023-09-30T01:32:41.465 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:41.462+0000 7fb68b7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:32:41.465 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:41.462+0000 7fb68b7fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:32:41.465 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:41.462+0000 7fb68b7fe700 10 monclient(hunting): _init_auth already have auth, reseting 2023-09-30T01:32:41.465 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:41.462+0000 7fb690b0b700 1 -- 172.21.15.88:0/3779817799 >> v2:172.21.15.88:3300/0 conn(0x7fb68c1332e0 msgr2=0x7fb68c133710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-09-30T01:32:41.466 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:41.462+0000 7fb690b0b700 1 -- 172.21.15.88:0/3779817799 >> v2:172.21.15.88:3300/0 conn(0x7fb68c1332e0 msgr2=0x7fb68c133710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-09-30T01:32:41.466 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:41.462+0000 7fb68bfff700 1 -- 172.21.15.88:0/3779817799 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7fb68c097a40 msgr2=0x7fb68c132700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2023-09-30T01:32:41.466 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:41.462+0000 7fb68bfff700 1 -- 172.21.15.88:0/3779817799 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7fb68c097a40 msgr2=0x7fb68c132700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-09-30T01:32:41.466 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:41.462+0000 7fb690b0b700 1 --2- 172.21.15.88:0/3779817799 >> v2:172.21.15.88:3300/0 conn(0x7fb68c1332e0 0x7fb68c133710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2023-09-30T01:32:41.466 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:41.462+0000 7fb68bfff700 1 --2- 172.21.15.88:0/3779817799 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7fb68c097a40 0x7fb68c132700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2023-09-30T01:32:41.466 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:41.462+0000 7fb690b0b700 1 --2- 172.21.15.88:0/3779817799 >> v2:172.21.15.88:3300/0 conn(0x7fb68c1332e0 0x7fb68c133710 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-09-30T01:32:41.466 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:41.462+0000 7fb68bfff700 1 --2- 172.21.15.88:0/3779817799 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7fb68c097a40 0x7fb68c132700 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-09-30T01:32:41.466 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:41.462+0000 7fb68b7fe700 1 -- 172.21.15.88:0/3779817799 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7fb68c1311c0 msgr2=0x7fb68c132c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-09-30T01:32:41.466 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:41.462+0000 7fb68b7fe700 1 -- 172.21.15.88:0/3779817799 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7fb68c1311c0 msgr2=0x7fb68c132c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-09-30T01:32:41.466 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:41.462+0000 7fb68b7fe700 1 --2- 172.21.15.88:0/3779817799 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7fb68c1311c0 0x7fb68c132c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2023-09-30T01:32:41.466 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:41.462+0000 7fb68b7fe700 1 --2- 172.21.15.88:0/3779817799 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7fb68c1311c0 0x7fb68c132c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-09-30T01:32:41.868 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:41.866+0000 7fb690b0b700 1 --2- 172.21.15.88:0/3779817799 >> v2:172.21.15.88:3300/0 conn(0x7fb68c1332e0 0x7fb68c133710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:32:41.868 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:41.866+0000 7fb68bfff700 1 --2- 172.21.15.88:0/3779817799 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7fb68c097a40 0x7fb68c132700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:32:41.868 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:41.866+0000 7fb690b0b700 10 monclient(hunting): get_auth_request con 0x7fb68c1332e0 auth_method 0 2023-09-30T01:32:41.868 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:41.866+0000 7fb690b0b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:32:41.868 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:41.866+0000 7fb690b0b700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:32:41.868 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:41.866+0000 7fb690b0b700 10 monclient(hunting): _init_auth already have auth, reseting 2023-09-30T01:32:41.869 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:41.866+0000 7fb68bfff700 10 monclient(hunting): get_auth_request con 0x7fb68c097a40 auth_method 0 2023-09-30T01:32:41.869 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:41.866+0000 7fb68bfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:32:41.869 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:41.866+0000 7fb68bfff700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:32:41.869 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:41.866+0000 7fb68bfff700 10 monclient(hunting): _init_auth already have auth, reseting 2023-09-30T01:32:41.869 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:41.866+0000 7fb68b7fe700 1 --2- 172.21.15.88:0/3779817799 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7fb68c1311c0 0x7fb68c132c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:32:41.869 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:41.866+0000 7fb690b0b700 1 -- 172.21.15.88:0/3779817799 >> v2:172.21.15.88:3300/0 conn(0x7fb68c1332e0 msgr2=0x7fb68c133710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-09-30T01:32:41.869 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:41.866+0000 7fb68b7fe700 10 monclient(hunting): get_auth_request con 0x7fb68c1311c0 auth_method 0 2023-09-30T01:32:41.869 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:41.866+0000 7fb690b0b700 1 -- 172.21.15.88:0/3779817799 >> v2:172.21.15.88:3300/0 conn(0x7fb68c1332e0 msgr2=0x7fb68c133710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-09-30T01:32:41.869 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:41.866+0000 7fb68b7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:32:41.869 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:41.866+0000 7fb68b7fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:32:41.870 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:41.866+0000 7fb68b7fe700 10 monclient(hunting): _init_auth already have auth, reseting 2023-09-30T01:32:41.870 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:41.866+0000 7fb690b0b700 1 --2- 172.21.15.88:0/3779817799 >> v2:172.21.15.88:3300/0 conn(0x7fb68c1332e0 0x7fb68c133710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2023-09-30T01:32:41.870 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:41.866+0000 7fb690b0b700 1 --2- 172.21.15.88:0/3779817799 >> v2:172.21.15.88:3300/0 conn(0x7fb68c1332e0 0x7fb68c133710 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-09-30T01:32:41.870 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:41.866+0000 7fb68bfff700 1 -- 172.21.15.88:0/3779817799 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7fb68c097a40 msgr2=0x7fb68c132700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2023-09-30T01:32:41.870 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:41.866+0000 7fb68bfff700 1 -- 172.21.15.88:0/3779817799 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7fb68c097a40 msgr2=0x7fb68c132700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-09-30T01:32:41.870 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:41.866+0000 7fb68bfff700 1 --2- 172.21.15.88:0/3779817799 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7fb68c097a40 0x7fb68c132700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2023-09-30T01:32:41.870 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:41.866+0000 7fb68b7fe700 1 -- 172.21.15.88:0/3779817799 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7fb68c1311c0 msgr2=0x7fb68c132c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-09-30T01:32:41.870 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:41.866+0000 7fb68b7fe700 1 -- 172.21.15.88:0/3779817799 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7fb68c1311c0 msgr2=0x7fb68c132c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-09-30T01:32:41.871 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:41.866+0000 7fb68b7fe700 1 --2- 172.21.15.88:0/3779817799 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7fb68c1311c0 0x7fb68c132c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2023-09-30T01:32:41.871 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:41.866+0000 7fb68bfff700 1 --2- 172.21.15.88:0/3779817799 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7fb68c097a40 0x7fb68c132700 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-09-30T01:32:41.871 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:41.866+0000 7fb68b7fe700 1 --2- 172.21.15.88:0/3779817799 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7fb68c1311c0 0x7fb68c132c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-09-30T01:32:42.672 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:42.670+0000 7fb690b0b700 1 --2- 172.21.15.88:0/3779817799 >> v2:172.21.15.88:3300/0 conn(0x7fb68c1332e0 0x7fb68c133710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:32:42.672 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:42.670+0000 7fb68bfff700 1 --2- 172.21.15.88:0/3779817799 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7fb68c097a40 0x7fb68c132700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:32:42.672 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:42.670+0000 7fb690b0b700 10 monclient(hunting): get_auth_request con 0x7fb68c1332e0 auth_method 0 2023-09-30T01:32:42.672 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:42.670+0000 7fb690b0b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:32:42.672 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:42.670+0000 7fb690b0b700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:32:42.673 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:42.670+0000 7fb690b0b700 10 monclient(hunting): _init_auth already have auth, reseting 2023-09-30T01:32:42.673 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:42.670+0000 7fb68bfff700 10 monclient(hunting): get_auth_request con 0x7fb68c097a40 auth_method 0 2023-09-30T01:32:42.673 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:42.670+0000 7fb68bfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:32:42.673 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:42.670+0000 7fb68bfff700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:32:42.673 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:42.670+0000 7fb68bfff700 10 monclient(hunting): _init_auth already have auth, reseting 2023-09-30T01:32:42.673 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:42.670+0000 7fb690b0b700 1 -- 172.21.15.88:0/3779817799 >> v2:172.21.15.88:3300/0 conn(0x7fb68c1332e0 msgr2=0x7fb68c133710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-09-30T01:32:42.673 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:42.670+0000 7fb690b0b700 1 -- 172.21.15.88:0/3779817799 >> v2:172.21.15.88:3300/0 conn(0x7fb68c1332e0 msgr2=0x7fb68c133710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-09-30T01:32:42.673 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:42.670+0000 7fb68b7fe700 1 --2- 172.21.15.88:0/3779817799 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7fb68c1311c0 0x7fb68c132c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:32:42.673 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:42.670+0000 7fb690b0b700 1 --2- 172.21.15.88:0/3779817799 >> v2:172.21.15.88:3300/0 conn(0x7fb68c1332e0 0x7fb68c133710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2023-09-30T01:32:42.673 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:42.670+0000 7fb68bfff700 1 -- 172.21.15.88:0/3779817799 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7fb68c097a40 msgr2=0x7fb68c132700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2023-09-30T01:32:42.673 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:42.670+0000 7fb68b7fe700 10 monclient(hunting): get_auth_request con 0x7fb68c1311c0 auth_method 0 2023-09-30T01:32:42.674 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:42.670+0000 7fb68b7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:32:42.674 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:42.670+0000 7fb68bfff700 1 -- 172.21.15.88:0/3779817799 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7fb68c097a40 msgr2=0x7fb68c132700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-09-30T01:32:42.674 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:42.670+0000 7fb68b7fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:32:42.674 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:42.670+0000 7fb68b7fe700 10 monclient(hunting): _init_auth already have auth, reseting 2023-09-30T01:32:42.674 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:42.670+0000 7fb68bfff700 1 --2- 172.21.15.88:0/3779817799 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7fb68c097a40 0x7fb68c132700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2023-09-30T01:32:42.674 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:42.670+0000 7fb690b0b700 1 --2- 172.21.15.88:0/3779817799 >> v2:172.21.15.88:3300/0 conn(0x7fb68c1332e0 0x7fb68c133710 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-09-30T01:32:42.674 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:42.670+0000 7fb68bfff700 1 --2- 172.21.15.88:0/3779817799 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7fb68c097a40 0x7fb68c132700 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-09-30T01:32:42.674 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:42.670+0000 7fb68b7fe700 1 -- 172.21.15.88:0/3779817799 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7fb68c1311c0 msgr2=0x7fb68c132c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-09-30T01:32:42.674 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:42.670+0000 7fb68b7fe700 1 -- 172.21.15.88:0/3779817799 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7fb68c1311c0 msgr2=0x7fb68c132c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-09-30T01:32:42.674 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:42.670+0000 7fb68b7fe700 1 --2- 172.21.15.88:0/3779817799 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7fb68c1311c0 0x7fb68c132c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2023-09-30T01:32:42.674 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:42.670+0000 7fb68b7fe700 1 --2- 172.21.15.88:0/3779817799 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7fb68c1311c0 0x7fb68c132c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-09-30T01:32:44.263 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:44.258+0000 7fb689ffb700 10 monclient(hunting): tick 2023-09-30T01:32:44.263 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:44.258+0000 7fb689ffb700 1 monclient(hunting): continuing hunt 2023-09-30T01:32:44.263 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:44.258+0000 7fb689ffb700 10 monclient(hunting): _reopen_session rank -1 2023-09-30T01:32:44.263 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:44.258+0000 7fb689ffb700 1 -- 172.21.15.88:0/3779817799 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7fb68c1311c0 msgr2=0x7fb68c132c40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-09-30T01:32:44.263 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:44.258+0000 7fb689ffb700 1 --2- 172.21.15.88:0/3779817799 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7fb68c1311c0 0x7fb68c132c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:32:44.263 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:44.258+0000 7fb689ffb700 1 -- 172.21.15.88:0/3779817799 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7fb68c097a40 msgr2=0x7fb68c132700 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-09-30T01:32:44.263 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:44.258+0000 7fb689ffb700 1 --2- 172.21.15.88:0/3779817799 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7fb68c097a40 0x7fb68c132700 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:32:44.264 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:44.258+0000 7fb689ffb700 1 -- 172.21.15.88:0/3779817799 >> v2:172.21.15.88:3300/0 conn(0x7fb68c1332e0 msgr2=0x7fb68c133710 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-09-30T01:32:44.264 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:44.258+0000 7fb689ffb700 1 --2- 172.21.15.88:0/3779817799 >> v2:172.21.15.88:3300/0 conn(0x7fb68c1332e0 0x7fb68c133710 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:32:44.264 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:44.258+0000 7fb689ffb700 10 monclient: _add_conns ranks=[10,7,0,1,6,2,3,8] 2023-09-30T01:32:44.264 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:44.258+0000 7fb689ffb700 1 --2- 172.21.15.88:0/3779817799 >> v2:172.21.15.107:3300/0 conn(0x7fb674003840 0x7fb674005c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:32:44.264 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:44.258+0000 7fb689ffb700 10 monclient(hunting): picked mon.noname-f con 0x7fb674003840 addr v2:172.21.15.107:3300/0 2023-09-30T01:32:44.264 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:44.258+0000 7fb689ffb700 1 --2- 172.21.15.88:0/3779817799 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7fb674006240 0x7fb6740086c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:32:44.264 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:44.258+0000 7fb689ffb700 10 monclient(hunting): picked mon.noname-i con 0x7fb674006240 addr [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] 2023-09-30T01:32:44.265 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:44.258+0000 7fb689ffb700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fb674008d80 addr v1:172.21.15.88:6789/0 2023-09-30T01:32:44.265 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:44.258+0000 7fb689ffb700 1 -- 172.21.15.88:0/3779817799 --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb68c04bb20 con 0x7fb674008d80 2023-09-30T01:32:44.265 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:44.258+0000 7fb689ffb700 10 monclient(hunting): start opening mon connection 2023-09-30T01:32:44.265 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:44.258+0000 7fb689ffb700 1 -- 172.21.15.88:0/3779817799 --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb68c131590 con 0x7fb674003840 2023-09-30T01:32:44.265 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:44.258+0000 7fb689ffb700 10 monclient(hunting): start opening mon connection 2023-09-30T01:32:44.265 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:44.258+0000 7fb689ffb700 1 -- 172.21.15.88:0/3779817799 --> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] -- mon_getmap magic: 0 v1 -- 0x7fb68c09a5d0 con 0x7fb674006240 2023-09-30T01:32:44.265 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:44.258+0000 7fb689ffb700 10 monclient(hunting): _renew_subs 2023-09-30T01:32:44.265 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:44.258+0000 7fb68bfff700 1 --2- 172.21.15.88:0/3779817799 >> v2:172.21.15.107:3300/0 conn(0x7fb674003840 0x7fb674005c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:32:44.266 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:44.258+0000 7fb68b7fe700 1 --2- 172.21.15.88:0/3779817799 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7fb674006240 0x7fb6740086c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:32:44.266 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:44.258+0000 7fb68b7fe700 10 monclient(hunting): get_auth_request con 0x7fb674006240 auth_method 0 2023-09-30T01:32:44.266 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:44.258+0000 7fb68b7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:32:44.266 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:44.258+0000 7fb68b7fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:32:44.266 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:44.258+0000 7fb68b7fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:32:44.266 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:44.258+0000 7fb68bfff700 10 monclient(hunting): get_auth_request con 0x7fb674003840 auth_method 0 2023-09-30T01:32:44.266 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:44.258+0000 7fb68bfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:32:44.267 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:44.258+0000 7fb68bfff700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:32:44.267 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:44.258+0000 7fb68bfff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:32:44.267 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:44.258+0000 7fb68b7fe700 1 -- 172.21.15.88:0/3779817799 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7fb674006240 msgr2=0x7fb6740086c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-09-30T01:32:44.267 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:44.258+0000 7fb68b7fe700 1 -- 172.21.15.88:0/3779817799 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7fb674006240 msgr2=0x7fb6740086c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-09-30T01:32:44.267 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:44.258+0000 7fb68bfff700 1 -- 172.21.15.88:0/3779817799 >> v2:172.21.15.107:3300/0 conn(0x7fb674003840 msgr2=0x7fb674005c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-09-30T01:32:44.267 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:44.258+0000 7fb68bfff700 1 -- 172.21.15.88:0/3779817799 >> v2:172.21.15.107:3300/0 conn(0x7fb674003840 msgr2=0x7fb674005c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-09-30T01:32:44.267 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:44.258+0000 7fb68b7fe700 1 --2- 172.21.15.88:0/3779817799 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7fb674006240 0x7fb6740086c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2023-09-30T01:32:44.267 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:44.258+0000 7fb68bfff700 1 --2- 172.21.15.88:0/3779817799 >> v2:172.21.15.107:3300/0 conn(0x7fb674003840 0x7fb674005c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2023-09-30T01:32:44.268 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:44.258+0000 7fb68b7fe700 1 --2- 172.21.15.88:0/3779817799 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7fb674006240 0x7fb6740086c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-09-30T01:32:44.268 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:44.258+0000 7fb68bfff700 1 --2- 172.21.15.88:0/3779817799 >> v2:172.21.15.107:3300/0 conn(0x7fb674003840 0x7fb674005c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-09-30T01:32:44.464 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:44.462+0000 7fb68b7fe700 1 --2- 172.21.15.88:0/3779817799 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7fb674006240 0x7fb6740086c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:32:44.464 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:44.462+0000 7fb68bfff700 1 --2- 172.21.15.88:0/3779817799 >> v2:172.21.15.107:3300/0 conn(0x7fb674003840 0x7fb674005c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:32:44.464 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:44.462+0000 7fb68b7fe700 10 monclient(hunting): get_auth_request con 0x7fb674006240 auth_method 0 2023-09-30T01:32:44.464 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:44.462+0000 7fb68b7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:32:44.464 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:44.462+0000 7fb68b7fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:32:44.465 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:44.462+0000 7fb68b7fe700 10 monclient(hunting): _init_auth already have auth, reseting 2023-09-30T01:32:44.465 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:44.462+0000 7fb68bfff700 10 monclient(hunting): get_auth_request con 0x7fb674003840 auth_method 0 2023-09-30T01:32:44.465 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:44.462+0000 7fb68bfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:32:44.465 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:44.462+0000 7fb68bfff700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:32:44.465 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:44.462+0000 7fb68bfff700 10 monclient(hunting): _init_auth already have auth, reseting 2023-09-30T01:32:44.465 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:44.462+0000 7fb68bfff700 1 -- 172.21.15.88:0/3779817799 >> v2:172.21.15.107:3300/0 conn(0x7fb674003840 msgr2=0x7fb674005c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-09-30T01:32:44.465 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:44.462+0000 7fb68b7fe700 1 -- 172.21.15.88:0/3779817799 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7fb674006240 msgr2=0x7fb6740086c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-09-30T01:32:44.465 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:44.462+0000 7fb68bfff700 1 -- 172.21.15.88:0/3779817799 >> v2:172.21.15.107:3300/0 conn(0x7fb674003840 msgr2=0x7fb674005c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-09-30T01:32:44.465 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:44.462+0000 7fb68b7fe700 1 -- 172.21.15.88:0/3779817799 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7fb674006240 msgr2=0x7fb6740086c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-09-30T01:32:44.465 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:44.462+0000 7fb68b7fe700 1 --2- 172.21.15.88:0/3779817799 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7fb674006240 0x7fb6740086c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2023-09-30T01:32:44.465 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:44.462+0000 7fb68bfff700 1 --2- 172.21.15.88:0/3779817799 >> v2:172.21.15.107:3300/0 conn(0x7fb674003840 0x7fb674005c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2023-09-30T01:32:44.465 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:44.462+0000 7fb68b7fe700 1 --2- 172.21.15.88:0/3779817799 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7fb674006240 0x7fb6740086c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-09-30T01:32:44.466 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:44.462+0000 7fb68bfff700 1 --2- 172.21.15.88:0/3779817799 >> v2:172.21.15.107:3300/0 conn(0x7fb674003840 0x7fb674005c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-09-30T01:32:44.869 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:44.866+0000 7fb68b7fe700 1 --2- 172.21.15.88:0/3779817799 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7fb674006240 0x7fb6740086c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:32:44.869 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:44.866+0000 7fb68bfff700 1 --2- 172.21.15.88:0/3779817799 >> v2:172.21.15.107:3300/0 conn(0x7fb674003840 0x7fb674005c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:32:44.869 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:44.866+0000 7fb68b7fe700 10 monclient(hunting): get_auth_request con 0x7fb674006240 auth_method 0 2023-09-30T01:32:44.869 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:44.866+0000 7fb68b7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:32:44.869 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:44.866+0000 7fb68b7fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:32:44.870 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:44.866+0000 7fb68b7fe700 10 monclient(hunting): _init_auth already have auth, reseting 2023-09-30T01:32:44.870 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:44.866+0000 7fb68bfff700 10 monclient(hunting): get_auth_request con 0x7fb674003840 auth_method 0 2023-09-30T01:32:44.870 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:44.866+0000 7fb68bfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:32:44.870 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:44.866+0000 7fb68bfff700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:32:44.870 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:44.866+0000 7fb68bfff700 10 monclient(hunting): _init_auth already have auth, reseting 2023-09-30T01:32:44.871 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:44.866+0000 7fb68b7fe700 1 -- 172.21.15.88:0/3779817799 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7fb674006240 msgr2=0x7fb6740086c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-09-30T01:32:44.871 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:44.866+0000 7fb68b7fe700 1 -- 172.21.15.88:0/3779817799 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7fb674006240 msgr2=0x7fb6740086c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-09-30T01:32:44.871 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:44.866+0000 7fb68b7fe700 1 --2- 172.21.15.88:0/3779817799 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7fb674006240 0x7fb6740086c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2023-09-30T01:32:44.871 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:44.866+0000 7fb68b7fe700 1 --2- 172.21.15.88:0/3779817799 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7fb674006240 0x7fb6740086c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-09-30T01:32:44.871 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:44.866+0000 7fb68bfff700 1 -- 172.21.15.88:0/3779817799 >> v2:172.21.15.107:3300/0 conn(0x7fb674003840 msgr2=0x7fb674005c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-09-30T01:32:44.871 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:44.866+0000 7fb68bfff700 1 -- 172.21.15.88:0/3779817799 >> v2:172.21.15.107:3300/0 conn(0x7fb674003840 msgr2=0x7fb674005c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-09-30T01:32:44.872 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:44.866+0000 7fb68bfff700 1 --2- 172.21.15.88:0/3779817799 >> v2:172.21.15.107:3300/0 conn(0x7fb674003840 0x7fb674005c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2023-09-30T01:32:44.872 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:44.866+0000 7fb68bfff700 1 --2- 172.21.15.88:0/3779817799 >> v2:172.21.15.107:3300/0 conn(0x7fb674003840 0x7fb674005c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-09-30T01:32:45.672 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:45.670+0000 7fb68b7fe700 1 --2- 172.21.15.88:0/3779817799 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7fb674006240 0x7fb6740086c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:32:45.672 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:45.670+0000 7fb68bfff700 1 --2- 172.21.15.88:0/3779817799 >> v2:172.21.15.107:3300/0 conn(0x7fb674003840 0x7fb674005c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:32:45.672 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:45.670+0000 7fb68b7fe700 10 monclient(hunting): get_auth_request con 0x7fb674006240 auth_method 0 2023-09-30T01:32:45.672 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:45.670+0000 7fb68b7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:32:45.672 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:45.670+0000 7fb68b7fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:32:45.672 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:45.670+0000 7fb68b7fe700 10 monclient(hunting): _init_auth already have auth, reseting 2023-09-30T01:32:45.673 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:45.670+0000 7fb68bfff700 10 monclient(hunting): get_auth_request con 0x7fb674003840 auth_method 0 2023-09-30T01:32:45.673 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:45.670+0000 7fb68bfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:32:45.673 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:45.670+0000 7fb68bfff700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:32:45.673 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:45.670+0000 7fb68bfff700 10 monclient(hunting): _init_auth already have auth, reseting 2023-09-30T01:32:45.673 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:45.670+0000 7fb68b7fe700 1 -- 172.21.15.88:0/3779817799 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7fb674006240 msgr2=0x7fb6740086c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-09-30T01:32:45.673 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:45.670+0000 7fb68b7fe700 1 -- 172.21.15.88:0/3779817799 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7fb674006240 msgr2=0x7fb6740086c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-09-30T01:32:45.673 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:45.670+0000 7fb68b7fe700 1 --2- 172.21.15.88:0/3779817799 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7fb674006240 0x7fb6740086c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2023-09-30T01:32:45.673 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:45.670+0000 7fb68bfff700 1 -- 172.21.15.88:0/3779817799 >> v2:172.21.15.107:3300/0 conn(0x7fb674003840 msgr2=0x7fb674005c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-09-30T01:32:45.673 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:45.670+0000 7fb68bfff700 1 -- 172.21.15.88:0/3779817799 >> v2:172.21.15.107:3300/0 conn(0x7fb674003840 msgr2=0x7fb674005c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-09-30T01:32:45.673 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:45.670+0000 7fb68bfff700 1 --2- 172.21.15.88:0/3779817799 >> v2:172.21.15.107:3300/0 conn(0x7fb674003840 0x7fb674005c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2023-09-30T01:32:45.673 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:45.670+0000 7fb68b7fe700 1 --2- 172.21.15.88:0/3779817799 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7fb674006240 0x7fb6740086c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-09-30T01:32:45.673 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:45.670+0000 7fb68bfff700 1 --2- 172.21.15.88:0/3779817799 >> v2:172.21.15.107:3300/0 conn(0x7fb674003840 0x7fb674005c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-09-30T01:32:47.263 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:47.258+0000 7fb689ffb700 10 monclient(hunting): tick 2023-09-30T01:32:47.263 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:47.258+0000 7fb689ffb700 1 monclient(hunting): continuing hunt 2023-09-30T01:32:47.263 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:47.258+0000 7fb689ffb700 10 monclient(hunting): _reopen_session rank -1 2023-09-30T01:32:47.263 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:47.258+0000 7fb689ffb700 1 -- 172.21.15.88:0/3779817799 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7fb674006240 msgr2=0x7fb6740086c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-09-30T01:32:47.263 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:47.258+0000 7fb689ffb700 1 --2- 172.21.15.88:0/3779817799 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7fb674006240 0x7fb6740086c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:32:47.263 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:47.258+0000 7fb689ffb700 1 -- 172.21.15.88:0/3779817799 >> v2:172.21.15.107:3300/0 conn(0x7fb674003840 msgr2=0x7fb674005c40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-09-30T01:32:47.264 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:47.258+0000 7fb689ffb700 1 --2- 172.21.15.88:0/3779817799 >> v2:172.21.15.107:3300/0 conn(0x7fb674003840 0x7fb674005c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:32:47.264 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:47.258+0000 7fb689ffb700 1 -- 172.21.15.88:0/3779817799 >> v1:172.21.15.88:6789/0 conn(0x7fb674008d80 legacy=0x7fb67400b230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:32:47.264 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:47.258+0000 7fb690b0b700 1 -- 172.21.15.88:0/3779817799 reap_dead start 2023-09-30T01:32:47.264 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:47.258+0000 7fb689ffb700 10 monclient: _add_conns ranks=[6,2,3,1,8] 2023-09-30T01:32:47.264 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:47.258+0000 7fb690b0b700 1 -- 172.21.15.88:0/3779817799 reap_dead start 2023-09-30T01:32:47.264 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:47.258+0000 7fb689ffb700 1 --2- 172.21.15.88:0/3779817799 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7fb67400cd50 0x7fb674005c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:32:47.264 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:47.258+0000 7fb689ffb700 10 monclient(hunting): picked mon.noname-d con 0x7fb67400cd50 addr [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] 2023-09-30T01:32:47.264 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:47.258+0000 7fb689ffb700 1 --2- 172.21.15.88:0/3779817799 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb67400d1c0 0x7fb6740086a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:32:47.264 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:47.258+0000 7fb689ffb700 10 monclient(hunting): picked mon.noname-b con 0x7fb67400d1c0 addr [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] 2023-09-30T01:32:47.264 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:47.258+0000 7fb689ffb700 1 --2- 172.21.15.88:0/3779817799 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7fb67400dec0 0x7fb6740102a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:32:47.264 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:47.258+0000 7fb689ffb700 10 monclient(hunting): picked mon.noname-g con 0x7fb67400dec0 addr [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] 2023-09-30T01:32:47.264 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:47.258+0000 7fb689ffb700 10 monclient(hunting): start opening mon connection 2023-09-30T01:32:47.265 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:47.258+0000 7fb689ffb700 1 -- 172.21.15.88:0/3779817799 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb68c131590 con 0x7fb67400d1c0 2023-09-30T01:32:47.265 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:47.258+0000 7fb689ffb700 10 monclient(hunting): start opening mon connection 2023-09-30T01:32:47.265 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:47.258+0000 7fb689ffb700 1 -- 172.21.15.88:0/3779817799 --> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb68c09a5d0 con 0x7fb67400dec0 2023-09-30T01:32:47.265 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:47.258+0000 7fb689ffb700 10 monclient(hunting): start opening mon connection 2023-09-30T01:32:47.265 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:47.258+0000 7fb689ffb700 1 -- 172.21.15.88:0/3779817799 --> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] -- mon_getmap magic: 0 v1 -- 0x7fb68c097e10 con 0x7fb67400cd50 2023-09-30T01:32:47.265 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:47.258+0000 7fb689ffb700 10 monclient(hunting): _renew_subs 2023-09-30T01:32:47.265 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:47.258+0000 7fb68bfff700 1 --2- 172.21.15.88:0/3779817799 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7fb67400cd50 0x7fb674005c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:32:47.265 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:47.258+0000 7fb68b7fe700 1 --2- 172.21.15.88:0/3779817799 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb67400d1c0 0x7fb6740086a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:32:47.265 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:47.258+0000 7fb68bfff700 1 -- 172.21.15.88:0/3779817799 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7fb67400cd50 msgr2=0x7fb674005c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-09-30T01:32:47.265 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:47.258+0000 7fb68bfff700 1 -- 172.21.15.88:0/3779817799 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7fb67400cd50 msgr2=0x7fb674005c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-09-30T01:32:47.265 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:47.258+0000 7fb68b7fe700 10 monclient(hunting): get_auth_request con 0x7fb67400d1c0 auth_method 0 2023-09-30T01:32:47.265 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:47.258+0000 7fb68b7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:32:47.266 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:47.258+0000 7fb68b7fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:32:47.266 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:47.258+0000 7fb68bfff700 1 --2- 172.21.15.88:0/3779817799 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7fb67400cd50 0x7fb674005c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2023-09-30T01:32:47.266 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:47.258+0000 7fb68b7fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:32:47.266 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:47.258+0000 7fb68bfff700 1 --2- 172.21.15.88:0/3779817799 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7fb67400cd50 0x7fb674005c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-09-30T01:32:47.266 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:47.258+0000 7fb690b0b700 1 --2- 172.21.15.88:0/3779817799 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7fb67400dec0 0x7fb6740102a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:32:47.266 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:47.258+0000 7fb68b7fe700 1 -- 172.21.15.88:0/3779817799 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb67400d1c0 msgr2=0x7fb6740086a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-09-30T01:32:47.266 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:47.258+0000 7fb690b0b700 10 monclient(hunting): get_auth_request con 0x7fb67400dec0 auth_method 0 2023-09-30T01:32:47.266 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:47.258+0000 7fb68b7fe700 1 -- 172.21.15.88:0/3779817799 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb67400d1c0 msgr2=0x7fb6740086a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-09-30T01:32:47.266 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:47.258+0000 7fb690b0b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:32:47.266 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:47.258+0000 7fb690b0b700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:32:47.266 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:47.258+0000 7fb690b0b700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:32:47.266 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:47.258+0000 7fb68b7fe700 1 --2- 172.21.15.88:0/3779817799 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb67400d1c0 0x7fb6740086a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2023-09-30T01:32:47.267 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:47.258+0000 7fb68b7fe700 1 --2- 172.21.15.88:0/3779817799 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb67400d1c0 0x7fb6740086a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-09-30T01:32:47.267 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:47.262+0000 7fb690b0b700 1 -- 172.21.15.88:0/3779817799 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7fb67400dec0 msgr2=0x7fb6740102a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2023-09-30T01:32:47.267 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:47.262+0000 7fb690b0b700 1 -- 172.21.15.88:0/3779817799 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7fb67400dec0 msgr2=0x7fb6740102a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-09-30T01:32:47.267 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:47.262+0000 7fb690b0b700 1 --2- 172.21.15.88:0/3779817799 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7fb67400dec0 0x7fb6740102a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2023-09-30T01:32:47.267 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:47.262+0000 7fb690b0b700 1 --2- 172.21.15.88:0/3779817799 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7fb67400dec0 0x7fb6740102a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-09-30T01:32:47.464 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:47.462+0000 7fb68bfff700 1 --2- 172.21.15.88:0/3779817799 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7fb67400cd50 0x7fb674005c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:32:47.464 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:47.462+0000 7fb68bfff700 10 monclient(hunting): get_auth_request con 0x7fb67400cd50 auth_method 0 2023-09-30T01:32:47.464 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:47.462+0000 7fb68bfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:32:47.464 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:47.462+0000 7fb68bfff700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:32:47.464 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:47.462+0000 7fb68bfff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:32:47.465 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:47.462+0000 7fb68b7fe700 1 --2- 172.21.15.88:0/3779817799 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb67400d1c0 0x7fb6740086a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:32:47.465 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:47.462+0000 7fb68b7fe700 10 monclient(hunting): get_auth_request con 0x7fb67400d1c0 auth_method 0 2023-09-30T01:32:47.465 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:47.462+0000 7fb68b7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:32:47.465 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:47.462+0000 7fb68b7fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:32:47.465 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:47.462+0000 7fb68b7fe700 10 monclient(hunting): _init_auth already have auth, reseting 2023-09-30T01:32:47.465 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:47.462+0000 7fb68bfff700 1 -- 172.21.15.88:0/3779817799 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7fb67400cd50 msgr2=0x7fb674005c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-09-30T01:32:47.465 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:47.462+0000 7fb68bfff700 1 -- 172.21.15.88:0/3779817799 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7fb67400cd50 msgr2=0x7fb674005c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-09-30T01:32:47.466 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:47.462+0000 7fb68bfff700 1 --2- 172.21.15.88:0/3779817799 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7fb67400cd50 0x7fb674005c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2023-09-30T01:32:47.466 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:47.462+0000 7fb68bfff700 1 --2- 172.21.15.88:0/3779817799 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7fb67400cd50 0x7fb674005c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-09-30T01:32:47.466 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:47.462+0000 7fb68b7fe700 1 -- 172.21.15.88:0/3779817799 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb67400d1c0 msgr2=0x7fb6740086a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-09-30T01:32:47.466 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:47.462+0000 7fb68b7fe700 1 -- 172.21.15.88:0/3779817799 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb67400d1c0 msgr2=0x7fb6740086a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-09-30T01:32:47.466 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:47.462+0000 7fb68b7fe700 1 --2- 172.21.15.88:0/3779817799 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb67400d1c0 0x7fb6740086a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2023-09-30T01:32:47.466 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:47.462+0000 7fb68b7fe700 1 --2- 172.21.15.88:0/3779817799 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb67400d1c0 0x7fb6740086a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-09-30T01:32:47.467 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:47.466+0000 7fb690b0b700 1 --2- 172.21.15.88:0/3779817799 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7fb67400dec0 0x7fb6740102a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:32:47.468 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:47.466+0000 7fb690b0b700 10 monclient(hunting): get_auth_request con 0x7fb67400dec0 auth_method 0 2023-09-30T01:32:47.468 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:47.466+0000 7fb690b0b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:32:47.468 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:47.466+0000 7fb690b0b700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:32:47.468 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:47.466+0000 7fb690b0b700 10 monclient(hunting): _init_auth already have auth, reseting 2023-09-30T01:32:47.468 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:47.466+0000 7fb690b0b700 1 -- 172.21.15.88:0/3779817799 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7fb67400dec0 msgr2=0x7fb6740102a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-09-30T01:32:47.468 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:47.466+0000 7fb690b0b700 1 -- 172.21.15.88:0/3779817799 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7fb67400dec0 msgr2=0x7fb6740102a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-09-30T01:32:47.468 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:47.466+0000 7fb690b0b700 1 --2- 172.21.15.88:0/3779817799 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7fb67400dec0 0x7fb6740102a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2023-09-30T01:32:47.469 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:47.466+0000 7fb690b0b700 1 --2- 172.21.15.88:0/3779817799 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7fb67400dec0 0x7fb6740102a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-09-30T01:32:47.868 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:47.866+0000 7fb68bfff700 1 --2- 172.21.15.88:0/3779817799 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7fb67400cd50 0x7fb674005c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:32:47.868 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:47.866+0000 7fb68b7fe700 1 --2- 172.21.15.88:0/3779817799 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb67400d1c0 0x7fb6740086a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:32:47.868 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:47.866+0000 7fb68bfff700 10 monclient(hunting): get_auth_request con 0x7fb67400cd50 auth_method 0 2023-09-30T01:32:47.868 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:47.866+0000 7fb68bfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:32:47.868 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:47.866+0000 7fb68bfff700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:32:47.868 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:47.866+0000 7fb68bfff700 10 monclient(hunting): _init_auth already have auth, reseting 2023-09-30T01:32:47.868 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:47.866+0000 7fb68b7fe700 10 monclient(hunting): get_auth_request con 0x7fb67400d1c0 auth_method 0 2023-09-30T01:32:47.868 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:47.866+0000 7fb68b7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:32:47.869 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:47.866+0000 7fb68b7fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:32:47.869 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:47.866+0000 7fb68b7fe700 10 monclient(hunting): _init_auth already have auth, reseting 2023-09-30T01:32:47.869 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:47.866+0000 7fb68bfff700 1 -- 172.21.15.88:0/3779817799 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7fb67400cd50 msgr2=0x7fb674005c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-09-30T01:32:47.869 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:47.866+0000 7fb68bfff700 1 -- 172.21.15.88:0/3779817799 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7fb67400cd50 msgr2=0x7fb674005c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-09-30T01:32:47.869 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:47.866+0000 7fb68bfff700 1 --2- 172.21.15.88:0/3779817799 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7fb67400cd50 0x7fb674005c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2023-09-30T01:32:47.869 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:47.866+0000 7fb68b7fe700 1 -- 172.21.15.88:0/3779817799 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb67400d1c0 msgr2=0x7fb6740086a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-09-30T01:32:47.869 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:47.866+0000 7fb68bfff700 1 --2- 172.21.15.88:0/3779817799 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7fb67400cd50 0x7fb674005c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-09-30T01:32:47.869 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:47.866+0000 7fb68b7fe700 1 -- 172.21.15.88:0/3779817799 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb67400d1c0 msgr2=0x7fb6740086a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-09-30T01:32:47.869 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:47.866+0000 7fb68b7fe700 1 --2- 172.21.15.88:0/3779817799 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb67400d1c0 0x7fb6740086a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2023-09-30T01:32:47.869 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:47.866+0000 7fb68b7fe700 1 --2- 172.21.15.88:0/3779817799 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb67400d1c0 0x7fb6740086a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-09-30T01:32:47.871 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:47.870+0000 7fb690b0b700 1 --2- 172.21.15.88:0/3779817799 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7fb67400dec0 0x7fb6740102a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:32:47.871 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:47.870+0000 7fb690b0b700 10 monclient(hunting): get_auth_request con 0x7fb67400dec0 auth_method 0 2023-09-30T01:32:47.871 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:47.870+0000 7fb690b0b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:32:47.872 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:47.870+0000 7fb690b0b700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:32:47.872 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:47.870+0000 7fb690b0b700 10 monclient(hunting): _init_auth already have auth, reseting 2023-09-30T01:32:47.872 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:47.870+0000 7fb690b0b700 1 -- 172.21.15.88:0/3779817799 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7fb67400dec0 msgr2=0x7fb6740102a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-09-30T01:32:47.872 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:47.870+0000 7fb690b0b700 1 -- 172.21.15.88:0/3779817799 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7fb67400dec0 msgr2=0x7fb6740102a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-09-30T01:32:47.872 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:47.870+0000 7fb690b0b700 1 --2- 172.21.15.88:0/3779817799 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7fb67400dec0 0x7fb6740102a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2023-09-30T01:32:47.872 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:47.870+0000 7fb690b0b700 1 --2- 172.21.15.88:0/3779817799 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7fb67400dec0 0x7fb6740102a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-09-30T01:32:48.672 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:48.670+0000 7fb68bfff700 1 --2- 172.21.15.88:0/3779817799 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7fb67400cd50 0x7fb674005c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:32:48.672 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:48.670+0000 7fb68b7fe700 1 --2- 172.21.15.88:0/3779817799 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb67400d1c0 0x7fb6740086a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:32:48.672 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:48.670+0000 7fb68b7fe700 10 monclient(hunting): get_auth_request con 0x7fb67400d1c0 auth_method 0 2023-09-30T01:32:48.672 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:48.670+0000 7fb68b7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:32:48.672 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:48.670+0000 7fb68b7fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:32:48.672 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:48.670+0000 7fb68b7fe700 10 monclient(hunting): _init_auth already have auth, reseting 2023-09-30T01:32:48.672 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:48.670+0000 7fb68bfff700 10 monclient(hunting): get_auth_request con 0x7fb67400cd50 auth_method 0 2023-09-30T01:32:48.672 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:48.670+0000 7fb68bfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:32:48.673 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:48.670+0000 7fb68bfff700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:32:48.673 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:48.670+0000 7fb68bfff700 10 monclient(hunting): _init_auth already have auth, reseting 2023-09-30T01:32:48.673 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:48.670+0000 7fb68b7fe700 1 -- 172.21.15.88:0/3779817799 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb67400d1c0 msgr2=0x7fb6740086a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-09-30T01:32:48.673 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:48.670+0000 7fb68b7fe700 1 -- 172.21.15.88:0/3779817799 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb67400d1c0 msgr2=0x7fb6740086a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-09-30T01:32:48.673 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:48.670+0000 7fb68b7fe700 1 --2- 172.21.15.88:0/3779817799 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb67400d1c0 0x7fb6740086a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2023-09-30T01:32:48.673 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:48.670+0000 7fb68bfff700 1 -- 172.21.15.88:0/3779817799 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7fb67400cd50 msgr2=0x7fb674005c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-09-30T01:32:48.673 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:48.670+0000 7fb68bfff700 1 -- 172.21.15.88:0/3779817799 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7fb67400cd50 msgr2=0x7fb674005c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-09-30T01:32:48.673 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:48.670+0000 7fb68b7fe700 1 --2- 172.21.15.88:0/3779817799 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb67400d1c0 0x7fb6740086a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-09-30T01:32:48.673 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:48.670+0000 7fb68bfff700 1 --2- 172.21.15.88:0/3779817799 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7fb67400cd50 0x7fb674005c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2023-09-30T01:32:48.673 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:48.670+0000 7fb68bfff700 1 --2- 172.21.15.88:0/3779817799 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7fb67400cd50 0x7fb674005c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-09-30T01:32:48.675 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:48.674+0000 7fb690b0b700 1 --2- 172.21.15.88:0/3779817799 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7fb67400dec0 0x7fb6740102a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:32:48.675 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:48.674+0000 7fb690b0b700 10 monclient(hunting): get_auth_request con 0x7fb67400dec0 auth_method 0 2023-09-30T01:32:48.675 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:48.674+0000 7fb690b0b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:32:48.676 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:48.674+0000 7fb690b0b700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:32:48.676 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:48.674+0000 7fb690b0b700 10 monclient(hunting): _init_auth already have auth, reseting 2023-09-30T01:32:48.676 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:48.674+0000 7fb690b0b700 1 -- 172.21.15.88:0/3779817799 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7fb67400dec0 msgr2=0x7fb6740102a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-09-30T01:32:48.676 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:48.674+0000 7fb690b0b700 1 -- 172.21.15.88:0/3779817799 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7fb67400dec0 msgr2=0x7fb6740102a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-09-30T01:32:48.676 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:48.674+0000 7fb690b0b700 1 --2- 172.21.15.88:0/3779817799 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7fb67400dec0 0x7fb6740102a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2023-09-30T01:32:48.676 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:48.674+0000 7fb690b0b700 1 --2- 172.21.15.88:0/3779817799 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7fb67400dec0 0x7fb6740102a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-09-30T01:32:50.264 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:50.258+0000 7fb689ffb700 10 monclient(hunting): tick 2023-09-30T01:32:50.264 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:50.258+0000 7fb689ffb700 1 monclient(hunting): continuing hunt 2023-09-30T01:32:50.264 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:50.258+0000 7fb689ffb700 10 monclient(hunting): _reopen_session rank -1 2023-09-30T01:32:50.264 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:50.258+0000 7fb689ffb700 1 -- 172.21.15.88:0/3779817799 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7fb67400cd50 msgr2=0x7fb674005c40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-09-30T01:32:50.265 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:50.258+0000 7fb689ffb700 1 --2- 172.21.15.88:0/3779817799 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7fb67400cd50 0x7fb674005c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:32:50.265 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:50.258+0000 7fb689ffb700 1 -- 172.21.15.88:0/3779817799 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7fb67400dec0 msgr2=0x7fb6740102a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-09-30T01:32:50.265 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:50.258+0000 7fb689ffb700 1 --2- 172.21.15.88:0/3779817799 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7fb67400dec0 0x7fb6740102a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:32:50.265 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:50.258+0000 7fb689ffb700 1 -- 172.21.15.88:0/3779817799 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb67400d1c0 msgr2=0x7fb6740086a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-09-30T01:32:50.265 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:50.258+0000 7fb689ffb700 1 --2- 172.21.15.88:0/3779817799 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb67400d1c0 0x7fb6740086a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:32:50.265 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:50.258+0000 7fb689ffb700 10 monclient: _add_conns ranks=[8,1] 2023-09-30T01:32:50.266 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:50.258+0000 7fb689ffb700 1 --2- 172.21.15.88:0/3779817799 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7fb67400bab0 0x7fb67400b230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:32:50.266 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:50.258+0000 7fb689ffb700 10 monclient(hunting): picked mon.noname-e con 0x7fb67400bab0 addr [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] 2023-09-30T01:32:50.266 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:50.258+0000 7fb689ffb700 10 monclient(hunting): picked mon.noname-f-legacy con 0x7fb67400be80 addr v1:172.21.15.107:6789/0 2023-09-30T01:32:50.266 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:50.258+0000 7fb689ffb700 1 -- 172.21.15.88:0/3779817799 --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb67400dbd0 con 0x7fb67400be80 2023-09-30T01:32:50.266 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:50.258+0000 7fb689ffb700 10 monclient(hunting): start opening mon connection 2023-09-30T01:32:50.266 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:50.258+0000 7fb689ffb700 1 -- 172.21.15.88:0/3779817799 --> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] -- mon_getmap magic: 0 v1 -- 0x7fb68c131590 con 0x7fb67400bab0 2023-09-30T01:32:50.267 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:50.258+0000 7fb689ffb700 10 monclient(hunting): _renew_subs 2023-09-30T01:32:50.267 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:50.258+0000 7fb68bfff700 1 --2- 172.21.15.88:0/3779817799 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7fb67400bab0 0x7fb67400b230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:32:50.267 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:50.262+0000 7fb68bfff700 10 monclient(hunting): get_auth_request con 0x7fb67400bab0 auth_method 0 2023-09-30T01:32:50.267 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:50.262+0000 7fb68bfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:32:50.267 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:50.262+0000 7fb68bfff700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:32:50.267 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:50.262+0000 7fb68bfff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:32:50.267 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:50.262+0000 7fb68bfff700 1 -- 172.21.15.88:0/3779817799 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7fb67400bab0 msgr2=0x7fb67400b230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-09-30T01:32:50.267 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:50.262+0000 7fb68bfff700 1 -- 172.21.15.88:0/3779817799 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7fb67400bab0 msgr2=0x7fb67400b230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-09-30T01:32:50.268 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:50.262+0000 7fb68bfff700 1 --2- 172.21.15.88:0/3779817799 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7fb67400bab0 0x7fb67400b230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2023-09-30T01:32:50.268 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:50.262+0000 7fb68bfff700 1 --2- 172.21.15.88:0/3779817799 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7fb67400bab0 0x7fb67400b230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-09-30T01:32:50.467 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:50.466+0000 7fb68bfff700 1 --2- 172.21.15.88:0/3779817799 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7fb67400bab0 0x7fb67400b230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:32:50.468 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:50.466+0000 7fb68bfff700 10 monclient(hunting): get_auth_request con 0x7fb67400bab0 auth_method 0 2023-09-30T01:32:50.468 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:50.466+0000 7fb68bfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:32:50.468 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:50.466+0000 7fb68bfff700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:32:50.468 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:50.466+0000 7fb68bfff700 10 monclient(hunting): _init_auth already have auth, reseting 2023-09-30T01:32:50.468 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:50.466+0000 7fb68bfff700 1 -- 172.21.15.88:0/3779817799 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7fb67400bab0 msgr2=0x7fb67400b230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-09-30T01:32:50.468 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:50.466+0000 7fb68bfff700 1 -- 172.21.15.88:0/3779817799 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7fb67400bab0 msgr2=0x7fb67400b230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-09-30T01:32:50.468 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:50.466+0000 7fb68bfff700 1 --2- 172.21.15.88:0/3779817799 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7fb67400bab0 0x7fb67400b230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2023-09-30T01:32:50.468 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:50.466+0000 7fb68bfff700 1 --2- 172.21.15.88:0/3779817799 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7fb67400bab0 0x7fb67400b230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-09-30T01:32:50.872 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:50.870+0000 7fb68bfff700 1 --2- 172.21.15.88:0/3779817799 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7fb67400bab0 0x7fb67400b230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:32:50.872 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:50.870+0000 7fb68bfff700 10 monclient(hunting): get_auth_request con 0x7fb67400bab0 auth_method 0 2023-09-30T01:32:50.872 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:50.870+0000 7fb68bfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:32:50.872 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:50.870+0000 7fb68bfff700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:32:50.872 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:50.870+0000 7fb68bfff700 10 monclient(hunting): _init_auth already have auth, reseting 2023-09-30T01:32:50.872 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:50.870+0000 7fb68bfff700 1 -- 172.21.15.88:0/3779817799 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7fb67400bab0 msgr2=0x7fb67400b230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-09-30T01:32:50.872 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:50.870+0000 7fb68bfff700 1 -- 172.21.15.88:0/3779817799 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7fb67400bab0 msgr2=0x7fb67400b230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-09-30T01:32:50.872 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:50.870+0000 7fb68bfff700 1 --2- 172.21.15.88:0/3779817799 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7fb67400bab0 0x7fb67400b230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2023-09-30T01:32:50.873 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:50.870+0000 7fb68bfff700 1 --2- 172.21.15.88:0/3779817799 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7fb67400bab0 0x7fb67400b230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-09-30T01:32:51.538 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:51.534+0000 7fb68affd700 1 -- 172.21.15.88:0/3779817799 <== mon.1 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 561411882 0 0) 0x7fb67400dbd0 con 0x7fb67400be80 2023-09-30T01:32:51.539 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:51.534+0000 7fb68affd700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:32:51.539 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:51.534+0000 7fb68affd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:32:51.539 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:51.534+0000 7fb68affd700 10 monclient(hunting): my global_id is 4105 2023-09-30T01:32:51.539 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:51.534+0000 7fb68affd700 1 -- 172.21.15.88:0/3779817799 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb678003570 con 0x7fb67400be80 2023-09-30T01:32:51.543 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:51.538+0000 7fb68affd700 1 -- 172.21.15.88:0/3779817799 <== mon.1 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 61116341 0 0) 0x7fb678003570 con 0x7fb67400be80 2023-09-30T01:32:51.543 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:51.538+0000 7fb68affd700 1 -- 172.21.15.88:0/3779817799 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7fb67400bab0 msgr2=0x7fb67400b230 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-09-30T01:32:51.543 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:51.538+0000 7fb68affd700 1 --2- 172.21.15.88:0/3779817799 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7fb67400bab0 0x7fb67400b230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:32:51.543 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:51.538+0000 7fb68affd700 10 monclient: _finish_hunting 0 2023-09-30T01:32:51.543 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:51.538+0000 7fb68affd700 1 monclient: found mon.noname-f-legacy 2023-09-30T01:32:51.543 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:51.538+0000 7fb68affd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:32:51.543 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:51.538+0000 7fb68affd700 10 monclient: _send_mon_message to mon.noname-f-legacy at v1:172.21.15.107:6789/0 2023-09-30T01:32:51.543 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:51.538+0000 7fb68affd700 1 -- 172.21.15.88:0/3779817799 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb674014840 con 0x7fb67400be80 2023-09-30T01:32:51.544 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:51.538+0000 7fb68affd700 10 monclient: _finish_auth 0 2023-09-30T01:32:51.544 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:51.538+0000 7fb68affd700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:32:51.544 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:51.538+0000 7fb68affd700 1 -- 172.21.15.88:0/3779817799 <== mon.1 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (unknown 176869241 0 0) 0x7fb684009dd0 con 0x7fb67400be80 2023-09-30T01:32:51.544 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:51.538+0000 7fb69130c700 5 monclient: authenticate success, global_id 4105 2023-09-30T01:32:51.544 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:51.538+0000 7fb69130c700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:32:51.546 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:51.542+0000 7fb68affd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:32:51.546 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:51.542+0000 7fb68affd700 10 monclient: got monmap 1 from mon.noname-f-legacy (according to old e1) 2023-09-30T01:32:51.546 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:51.542+0000 7fb68affd700 10 monclient: dump: 2023-09-30T01:32:51.546 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:32:51.547 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:32:51.547 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:32:51.547 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:32:51.547 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:32:51.547 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:32:51.547 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:32:51.547 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:32:51.547 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:32:51.547 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:32:51.548 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:32:51.548 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:32:51.548 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:32:51.548 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:32:51.548 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:32:51.548 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:32:51.548 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:51.542+0000 7fb68affd700 1 monclient: mon.f has (v2) addrs [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] but i'm connected to v1:172.21.15.107:6789/0, reconnecting 2023-09-30T01:32:51.548 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:51.542+0000 7fb68affd700 10 monclient: _reopen_session rank -1 2023-09-30T01:32:51.548 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:51.542+0000 7fb68affd700 1 -- 172.21.15.88:0/3779817799 >> v1:172.21.15.107:6789/0 conn(0x7fb67400be80 legacy=0x7fb67400c280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:32:51.549 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:51.542+0000 7fb68affd700 10 monclient: _add_conns ranks=[3,1,6,8,7,0,4,2,5] 2023-09-30T01:32:51.549 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:51.542+0000 7fb68affd700 1 --2- 172.21.15.88:0/3779817799 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7fb67400dec0 0x7fb678004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:32:51.549 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:51.542+0000 7fb68affd700 10 monclient(hunting): picked mon.g con 0x7fb67400dec0 addr [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] 2023-09-30T01:32:51.549 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:51.542+0000 7fb68affd700 1 --2- 172.21.15.88:0/3779817799 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb6780043b0 0x7fb678004db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:32:51.549 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:51.542+0000 7fb68affd700 10 monclient(hunting): picked mon.f con 0x7fb6780043b0 addr [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] 2023-09-30T01:32:51.549 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:51.542+0000 7fb68affd700 1 --2- 172.21.15.88:0/3779817799 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7fb67400cd50 0x7fb67800ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:32:51.549 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:51.542+0000 7fb68affd700 10 monclient(hunting): picked mon.d con 0x7fb67400cd50 addr [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] 2023-09-30T01:32:51.549 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:51.542+0000 7fb68affd700 10 monclient(hunting): start opening mon connection 2023-09-30T01:32:51.549 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:51.542+0000 7fb68affd700 1 -- 172.21.15.88:0/3779817799 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb68c131590 con 0x7fb6780043b0 2023-09-30T01:32:51.550 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:51.542+0000 7fb68affd700 10 monclient(hunting): start opening mon connection 2023-09-30T01:32:51.550 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:51.542+0000 7fb68affd700 1 -- 172.21.15.88:0/3779817799 --> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb67800bfa0 con 0x7fb67400dec0 2023-09-30T01:32:51.550 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:51.542+0000 7fb68affd700 10 monclient(hunting): start opening mon connection 2023-09-30T01:32:51.550 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:51.542+0000 7fb68affd700 1 -- 172.21.15.88:0/3779817799 --> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] -- mon_getmap magic: 0 v1 -- 0x7fb6780148f0 con 0x7fb67400cd50 2023-09-30T01:32:51.550 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:51.542+0000 7fb68affd700 10 monclient(hunting): _renew_subs 2023-09-30T01:32:51.550 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:51.542+0000 7fb68affd700 10 monclient(hunting): _finish_auth 0 2023-09-30T01:32:51.550 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:51.542+0000 7fb69130c700 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2023-09-30T01:32:51.550 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:51.542+0000 7fb690b0b700 1 -- 172.21.15.88:0/3779817799 reap_dead start 2023-09-30T01:32:51.550 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:51.542+0000 7fb68b7fe700 1 --2- 172.21.15.88:0/3779817799 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb6780043b0 0x7fb678004db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:32:51.551 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:51.542+0000 7fb68b7fe700 10 monclient(hunting): get_auth_request con 0x7fb6780043b0 auth_method 0 2023-09-30T01:32:51.551 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:51.542+0000 7fb68b7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:32:51.551 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:51.542+0000 7fb68b7fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:32:51.551 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:51.542+0000 7fb68b7fe700 10 monclient(hunting): _init_auth already have auth, reseting 2023-09-30T01:32:51.551 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:51.542+0000 7fb68bfff700 1 --2- 172.21.15.88:0/3779817799 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7fb67400dec0 0x7fb678004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:32:51.551 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:51.542+0000 7fb68bfff700 10 monclient(hunting): get_auth_request con 0x7fb67400dec0 auth_method 0 2023-09-30T01:32:51.551 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:51.542+0000 7fb68bfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:32:51.551 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:51.542+0000 7fb68bfff700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:32:51.551 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:51.542+0000 7fb68bfff700 10 monclient(hunting): _init_auth already have auth, reseting 2023-09-30T01:32:51.552 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:51.542+0000 7fb690b0b700 1 --2- 172.21.15.88:0/3779817799 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7fb67400cd50 0x7fb67800ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:32:51.552 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:51.542+0000 7fb68b7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:32:51.552 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:51.542+0000 7fb68b7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:32:51.552 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:51.542+0000 7fb68b7fe700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-09-30T01:32:51.552 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:51.542+0000 7fb690b0b700 10 monclient(hunting): get_auth_request con 0x7fb67400cd50 auth_method 0 2023-09-30T01:32:51.552 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:51.542+0000 7fb690b0b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:32:51.552 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:51.542+0000 7fb690b0b700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:32:51.552 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:51.542+0000 7fb690b0b700 10 monclient(hunting): _init_auth already have auth, reseting 2023-09-30T01:32:51.552 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:51.542+0000 7fb68bfff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:32:51.553 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:51.542+0000 7fb68bfff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:32:51.553 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:51.542+0000 7fb68bfff700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-09-30T01:32:51.553 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:51.542+0000 7fb690b0b700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:32:51.553 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:51.542+0000 7fb690b0b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:32:51.553 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:51.542+0000 7fb690b0b700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-09-30T01:32:51.553 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:51.542+0000 7fb68b7fe700 10 monclient(hunting): handle_auth_done global_id 4105 payload 293 2023-09-30T01:32:51.553 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:51.542+0000 7fb68b7fe700 1 -- 172.21.15.88:0/3779817799 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7fb67400cd50 msgr2=0x7fb67800ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:32:51.553 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:51.542+0000 7fb68b7fe700 1 --2- 172.21.15.88:0/3779817799 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7fb67400cd50 0x7fb67800ba60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:32:51.554 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:51.542+0000 7fb68b7fe700 1 -- 172.21.15.88:0/3779817799 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7fb67400dec0 msgr2=0x7fb678004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:32:51.554 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:51.542+0000 7fb68b7fe700 1 --2- 172.21.15.88:0/3779817799 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7fb67400dec0 0x7fb678004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:32:51.554 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:51.542+0000 7fb68b7fe700 10 monclient: _finish_hunting 0 2023-09-30T01:32:51.554 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:51.542+0000 7fb68b7fe700 1 monclient: found mon.f 2023-09-30T01:32:51.554 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:51.542+0000 7fb68b7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:32:51.554 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:51.542+0000 7fb68b7fe700 10 monclient: _send_mon_message to mon.f at v2:172.21.15.107:3300/0 2023-09-30T01:32:51.554 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:51.542+0000 7fb68b7fe700 1 -- 172.21.15.88:0/3779817799 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb678003570 con 0x7fb6780043b0 2023-09-30T01:32:51.554 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:51.542+0000 7fb68b7fe700 10 monclient: _finish_auth 0 2023-09-30T01:32:51.554 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:51.542+0000 7fb68b7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:32:51.555 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:51.542+0000 7fb690b0b700 1 --2- 172.21.15.88:0/3779817799 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7fb67400cd50 0x7fb67800ba60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:32:51.555 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:51.542+0000 7fb68b7fe700 1 --2- 172.21.15.88:0/3779817799 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb6780043b0 0x7fb678004db0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fb68401a9c0 tx=0x7fb68400b040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cae3990a1d1dbb7e server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:32:51.555 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:51.542+0000 7fb68affd700 1 -- 172.21.15.88:0/3779817799 <== mon.1 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7fb684019ac0 con 0x7fb6780043b0 2023-09-30T01:32:51.555 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:51.542+0000 7fb68affd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:32:51.555 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:51.542+0000 7fb68affd700 10 monclient: got monmap 1 from mon.f (according to old e1) 2023-09-30T01:32:51.555 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:51.542+0000 7fb68affd700 10 monclient: dump: 2023-09-30T01:32:51.555 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:32:51.555 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:32:51.555 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:32:51.556 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:32:51.556 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:32:51.556 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:32:51.556 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:32:51.556 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:32:51.556 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:32:51.556 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:32:51.556 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:32:51.556 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:32:51.557 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:32:51.557 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:32:51.557 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:32:51.557 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:32:51.557 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:51.542+0000 7fb68affd700 1 -- 172.21.15.88:0/3779817799 <== mon.1 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb68400f040 con 0x7fb6780043b0 2023-09-30T01:32:51.557 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:51.542+0000 7fb68affd700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:32:51.557 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:51.542+0000 7fb68affd700 1 -- 172.21.15.88:0/3779817799 <== mon.1 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7fb684019c40 con 0x7fb6780043b0 2023-09-30T01:32:51.557 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:51.542+0000 7fb68affd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:32:51.557 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:51.542+0000 7fb68affd700 10 monclient: got monmap 1 from mon.f (according to old e1) 2023-09-30T01:32:51.557 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:51.542+0000 7fb68affd700 10 monclient: dump: 2023-09-30T01:32:51.558 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:32:51.558 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:32:51.558 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:32:51.558 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:32:51.558 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:32:51.558 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:32:51.558 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:32:51.558 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:32:51.558 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:32:51.558 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:32:51.559 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:32:51.559 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:32:51.559 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:32:51.559 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:32:51.559 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:32:51.559 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:32:51.559 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:51.542+0000 7fb69130c700 10 monclient: get_monmap_and_config success 2023-09-30T01:32:51.559 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:51.542+0000 7fb69130c700 4 set_mon_vals no callback set 2023-09-30T01:32:51.559 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:51.542+0000 7fb69130c700 10 monclient: shutdown 2023-09-30T01:32:51.560 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:51.542+0000 7fb69130c700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:32:51.560 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:51.542+0000 7fb69130c700 1 -- 172.21.15.88:0/3779817799 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb6780043b0 msgr2=0x7fb678004db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:32:51.560 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:51.542+0000 7fb69130c700 1 --2- 172.21.15.88:0/3779817799 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb6780043b0 0x7fb678004db0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fb68401a9c0 tx=0x7fb68400b040 comp rx=0 tx=0).stop 2023-09-30T01:32:51.560 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:51.542+0000 7fb69130c700 1 -- 172.21.15.88:0/3779817799 shutdown_connections 2023-09-30T01:32:51.560 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:51.542+0000 7fb69130c700 1 --2- 172.21.15.88:0/3779817799 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7fb67400cd50 0x7fb67800ba60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:32:51.560 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:51.542+0000 7fb69130c700 1 --2- 172.21.15.88:0/3779817799 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb6780043b0 0x7fb678004db0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:32:51.560 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:51.542+0000 7fb69130c700 1 --2- 172.21.15.88:0/3779817799 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7fb67400dec0 0x7fb678004810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:32:51.560 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:51.542+0000 7fb69130c700 1 -- 172.21.15.88:0/3779817799 >> 172.21.15.88:0/3779817799 conn(0x7fb68c06e5e0 msgr2=0x7fb68c06e9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:32:51.564 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:51.542+0000 7fb69130c700 1 -- 172.21.15.88:0/3779817799 shutdown_connections 2023-09-30T01:32:51.564 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:51.542+0000 7fb69130c700 1 -- 172.21.15.88:0/3779817799 wait complete. 2023-09-30T01:32:51.564 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:51.546+0000 7fb69130c700 10 monclient: build_initial_monmap 2023-09-30T01:32:51.564 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:51.546+0000 7fb69130c700 10 monclient: monmap: 2023-09-30T01:32:51.565 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2023-09-30T01:32:51.565 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:32:51.565 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 0.000000 2023-09-30T01:32:51.565 INFO:teuthology.orchestra.run.smithi088.stderr:created 0.000000 2023-09-30T01:32:51.565 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2023-09-30T01:32:51.565 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 1 2023-09-30T01:32:51.565 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.noname-a 2023-09-30T01:32:51.565 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.noname-f 2023-09-30T01:32:51.565 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.noname-b 2023-09-30T01:32:51.566 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.noname-g 2023-09-30T01:32:51.566 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.noname-c 2023-09-30T01:32:51.566 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.noname-h 2023-09-30T01:32:51.566 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.noname-d 2023-09-30T01:32:51.566 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.noname-i 2023-09-30T01:32:51.566 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.noname-e 2023-09-30T01:32:51.566 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:32:51.566 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:51.546+0000 7fb69130c700 1 Processor -- start 2023-09-30T01:32:51.566 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:51.546+0000 7fb69130c700 1 -- start start 2023-09-30T01:32:51.566 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:51.546+0000 7fb69130c700 10 monclient: init 2023-09-30T01:32:51.567 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:51.546+0000 7fb69130c700 10 monclient: _reopen_session rank -1 2023-09-30T01:32:51.567 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:51.546+0000 7fb69130c700 10 monclient: _add_conns ranks=[5,8,7,4,2,1,6,3,0] 2023-09-30T01:32:51.567 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:51.546+0000 7fb69130c700 1 --2- >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7fb67400cd50 0x7fb68c13d920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:32:51.567 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:51.546+0000 7fb69130c700 10 monclient(hunting): picked mon.noname-h con 0x7fb67400cd50 addr [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] 2023-09-30T01:32:51.567 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:51.546+0000 7fb69130c700 1 --2- >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7fb67400dec0 0x7fb68c0a2520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:32:51.567 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:51.546+0000 7fb69130c700 10 monclient(hunting): picked mon.noname-e con 0x7fb67400dec0 addr [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] 2023-09-30T01:32:51.567 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:51.546+0000 7fb69130c700 1 --2- >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7fb6780043b0 0x7fb68c06c3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:32:51.567 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:51.546+0000 7fb69130c700 10 monclient(hunting): picked mon.noname-i con 0x7fb6780043b0 addr [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] 2023-09-30T01:32:51.567 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:51.546+0000 7fb69130c700 10 monclient(hunting): start opening mon connection 2023-09-30T01:32:51.568 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:51.546+0000 7fb69130c700 1 -- --> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] -- mon_getmap magic: 0 v1 -- 0x7fb68c099c30 con 0x7fb67400cd50 2023-09-30T01:32:51.568 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:51.546+0000 7fb69130c700 10 monclient(hunting): start opening mon connection 2023-09-30T01:32:51.568 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:51.546+0000 7fb69130c700 1 -- --> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] -- mon_getmap magic: 0 v1 -- 0x7fb68c0996b0 con 0x7fb6780043b0 2023-09-30T01:32:51.568 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:51.546+0000 7fb69130c700 10 monclient(hunting): start opening mon connection 2023-09-30T01:32:51.568 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:51.546+0000 7fb69130c700 1 -- --> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] -- mon_getmap magic: 0 v1 -- 0x7fb68c13e7e0 con 0x7fb67400dec0 2023-09-30T01:32:51.568 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:51.546+0000 7fb69130c700 10 monclient(hunting): _renew_subs 2023-09-30T01:32:51.568 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:51.546+0000 7fb69130c700 10 monclient(hunting): authenticate will time out at 843.586508s 2023-09-30T01:32:51.568 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:51.546+0000 7fb68bfff700 1 --2- >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7fb67400cd50 0x7fb68c13d920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:32:51.568 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:51.546+0000 7fb68b7fe700 1 --2- >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7fb67400dec0 0x7fb68c0a2520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:32:51.568 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:51.546+0000 7fb68b7fe700 1 --2- >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7fb67400dec0 0x7fb68c0a2520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3304/0 says I am v2:172.21.15.88:52298/0 (socket says 172.21.15.88:52298) 2023-09-30T01:32:51.569 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:51.546+0000 7fb68bfff700 1 --2- >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7fb67400cd50 0x7fb68c13d920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3302/0 says I am v2:172.21.15.88:43278/0 (socket says 172.21.15.88:43278) 2023-09-30T01:32:51.569 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:51.546+0000 7fb68b7fe700 1 -- 172.21.15.88:0/3403977142 learned_addr learned my addr 172.21.15.88:0/3403977142 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-09-30T01:32:51.569 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:51.546+0000 7fb68b7fe700 10 monclient(hunting): get_auth_request con 0x7fb67400dec0 auth_method 0 2023-09-30T01:32:51.569 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:51.546+0000 7fb68b7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:32:51.569 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:51.546+0000 7fb68b7fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:32:51.569 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:51.546+0000 7fb68b7fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:32:51.569 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:51.546+0000 7fb68bfff700 10 monclient(hunting): get_auth_request con 0x7fb67400cd50 auth_method 0 2023-09-30T01:32:51.569 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:51.546+0000 7fb68bfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:32:51.569 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:51.546+0000 7fb68bfff700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:32:51.570 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:51.546+0000 7fb68bfff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:32:51.570 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:51.546+0000 7fb690b0b700 1 --2- 172.21.15.88:0/3403977142 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7fb6780043b0 0x7fb68c06c3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:32:51.570 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:51.546+0000 7fb690b0b700 10 monclient(hunting): get_auth_request con 0x7fb6780043b0 auth_method 0 2023-09-30T01:32:51.570 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:51.546+0000 7fb690b0b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:32:51.570 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:51.546+0000 7fb690b0b700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:32:51.570 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:51.546+0000 7fb690b0b700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:32:51.570 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:51.546+0000 7fb68b7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:32:51.570 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:51.546+0000 7fb68b7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:32:51.570 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:51.546+0000 7fb68b7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:32:51.570 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:51.546+0000 7fb68bfff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:32:51.570 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:51.546+0000 7fb68bfff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:32:51.571 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:51.546+0000 7fb68bfff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:32:51.571 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:51.546+0000 7fb690b0b700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:32:51.571 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:51.546+0000 7fb690b0b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:32:51.571 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:51.546+0000 7fb690b0b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:32:51.571 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:51.546+0000 7fb68b7fe700 10 monclient(hunting): handle_auth_done global_id 4112 payload 824 2023-09-30T01:32:51.571 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:51.546+0000 7fb68b7fe700 1 -- 172.21.15.88:0/3403977142 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7fb6780043b0 msgr2=0x7fb68c06c3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:32:51.571 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:51.546+0000 7fb68b7fe700 1 --2- 172.21.15.88:0/3403977142 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7fb6780043b0 0x7fb68c06c3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:32:51.571 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:51.546+0000 7fb68b7fe700 1 -- 172.21.15.88:0/3403977142 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7fb67400cd50 msgr2=0x7fb68c13d920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:32:51.573 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:51.546+0000 7fb68b7fe700 1 --2- 172.21.15.88:0/3403977142 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7fb67400cd50 0x7fb68c13d920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:32:51.573 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:51.546+0000 7fb68b7fe700 10 monclient: _finish_hunting 0 2023-09-30T01:32:51.573 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:51.546+0000 7fb68b7fe700 1 monclient: found mon.noname-e 2023-09-30T01:32:51.573 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:51.546+0000 7fb68b7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:32:51.573 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:51.546+0000 7fb68b7fe700 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.88:3304/0 2023-09-30T01:32:51.574 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:51.546+0000 7fb68b7fe700 1 -- 172.21.15.88:0/3403977142 --> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb68c09c710 con 0x7fb67400dec0 2023-09-30T01:32:51.574 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:51.546+0000 7fb68b7fe700 1 --2- 172.21.15.88:0/3403977142 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7fb67400dec0 0x7fb68c0a2520 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fb6780041e0 tx=0x7fb684029000 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:32:51.574 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:51.550+0000 7fb673fff700 1 -- 172.21.15.88:0/3403977142 <== mon.8 v2:172.21.15.88:3304/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7fb684019ea0 con 0x7fb67400dec0 2023-09-30T01:32:51.574 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:51.550+0000 7fb673fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:32:51.574 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:51.550+0000 7fb673fff700 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2023-09-30T01:32:51.574 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:51.550+0000 7fb673fff700 10 monclient: dump: 2023-09-30T01:32:51.574 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:32:51.574 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:32:51.574 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:32:51.574 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:32:51.574 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:32:51.575 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:32:51.575 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:32:51.575 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:32:51.575 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:32:51.575 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:32:51.575 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:32:51.575 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:32:51.575 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:32:51.575 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:32:51.575 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:32:51.575 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:32:51.575 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:51.550+0000 7fb673fff700 10 monclient: _finish_auth 0 2023-09-30T01:32:51.576 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:51.550+0000 7fb673fff700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:32:51.576 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:51.550+0000 7fb69130c700 5 monclient: authenticate success, global_id 4112 2023-09-30T01:32:51.576 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:51.550+0000 7fb69130c700 10 monclient: _renew_subs 2023-09-30T01:32:51.576 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:51.550+0000 7fb69130c700 10 monclient: _send_mon_message to mon.e at v2:172.21.15.88:3304/0 2023-09-30T01:32:51.576 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:51.550+0000 7fb69130c700 1 -- 172.21.15.88:0/3403977142 --> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb68c09c9f0 con 0x7fb67400dec0 2023-09-30T01:32:51.576 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:51.550+0000 7fb69130c700 10 monclient: _renew_subs 2023-09-30T01:32:51.576 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:51.550+0000 7fb69130c700 10 monclient: _send_mon_message to mon.e at v2:172.21.15.88:3304/0 2023-09-30T01:32:51.576 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:51.550+0000 7fb69130c700 1 -- 172.21.15.88:0/3403977142 --> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb68c09b820 con 0x7fb67400dec0 2023-09-30T01:32:51.576 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:51.550+0000 7fb673fff700 1 -- 172.21.15.88:0/3403977142 <== mon.8 v2:172.21.15.88:3304/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb68402f440 con 0x7fb67400dec0 2023-09-30T01:32:51.576 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:51.550+0000 7fb673fff700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:32:51.576 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:51.550+0000 7fb673fff700 1 -- 172.21.15.88:0/3403977142 <== mon.8 v2:172.21.15.88:3304/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7fb68402fa70 con 0x7fb67400dec0 2023-09-30T01:32:51.576 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:51.550+0000 7fb673fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:32:51.576 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:51.550+0000 7fb69130c700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:32:51.577 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:51.550+0000 7fb673fff700 10 monclient: got monmap 1 from mon.e (according to old e1) 2023-09-30T01:32:51.577 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:51.550+0000 7fb68a7fc700 4 set_mon_vals no callback set 2023-09-30T01:32:51.577 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:51.550+0000 7fb673fff700 10 monclient: dump: 2023-09-30T01:32:51.577 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:32:51.577 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:32:51.577 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:32:51.577 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:32:51.577 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:32:51.577 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:32:51.577 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:32:51.577 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:32:51.577 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:32:51.577 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:32:51.578 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:32:51.578 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:32:51.578 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:32:51.578 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:32:51.578 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:32:51.578 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:32:51.578 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:51.550+0000 7fb673fff700 1 -- 172.21.15.88:0/3403977142 <== mon.8 v2:172.21.15.88:3304/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7fb684003040 con 0x7fb67400dec0 2023-09-30T01:32:51.578 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:51.550+0000 7fb673fff700 1 -- 172.21.15.88:0/3403977142 <== mon.8 v2:172.21.15.88:3304/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7fb6840387f0 con 0x7fb67400dec0 2023-09-30T01:32:51.578 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:51.554+0000 7fb69130c700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:32:51.578 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:51.554+0000 7fb69130c700 10 monclient: _send_mon_message to mon.e at v2:172.21.15.88:3304/0 2023-09-30T01:32:51.578 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:51.554+0000 7fb69130c700 1 -- 172.21.15.88:0/3403977142 --> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb68c09ce80 con 0x7fb67400dec0 2023-09-30T01:32:51.578 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:51.558+0000 7fb673fff700 1 -- 172.21.15.88:0/3403977142 <== mon.8 v2:172.21.15.88:3304/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7fb6840566d0 con 0x7fb67400dec0 2023-09-30T01:32:51.578 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:51.558+0000 7fb673fff700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:32:51.579 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:51.558+0000 7fb673fff700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:32:51.624 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:51.622+0000 7fb69130c700 10 start_mon_command cmd=[{"prefix": "osd crush tunables", "profile": "default"}] 2023-09-30T01:32:51.625 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:51.622+0000 7fb69130c700 10 monclient: _send_command 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2023-09-30T01:32:51.625 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:51.622+0000 7fb69130c700 10 monclient: _send_mon_message to mon.e at v2:172.21.15.88:3304/0 2023-09-30T01:32:51.625 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:51.622+0000 7fb69130c700 1 -- 172.21.15.88:0/3403977142 --> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7fb68c071960 con 0x7fb67400dec0 2023-09-30T01:32:51.634 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:51.630+0000 7fb673fff700 1 -- 172.21.15.88:0/3403977142 <== mon.8 v2:172.21.15.88:3304/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) v1 ==== 124+0+0 (secure 0 0 0) 0x7fb68c09ce80 con 0x7fb67400dec0 2023-09-30T01:32:51.634 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:51.630+0000 7fb673fff700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2023-09-30T01:32:51.634 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:51.630+0000 7fb673fff700 10 monclient: _finish_command 2 = system:0 adjusted tunables profile to default 2023-09-30T01:32:51.636 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:51.634+0000 7fb69130c700 10 monclient: shutdown 2023-09-30T01:32:51.636 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:51.634+0000 7fb69130c700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:32:51.636 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:51.634+0000 7fb69130c700 1 -- 172.21.15.88:0/3403977142 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7fb67400dec0 msgr2=0x7fb68c0a2520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:32:51.636 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:51.634+0000 7fb69130c700 1 --2- 172.21.15.88:0/3403977142 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7fb67400dec0 0x7fb68c0a2520 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fb6780041e0 tx=0x7fb684029000 comp rx=0 tx=0).stop 2023-09-30T01:32:51.636 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:51.634+0000 7fb69130c700 1 -- 172.21.15.88:0/3403977142 shutdown_connections 2023-09-30T01:32:51.637 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:51.634+0000 7fb69130c700 1 --2- 172.21.15.88:0/3403977142 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7fb6780043b0 0x7fb68c06c3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:32:51.637 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:51.634+0000 7fb69130c700 1 --2- 172.21.15.88:0/3403977142 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7fb67400dec0 0x7fb68c0a2520 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:32:51.637 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:51.634+0000 7fb69130c700 1 --2- 172.21.15.88:0/3403977142 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7fb67400cd50 0x7fb68c13d920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:32:51.637 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:51.634+0000 7fb69130c700 1 -- 172.21.15.88:0/3403977142 >> 172.21.15.88:0/3403977142 conn(0x7fb68c06e5e0 msgr2=0x7fb68c12e200 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:32:51.637 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:51.634+0000 7fb69130c700 1 -- 172.21.15.88:0/3403977142 shutdown_connections 2023-09-30T01:32:51.637 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:51.634+0000 7fb69130c700 1 -- 172.21.15.88:0/3403977142 wait complete. 2023-09-30T01:32:51.637 INFO:teuthology.orchestra.run.smithi088.stderr:adjusted tunables profile to default 2023-09-30T01:32:51.648 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2023-09-30T01:32:51.648 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2023-09-30T01:32:51.648 DEBUG:teuthology.orchestra.run.smithi088:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2023-09-30T01:32:51.662 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2023-09-30T01:32:51.662 DEBUG:teuthology.orchestra.run.smithi088:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2023-09-30T01:32:51.673 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2023-09-30T01:32:51.673 DEBUG:teuthology.orchestra.run.smithi088:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2023-09-30T01:32:51.682 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-09-30T01:32:51.682 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2023-09-30T01:32:51.699 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-09-30T01:32:51.699 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2023-09-30T01:32:51.731 INFO:tasks.ceph.mgr.x.smithi107.stderr:2023-09-30T01:32:51.725+0000 7f46b28dd000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-09-30T01:32:51.747 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-09-30T01:32:51.747 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2023-09-30T01:32:51.807 DEBUG:teuthology.orchestra.run.smithi107:> sudo ceph --cluster ceph osd new 1243f825-b99c-4546-a77d-90b8637698b4 0 2023-09-30T01:32:51.910 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:51.905+0000 7f7016b16700 10 monclient: get_monmap_and_config 2023-09-30T01:32:51.910 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:51.905+0000 7f7016b16700 10 monclient: build_initial_monmap 2023-09-30T01:32:51.910 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:51.905+0000 7f7016b16700 10 monclient: monmap: 2023-09-30T01:32:51.910 INFO:teuthology.orchestra.run.smithi107.stderr:epoch 0 2023-09-30T01:32:51.910 INFO:teuthology.orchestra.run.smithi107.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:32:51.910 INFO:teuthology.orchestra.run.smithi107.stderr:last_changed 2023-09-30T01:32:51.910089+0000 2023-09-30T01:32:51.910 INFO:teuthology.orchestra.run.smithi107.stderr:created 2023-09-30T01:32:51.910089+0000 2023-09-30T01:32:51.910 INFO:teuthology.orchestra.run.smithi107.stderr:min_mon_release 0 (unknown) 2023-09-30T01:32:51.911 INFO:teuthology.orchestra.run.smithi107.stderr:election_strategy: 3 2023-09-30T01:32:51.911 INFO:teuthology.orchestra.run.smithi107.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a-legacy 2023-09-30T01:32:51.911 INFO:teuthology.orchestra.run.smithi107.stderr:1: v1:172.21.15.107:6789/0 mon.noname-f-legacy 2023-09-30T01:32:51.911 INFO:teuthology.orchestra.run.smithi107.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.noname-b 2023-09-30T01:32:51.911 INFO:teuthology.orchestra.run.smithi107.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.noname-g 2023-09-30T01:32:51.911 INFO:teuthology.orchestra.run.smithi107.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.noname-c 2023-09-30T01:32:51.911 INFO:teuthology.orchestra.run.smithi107.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.noname-h 2023-09-30T01:32:51.911 INFO:teuthology.orchestra.run.smithi107.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.noname-d 2023-09-30T01:32:51.911 INFO:teuthology.orchestra.run.smithi107.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.noname-i 2023-09-30T01:32:51.911 INFO:teuthology.orchestra.run.smithi107.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.noname-e 2023-09-30T01:32:51.911 INFO:teuthology.orchestra.run.smithi107.stderr:9: v2:172.21.15.88:3300/0 mon.noname-a 2023-09-30T01:32:51.911 INFO:teuthology.orchestra.run.smithi107.stderr:10: v2:172.21.15.107:3300/0 mon.noname-f 2023-09-30T01:32:51.912 INFO:teuthology.orchestra.run.smithi107.stderr: 2023-09-30T01:32:51.920 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:51.905+0000 7f7016b16700 1 Processor -- start 2023-09-30T01:32:51.920 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:51.905+0000 7f7016b16700 1 -- start start 2023-09-30T01:32:51.920 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:51.905+0000 7f7016b16700 10 monclient: init 2023-09-30T01:32:51.920 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:51.905+0000 7f7016b16700 10 monclient: _reopen_session rank -1 2023-09-30T01:32:51.920 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:51.905+0000 7f7016b16700 10 monclient: _add_conns ranks=[2,5,3,9,6,10,7,1,0,4,8] 2023-09-30T01:32:51.920 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:51.905+0000 7f7016b16700 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f7010097760 0x7f7010098ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:32:51.920 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:51.905+0000 7f7016b16700 10 monclient(hunting): picked mon.noname-b con 0x7f7010097760 addr [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] 2023-09-30T01:32:51.920 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:51.905+0000 7f7016b16700 1 --2- >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7f7010099400 0x7f701009dd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:32:51.920 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:51.905+0000 7f7016b16700 10 monclient(hunting): picked mon.noname-h con 0x7f7010099400 addr [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] 2023-09-30T01:32:51.920 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:51.905+0000 7f7016b16700 1 --2- >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7f701009e2b0 0x7f701009e6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:32:51.921 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:51.905+0000 7f7016b16700 10 monclient(hunting): picked mon.noname-g con 0x7f701009e2b0 addr [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] 2023-09-30T01:32:51.921 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:51.905+0000 7f7016b16700 10 monclient(hunting): start opening mon connection 2023-09-30T01:32:51.921 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:51.905+0000 7f7016b16700 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7010097fb0 con 0x7f7010097760 2023-09-30T01:32:51.921 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:51.905+0000 7f7016b16700 10 monclient(hunting): start opening mon connection 2023-09-30T01:32:51.921 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:51.905+0000 7f7016b16700 1 -- --> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f701009ec20 con 0x7f701009e2b0 2023-09-30T01:32:51.921 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:51.905+0000 7f7016b16700 10 monclient(hunting): start opening mon connection 2023-09-30T01:32:51.921 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:51.905+0000 7f7016b16700 1 -- --> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] -- mon_getmap magic: 0 v1 -- 0x7f70100987c0 con 0x7f7010099400 2023-09-30T01:32:51.921 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:51.905+0000 7f7016b16700 10 monclient(hunting): _renew_subs 2023-09-30T01:32:51.921 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:51.905+0000 7f7016b16700 10 monclient(hunting): authenticate will time out at 855.331744s 2023-09-30T01:32:51.922 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:51.905+0000 7f7016315700 1 --2- >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7f701009e2b0 0x7f701009e6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:32:51.922 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:51.905+0000 7f7015313700 1 --2- >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7f7010099400 0x7f701009dd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:32:51.922 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:51.905+0000 7f7016315700 1 --2- >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7f701009e2b0 0x7f701009e6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3301/0 says I am v2:172.21.15.107:39462/0 (socket says 172.21.15.107:39462) 2023-09-30T01:32:51.922 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:51.905+0000 7f7015313700 1 --2- >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7f7010099400 0x7f701009dd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3302/0 says I am v2:172.21.15.107:36230/0 (socket says 172.21.15.107:36230) 2023-09-30T01:32:51.922 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:51.905+0000 7f7016315700 1 -- 172.21.15.107:0/1957206680 learned_addr learned my addr 172.21.15.107:0/1957206680 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-09-30T01:32:51.922 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:51.905+0000 7f7016315700 10 monclient(hunting): get_auth_request con 0x7f701009e2b0 auth_method 0 2023-09-30T01:32:51.922 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:51.905+0000 7f7016315700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:32:51.922 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:51.905+0000 7f7016315700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:32:51.922 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:51.905+0000 7f7016315700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:32:51.922 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:51.909+0000 7f7015313700 10 monclient(hunting): get_auth_request con 0x7f7010099400 auth_method 0 2023-09-30T01:32:51.922 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:51.909+0000 7f7015313700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:32:51.923 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:51.909+0000 7f7015313700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:32:51.923 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:51.909+0000 7f7015313700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:32:51.923 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:51.909+0000 7f7016315700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:32:51.923 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:51.909+0000 7f7016315700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:32:51.923 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:51.909+0000 7f7016315700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:32:51.923 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:51.909+0000 7f7015313700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:32:51.923 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:51.909+0000 7f7015313700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:32:51.923 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:51.909+0000 7f7015313700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:32:51.923 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:51.909+0000 7f7015b14700 1 --2- 172.21.15.107:0/1957206680 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f7010097760 0x7f7010098ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:32:51.923 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:51.909+0000 7f7015b14700 10 monclient(hunting): get_auth_request con 0x7f7010097760 auth_method 0 2023-09-30T01:32:51.924 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:51.909+0000 7f7015b14700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:32:51.924 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:51.909+0000 7f7015b14700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:32:51.924 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:51.909+0000 7f7015b14700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:32:51.924 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:51.909+0000 7f7016315700 10 monclient(hunting): handle_auth_done global_id 4107 payload 274 2023-09-30T01:32:51.924 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:51.909+0000 7f7016315700 1 -- 172.21.15.107:0/1957206680 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7f7010099400 msgr2=0x7f701009dd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:32:51.924 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:51.909+0000 7f7016315700 1 --2- 172.21.15.107:0/1957206680 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7f7010099400 0x7f701009dd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:32:51.924 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:51.909+0000 7f7016315700 1 -- 172.21.15.107:0/1957206680 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f7010097760 msgr2=0x7f7010098ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:32:51.924 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:51.909+0000 7f7016315700 1 --2- 172.21.15.107:0/1957206680 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f7010097760 0x7f7010098ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:32:51.924 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:51.909+0000 7f7016315700 10 monclient: _finish_hunting 0 2023-09-30T01:32:51.924 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:51.909+0000 7f7016315700 1 monclient: found mon.noname-g 2023-09-30T01:32:51.924 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:51.909+0000 7f7016315700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:32:51.924 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:51.909+0000 7f7016315700 10 monclient: _send_mon_message to mon.noname-g at v2:172.21.15.107:3301/0 2023-09-30T01:32:51.925 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:51.909+0000 7f7016315700 1 -- 172.21.15.107:0/1957206680 --> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f701004bb20 con 0x7f701009e2b0 2023-09-30T01:32:51.925 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:51.909+0000 7f7015313700 1 --2- 172.21.15.107:0/1957206680 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7f7010099400 0x7f701009dd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:32:51.925 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:51.909+0000 7f7015b14700 1 --2- 172.21.15.107:0/1957206680 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f7010097760 0x7f7010098ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-30T01:32:51.925 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:51.909+0000 7f7016315700 1 --2- 172.21.15.107:0/1957206680 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7f701009e2b0 0x7f701009e6e0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f700c009a10 tx=0x7f700c009250 comp rx=0 tx=0).ready entity=mon.3 client_cookie=1dc1b9613bdcdadc server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:32:51.925 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:51.909+0000 7f7014b12700 1 -- 172.21.15.107:0/1957206680 <== mon.3 v2:172.21.15.107:3301/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f700c00bca0 con 0x7f701009e2b0 2023-09-30T01:32:51.925 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:51.909+0000 7f7014b12700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:32:51.925 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:51.909+0000 7f7014b12700 10 monclient: got monmap 1 from mon.noname-g (according to old e1) 2023-09-30T01:32:51.925 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:51.909+0000 7f7014b12700 10 monclient: dump: 2023-09-30T01:32:51.925 INFO:teuthology.orchestra.run.smithi107.stderr:epoch 1 2023-09-30T01:32:51.925 INFO:teuthology.orchestra.run.smithi107.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:32:51.925 INFO:teuthology.orchestra.run.smithi107.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:32:51.925 INFO:teuthology.orchestra.run.smithi107.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:32:51.925 INFO:teuthology.orchestra.run.smithi107.stderr:min_mon_release 17 (quincy) 2023-09-30T01:32:51.926 INFO:teuthology.orchestra.run.smithi107.stderr:election_strategy: 3 2023-09-30T01:32:51.926 INFO:teuthology.orchestra.run.smithi107.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:32:51.926 INFO:teuthology.orchestra.run.smithi107.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:32:51.926 INFO:teuthology.orchestra.run.smithi107.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:32:51.926 INFO:teuthology.orchestra.run.smithi107.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:32:51.926 INFO:teuthology.orchestra.run.smithi107.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:32:51.926 INFO:teuthology.orchestra.run.smithi107.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:32:51.926 INFO:teuthology.orchestra.run.smithi107.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:32:51.926 INFO:teuthology.orchestra.run.smithi107.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:32:51.926 INFO:teuthology.orchestra.run.smithi107.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:32:51.926 INFO:teuthology.orchestra.run.smithi107.stderr: 2023-09-30T01:32:51.926 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:51.909+0000 7f7014b12700 10 monclient: _finish_auth 0 2023-09-30T01:32:51.927 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:51.909+0000 7f7014b12700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:32:51.927 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:51.909+0000 7f7014b12700 1 -- 172.21.15.107:0/1957206680 <== mon.3 v2:172.21.15.107:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f700c00be20 con 0x7f701009e2b0 2023-09-30T01:32:51.927 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:51.909+0000 7f7016b16700 5 monclient: authenticate success, global_id 4107 2023-09-30T01:32:51.927 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:51.909+0000 7f7016b16700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:32:51.927 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:51.909+0000 7f7014b12700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:32:51.927 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:51.909+0000 7f7016b16700 10 monclient: get_monmap_and_config success 2023-09-30T01:32:51.927 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:51.909+0000 7f7016b16700 4 set_mon_vals no callback set 2023-09-30T01:32:51.927 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:51.909+0000 7f7014b12700 1 -- 172.21.15.107:0/1957206680 <== mon.3 v2:172.21.15.107:3301/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f700c0118a0 con 0x7f701009e2b0 2023-09-30T01:32:51.927 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:51.909+0000 7f7014b12700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:32:51.928 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:51.909+0000 7f7014b12700 10 monclient: got monmap 1 from mon.g (according to old e1) 2023-09-30T01:32:51.928 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:51.909+0000 7f7014b12700 10 monclient: dump: 2023-09-30T01:32:51.928 INFO:teuthology.orchestra.run.smithi107.stderr:epoch 1 2023-09-30T01:32:51.928 INFO:teuthology.orchestra.run.smithi107.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:32:51.928 INFO:teuthology.orchestra.run.smithi107.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:32:51.928 INFO:teuthology.orchestra.run.smithi107.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:32:51.928 INFO:teuthology.orchestra.run.smithi107.stderr:min_mon_release 17 (quincy) 2023-09-30T01:32:51.928 INFO:teuthology.orchestra.run.smithi107.stderr:election_strategy: 3 2023-09-30T01:32:51.928 INFO:teuthology.orchestra.run.smithi107.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:32:51.928 INFO:teuthology.orchestra.run.smithi107.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:32:51.929 INFO:teuthology.orchestra.run.smithi107.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:32:51.929 INFO:teuthology.orchestra.run.smithi107.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:32:51.929 INFO:teuthology.orchestra.run.smithi107.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:32:51.929 INFO:teuthology.orchestra.run.smithi107.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:32:51.929 INFO:teuthology.orchestra.run.smithi107.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:32:51.929 INFO:teuthology.orchestra.run.smithi107.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:32:51.929 INFO:teuthology.orchestra.run.smithi107.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:32:51.929 INFO:teuthology.orchestra.run.smithi107.stderr: 2023-09-30T01:32:51.929 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:51.909+0000 7f7016b16700 10 monclient: shutdown 2023-09-30T01:32:51.929 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:51.909+0000 7f7016b16700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:32:51.930 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:51.909+0000 7f7016b16700 1 -- 172.21.15.107:0/1957206680 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7f701009e2b0 msgr2=0x7f701009e6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:32:51.930 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:51.909+0000 7f7016b16700 1 --2- 172.21.15.107:0/1957206680 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7f701009e2b0 0x7f701009e6e0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f700c009a10 tx=0x7f700c009250 comp rx=0 tx=0).stop 2023-09-30T01:32:51.930 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:51.909+0000 7f7016b16700 1 -- 172.21.15.107:0/1957206680 shutdown_connections 2023-09-30T01:32:51.930 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:51.909+0000 7f7016b16700 1 --2- 172.21.15.107:0/1957206680 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7f701009e2b0 0x7f701009e6e0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:32:51.930 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:51.909+0000 7f7016b16700 1 --2- 172.21.15.107:0/1957206680 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7f7010099400 0x7f701009dd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:32:51.930 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:51.909+0000 7f7016b16700 1 --2- 172.21.15.107:0/1957206680 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f7010097760 0x7f7010098ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:32:51.930 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:51.909+0000 7f7016b16700 1 -- 172.21.15.107:0/1957206680 >> 172.21.15.107:0/1957206680 conn(0x7f701009fe70 msgr2=0x7f70100a0270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:32:51.930 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:51.909+0000 7f7016b16700 1 -- 172.21.15.107:0/1957206680 shutdown_connections 2023-09-30T01:32:51.930 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:51.909+0000 7f7016b16700 1 -- 172.21.15.107:0/1957206680 wait complete. 2023-09-30T01:32:51.930 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:51.909+0000 7f7016b16700 10 monclient: build_initial_monmap 2023-09-30T01:32:51.930 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:51.909+0000 7f7016b16700 10 monclient: monmap: 2023-09-30T01:32:51.930 INFO:teuthology.orchestra.run.smithi107.stderr:epoch 0 2023-09-30T01:32:51.931 INFO:teuthology.orchestra.run.smithi107.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:32:51.931 INFO:teuthology.orchestra.run.smithi107.stderr:last_changed 0.000000 2023-09-30T01:32:51.931 INFO:teuthology.orchestra.run.smithi107.stderr:created 0.000000 2023-09-30T01:32:51.931 INFO:teuthology.orchestra.run.smithi107.stderr:min_mon_release 0 (unknown) 2023-09-30T01:32:51.931 INFO:teuthology.orchestra.run.smithi107.stderr:election_strategy: 1 2023-09-30T01:32:51.931 INFO:teuthology.orchestra.run.smithi107.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.noname-a 2023-09-30T01:32:51.931 INFO:teuthology.orchestra.run.smithi107.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.noname-f 2023-09-30T01:32:51.931 INFO:teuthology.orchestra.run.smithi107.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.noname-b 2023-09-30T01:32:51.931 INFO:teuthology.orchestra.run.smithi107.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.noname-g 2023-09-30T01:32:51.931 INFO:teuthology.orchestra.run.smithi107.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.noname-c 2023-09-30T01:32:51.931 INFO:teuthology.orchestra.run.smithi107.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.noname-h 2023-09-30T01:32:51.931 INFO:teuthology.orchestra.run.smithi107.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.noname-d 2023-09-30T01:32:51.932 INFO:teuthology.orchestra.run.smithi107.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.noname-i 2023-09-30T01:32:51.932 INFO:teuthology.orchestra.run.smithi107.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.noname-e 2023-09-30T01:32:51.932 INFO:teuthology.orchestra.run.smithi107.stderr: 2023-09-30T01:32:51.932 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:51.909+0000 7f7016b16700 1 Processor -- start 2023-09-30T01:32:51.932 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:51.909+0000 7f7016b16700 1 -- start start 2023-09-30T01:32:51.932 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:51.909+0000 7f7016b16700 10 monclient: init 2023-09-30T01:32:51.932 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:51.909+0000 7f7016b16700 10 monclient: _reopen_session rank -1 2023-09-30T01:32:51.932 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:51.909+0000 7f7016b16700 10 monclient: _add_conns ranks=[1,0,5,2,7,4,3,8,6] 2023-09-30T01:32:51.932 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:51.909+0000 7f7016b16700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7010097760 0x7f70101401f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:32:51.932 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:51.909+0000 7f7016b16700 10 monclient(hunting): picked mon.noname-f con 0x7f7010097760 addr [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] 2023-09-30T01:32:51.932 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:51.909+0000 7f7016b16700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7010099400 0x7f7010136ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:32:51.932 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:51.909+0000 7f7016b16700 10 monclient(hunting): picked mon.noname-a con 0x7f7010099400 addr [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] 2023-09-30T01:32:51.932 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:51.909+0000 7f7016b16700 1 --2- >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7f701009e2b0 0x7f701006d020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:32:51.933 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:51.909+0000 7f7016b16700 10 monclient(hunting): picked mon.noname-h con 0x7f701009e2b0 addr [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] 2023-09-30T01:32:51.933 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:51.909+0000 7f7016b16700 10 monclient(hunting): start opening mon connection 2023-09-30T01:32:51.933 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:51.909+0000 7f7016b16700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70101340f0 con 0x7f7010099400 2023-09-30T01:32:51.933 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:51.909+0000 7f7016b16700 10 monclient(hunting): start opening mon connection 2023-09-30T01:32:51.933 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:51.909+0000 7f7016b16700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70101346b0 con 0x7f7010097760 2023-09-30T01:32:51.933 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:51.909+0000 7f7016b16700 10 monclient(hunting): start opening mon connection 2023-09-30T01:32:51.933 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:51.909+0000 7f7016b16700 1 -- --> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] -- mon_getmap magic: 0 v1 -- 0x7f7010134990 con 0x7f701009e2b0 2023-09-30T01:32:51.933 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:51.909+0000 7f7016b16700 10 monclient(hunting): _renew_subs 2023-09-30T01:32:51.933 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:51.909+0000 7f7016b16700 10 monclient(hunting): authenticate will time out at 855.335935s 2023-09-30T01:32:51.934 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:51.913+0000 7f7015b14700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7010097760 0x7f70101401f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:32:51.934 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:51.913+0000 7f7016315700 1 --2- >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7f701009e2b0 0x7f701006d020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:32:51.934 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:51.913+0000 7f7015b14700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7010097760 0x7f70101401f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:48808/0 (socket says 172.21.15.107:48808) 2023-09-30T01:32:51.934 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:51.913+0000 7f7015b14700 1 -- 172.21.15.107:0/3767618425 learned_addr learned my addr 172.21.15.107:0/3767618425 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-09-30T01:32:51.934 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:51.913+0000 7f7015b14700 10 monclient(hunting): get_auth_request con 0x7f7010097760 auth_method 0 2023-09-30T01:32:51.934 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:51.913+0000 7f7015b14700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:32:51.934 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:51.913+0000 7f7015b14700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:32:51.934 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:51.913+0000 7f7015b14700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:32:51.934 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:51.913+0000 7f7016315700 10 monclient(hunting): get_auth_request con 0x7f701009e2b0 auth_method 0 2023-09-30T01:32:51.934 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:51.913+0000 7f7016315700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:32:51.935 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:51.913+0000 7f7016315700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:32:51.935 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:51.913+0000 7f7016315700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:32:51.935 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:51.913+0000 7f7015b14700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:32:51.935 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:51.913+0000 7f7015b14700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:32:51.935 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:51.913+0000 7f7015b14700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:32:51.935 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:51.913+0000 7f7016315700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:32:51.935 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:51.913+0000 7f7016315700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:32:51.935 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:51.913+0000 7f7016315700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:32:51.935 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:51.913+0000 7f7015313700 1 --2- 172.21.15.107:0/3767618425 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7010099400 0x7f7010136ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:32:51.935 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:51.913+0000 7f7015313700 10 monclient(hunting): get_auth_request con 0x7f7010099400 auth_method 0 2023-09-30T01:32:51.935 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:51.913+0000 7f7015313700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:32:51.935 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:51.913+0000 7f7015313700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:32:51.936 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:51.913+0000 7f7015313700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:32:51.936 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:51.913+0000 7f7016315700 10 monclient(hunting): handle_auth_done global_id 4127 payload 824 2023-09-30T01:32:51.936 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:51.913+0000 7f7016315700 1 -- 172.21.15.107:0/3767618425 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7010097760 msgr2=0x7f70101401f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:32:51.936 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:51.913+0000 7f7016315700 1 --2- 172.21.15.107:0/3767618425 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7010097760 0x7f70101401f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:32:51.936 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:51.913+0000 7f7016315700 1 -- 172.21.15.107:0/3767618425 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7010099400 msgr2=0x7f7010136ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:32:51.936 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:51.913+0000 7f7016315700 1 --2- 172.21.15.107:0/3767618425 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7010099400 0x7f7010136ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:32:51.936 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:51.913+0000 7f7016315700 10 monclient: _finish_hunting 0 2023-09-30T01:32:51.936 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:51.913+0000 7f7016315700 1 monclient: found mon.noname-h 2023-09-30T01:32:51.936 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:51.913+0000 7f7016315700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:32:51.936 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:51.913+0000 7f7016315700 10 monclient: _send_mon_message to mon.noname-h at v2:172.21.15.107:3302/0 2023-09-30T01:32:51.936 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:51.913+0000 7f7016315700 1 -- 172.21.15.107:0/3767618425 --> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7000009420 con 0x7f701009e2b0 2023-09-30T01:32:51.936 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:51.913+0000 7f7015313700 1 --2- 172.21.15.107:0/3767618425 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7010099400 0x7f7010136ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-30T01:32:51.937 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:51.913+0000 7f7016315700 1 --2- 172.21.15.107:0/3767618425 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7f701009e2b0 0x7f701006d020 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f700c016000 tx=0x7f700c017000 comp rx=0 tx=0).ready entity=mon.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:32:51.937 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:51.913+0000 7f7005ffb700 1 -- 172.21.15.107:0/3767618425 <== mon.5 v2:172.21.15.107:3302/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f700c006a90 con 0x7f701009e2b0 2023-09-30T01:32:51.937 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:51.913+0000 7f7005ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:32:51.937 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:51.913+0000 7f7005ffb700 10 monclient: got monmap 1 from mon.noname-h (according to old e1) 2023-09-30T01:32:51.937 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:51.913+0000 7f7005ffb700 10 monclient: dump: 2023-09-30T01:32:51.937 INFO:teuthology.orchestra.run.smithi107.stderr:epoch 1 2023-09-30T01:32:51.937 INFO:teuthology.orchestra.run.smithi107.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:32:51.937 INFO:teuthology.orchestra.run.smithi107.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:32:51.937 INFO:teuthology.orchestra.run.smithi107.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:32:51.937 INFO:teuthology.orchestra.run.smithi107.stderr:min_mon_release 17 (quincy) 2023-09-30T01:32:51.937 INFO:teuthology.orchestra.run.smithi107.stderr:election_strategy: 3 2023-09-30T01:32:51.937 INFO:teuthology.orchestra.run.smithi107.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:32:51.937 INFO:teuthology.orchestra.run.smithi107.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:32:51.938 INFO:teuthology.orchestra.run.smithi107.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:32:51.938 INFO:teuthology.orchestra.run.smithi107.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:32:51.938 INFO:teuthology.orchestra.run.smithi107.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:32:51.938 INFO:teuthology.orchestra.run.smithi107.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:32:51.938 INFO:teuthology.orchestra.run.smithi107.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:32:51.938 INFO:teuthology.orchestra.run.smithi107.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:32:51.938 INFO:teuthology.orchestra.run.smithi107.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:32:51.938 INFO:teuthology.orchestra.run.smithi107.stderr: 2023-09-30T01:32:51.938 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:51.913+0000 7f7005ffb700 10 monclient: _finish_auth 0 2023-09-30T01:32:51.938 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:51.913+0000 7f7005ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:32:51.938 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:51.913+0000 7f7016b16700 5 monclient: authenticate success, global_id 4127 2023-09-30T01:32:51.938 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:51.913+0000 7f7005ffb700 1 -- 172.21.15.107:0/3767618425 <== mon.5 v2:172.21.15.107:3302/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f700c0118a0 con 0x7f701009e2b0 2023-09-30T01:32:51.938 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:51.913+0000 7f7005ffb700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:32:51.939 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:51.913+0000 7f7005ffb700 1 -- 172.21.15.107:0/3767618425 <== mon.5 v2:172.21.15.107:3302/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f700c011e90 con 0x7f701009e2b0 2023-09-30T01:32:51.939 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:51.913+0000 7f7005ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:32:51.939 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:51.913+0000 7f7005ffb700 10 monclient: got monmap 1 from mon.h (according to old e1) 2023-09-30T01:32:51.939 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:51.913+0000 7f70077fe700 4 set_mon_vals no callback set 2023-09-30T01:32:51.939 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:51.913+0000 7f7005ffb700 10 monclient: dump: 2023-09-30T01:32:51.939 INFO:teuthology.orchestra.run.smithi107.stderr:epoch 1 2023-09-30T01:32:51.939 INFO:teuthology.orchestra.run.smithi107.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:32:51.939 INFO:teuthology.orchestra.run.smithi107.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:32:51.939 INFO:teuthology.orchestra.run.smithi107.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:32:51.940 INFO:teuthology.orchestra.run.smithi107.stderr:min_mon_release 17 (quincy) 2023-09-30T01:32:51.940 INFO:teuthology.orchestra.run.smithi107.stderr:election_strategy: 3 2023-09-30T01:32:51.940 INFO:teuthology.orchestra.run.smithi107.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:32:51.940 INFO:teuthology.orchestra.run.smithi107.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:32:51.940 INFO:teuthology.orchestra.run.smithi107.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:32:51.940 INFO:teuthology.orchestra.run.smithi107.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:32:51.940 INFO:teuthology.orchestra.run.smithi107.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:32:51.940 INFO:teuthology.orchestra.run.smithi107.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:32:51.940 INFO:teuthology.orchestra.run.smithi107.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:32:51.940 INFO:teuthology.orchestra.run.smithi107.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:32:51.940 INFO:teuthology.orchestra.run.smithi107.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:32:51.941 INFO:teuthology.orchestra.run.smithi107.stderr: 2023-09-30T01:32:51.941 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:51.913+0000 7f7016b16700 10 monclient: _renew_subs 2023-09-30T01:32:51.941 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:51.913+0000 7f7016b16700 10 monclient: _send_mon_message to mon.h at v2:172.21.15.107:3302/0 2023-09-30T01:32:51.941 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:51.913+0000 7f7016b16700 1 -- 172.21.15.107:0/3767618425 --> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f700c0094f0 con 0x7f701009e2b0 2023-09-30T01:32:51.941 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:51.913+0000 7f7016b16700 10 monclient: _renew_subs 2023-09-30T01:32:51.941 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:51.913+0000 7f7016b16700 10 monclient: _send_mon_message to mon.h at v2:172.21.15.107:3302/0 2023-09-30T01:32:51.941 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:51.913+0000 7f7016b16700 1 -- 172.21.15.107:0/3767618425 --> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7010133200 con 0x7f701009e2b0 2023-09-30T01:32:51.941 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:51.913+0000 7f7005ffb700 1 -- 172.21.15.107:0/3767618425 <== mon.5 v2:172.21.15.107:3302/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7f700c011420 con 0x7f701009e2b0 2023-09-30T01:32:51.941 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:51.913+0000 7f7005ffb700 1 -- 172.21.15.107:0/3767618425 <== mon.5 v2:172.21.15.107:3302/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 721+0+0 (secure 0 0 0) 0x7f700c02e750 con 0x7f701009e2b0 2023-09-30T01:32:51.941 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:51.913+0000 7f7016b16700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:32:51.941 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:51.913+0000 7f7016b16700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:32:51.941 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:51.913+0000 7f7016b16700 10 monclient: _send_mon_message to mon.h at v2:172.21.15.107:3302/0 2023-09-30T01:32:51.942 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:51.913+0000 7f7016b16700 1 -- 172.21.15.107:0/3767618425 --> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7010071960 con 0x7f701009e2b0 2023-09-30T01:32:51.942 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:51.917+0000 7f7005ffb700 1 -- 172.21.15.107:0/3767618425 <== mon.5 v2:172.21.15.107:3302/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7f700c015020 con 0x7f701009e2b0 2023-09-30T01:32:51.942 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:51.917+0000 7f7005ffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:32:51.942 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:51.917+0000 7f7005ffb700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:32:51.998 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:51.993+0000 7f7016b16700 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "1243f825-b99c-4546-a77d-90b8637698b4", "id": 0}] 2023-09-30T01:32:51.998 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:51.993+0000 7f7016b16700 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "1243f825-b99c-4546-a77d-90b8637698b4", "id": 0}] 2023-09-30T01:32:51.998 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:51.993+0000 7f7016b16700 10 monclient: _send_mon_message to mon.h at v2:172.21.15.107:3302/0 2023-09-30T01:32:51.998 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:51.993+0000 7f7016b16700 1 -- 172.21.15.107:0/3767618425 --> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] -- mon_command({"prefix": "osd new", "uuid": "1243f825-b99c-4546-a77d-90b8637698b4", "id": 0} v 0) v1 -- 0x7f7010067b60 con 0x7f701009e2b0 2023-09-30T01:32:52.009 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.005+0000 7f7005ffb700 1 -- 172.21.15.107:0/3767618425 <== mon.5 v2:172.21.15.107:3302/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "1243f825-b99c-4546-a77d-90b8637698b4", "id": 0}]=0 v3) v1 ==== 112+0+2 (secure 0 0 0) 0x7f700c011ce0 con 0x7f701009e2b0 2023-09-30T01:32:52.010 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.005+0000 7f7005ffb700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "1243f825-b99c-4546-a77d-90b8637698b4", "id": 0}] 2023-09-30T01:32:52.010 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.005+0000 7f7005ffb700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:32:52.010 INFO:teuthology.orchestra.run.smithi107.stdout:0 2023-09-30T01:32:52.011 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.005+0000 7f7016b16700 10 monclient: shutdown 2023-09-30T01:32:52.011 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.005+0000 7f7016b16700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:32:52.011 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.005+0000 7f7016b16700 1 -- 172.21.15.107:0/3767618425 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7f701009e2b0 msgr2=0x7f701006d020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:32:52.011 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.005+0000 7f7016b16700 1 --2- 172.21.15.107:0/3767618425 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7f701009e2b0 0x7f701006d020 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f700c016000 tx=0x7f700c017000 comp rx=0 tx=0).stop 2023-09-30T01:32:52.011 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.005+0000 7f7016b16700 1 -- 172.21.15.107:0/3767618425 shutdown_connections 2023-09-30T01:32:52.011 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.005+0000 7f7016b16700 1 --2- 172.21.15.107:0/3767618425 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7f701009e2b0 0x7f701006d020 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:32:52.011 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.005+0000 7f7016b16700 1 --2- 172.21.15.107:0/3767618425 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7010099400 0x7f7010136ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:32:52.011 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.005+0000 7f7016b16700 1 --2- 172.21.15.107:0/3767618425 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7010097760 0x7f70101401f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:32:52.012 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.005+0000 7f7016b16700 1 -- 172.21.15.107:0/3767618425 >> 172.21.15.107:0/3767618425 conn(0x7f701009fe70 msgr2=0x7f7010131ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:32:52.012 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.005+0000 7f7016b16700 1 -- 172.21.15.107:0/3767618425 shutdown_connections 2023-09-30T01:32:52.012 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.005+0000 7f7016b16700 1 -- 172.21.15.107:0/3767618425 wait complete. 2023-09-30T01:32:52.025 DEBUG:teuthology.orchestra.run.smithi107:> sudo ceph --cluster ceph osd new e9831b75-399b-4d6d-bf97-9ef104adfdb5 1 2023-09-30T01:32:52.130 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.125+0000 7fcf76f2c700 10 monclient: get_monmap_and_config 2023-09-30T01:32:52.130 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.125+0000 7fcf76f2c700 10 monclient: build_initial_monmap 2023-09-30T01:32:52.130 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.125+0000 7fcf76f2c700 10 monclient: monmap: 2023-09-30T01:32:52.130 INFO:teuthology.orchestra.run.smithi107.stderr:epoch 0 2023-09-30T01:32:52.130 INFO:teuthology.orchestra.run.smithi107.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:32:52.131 INFO:teuthology.orchestra.run.smithi107.stderr:last_changed 2023-09-30T01:32:52.130292+0000 2023-09-30T01:32:52.131 INFO:teuthology.orchestra.run.smithi107.stderr:created 2023-09-30T01:32:52.130292+0000 2023-09-30T01:32:52.131 INFO:teuthology.orchestra.run.smithi107.stderr:min_mon_release 0 (unknown) 2023-09-30T01:32:52.131 INFO:teuthology.orchestra.run.smithi107.stderr:election_strategy: 3 2023-09-30T01:32:52.131 INFO:teuthology.orchestra.run.smithi107.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a-legacy 2023-09-30T01:32:52.131 INFO:teuthology.orchestra.run.smithi107.stderr:1: v1:172.21.15.107:6789/0 mon.noname-f-legacy 2023-09-30T01:32:52.131 INFO:teuthology.orchestra.run.smithi107.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.noname-b 2023-09-30T01:32:52.131 INFO:teuthology.orchestra.run.smithi107.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.noname-g 2023-09-30T01:32:52.131 INFO:teuthology.orchestra.run.smithi107.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.noname-c 2023-09-30T01:32:52.131 INFO:teuthology.orchestra.run.smithi107.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.noname-h 2023-09-30T01:32:52.131 INFO:teuthology.orchestra.run.smithi107.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.noname-d 2023-09-30T01:32:52.132 INFO:teuthology.orchestra.run.smithi107.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.noname-i 2023-09-30T01:32:52.132 INFO:teuthology.orchestra.run.smithi107.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.noname-e 2023-09-30T01:32:52.132 INFO:teuthology.orchestra.run.smithi107.stderr:9: v2:172.21.15.88:3300/0 mon.noname-a 2023-09-30T01:32:52.132 INFO:teuthology.orchestra.run.smithi107.stderr:10: v2:172.21.15.107:3300/0 mon.noname-f 2023-09-30T01:32:52.132 INFO:teuthology.orchestra.run.smithi107.stderr: 2023-09-30T01:32:52.141 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.125+0000 7fcf76f2c700 1 Processor -- start 2023-09-30T01:32:52.142 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.125+0000 7fcf76f2c700 1 -- start start 2023-09-30T01:32:52.142 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.125+0000 7fcf76f2c700 10 monclient: init 2023-09-30T01:32:52.142 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.125+0000 7fcf76f2c700 10 monclient: _reopen_session rank -1 2023-09-30T01:32:52.142 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.125+0000 7fcf76f2c700 10 monclient: _add_conns ranks=[9,5,8,10,2,3,0,6,4,7,1] 2023-09-30T01:32:52.142 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.125+0000 7fcf76f2c700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fcf70130560 0x7fcf700696c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:32:52.142 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.125+0000 7fcf76f2c700 10 monclient(hunting): picked mon.noname-a con 0x7fcf70130560 addr v2:172.21.15.88:3300/0 2023-09-30T01:32:52.142 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.125+0000 7fcf76f2c700 1 --2- >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7fcf70130930 0x7fcf7009edf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:32:52.142 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.125+0000 7fcf76f2c700 10 monclient(hunting): picked mon.noname-h con 0x7fcf70130930 addr [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] 2023-09-30T01:32:52.142 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.125+0000 7fcf76f2c700 1 --2- >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7fcf70130d60 0x7fcf7009f330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:32:52.142 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.125+0000 7fcf76f2c700 10 monclient(hunting): picked mon.noname-e con 0x7fcf70130d60 addr [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] 2023-09-30T01:32:52.142 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.125+0000 7fcf76f2c700 10 monclient(hunting): start opening mon connection 2023-09-30T01:32:52.143 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.125+0000 7fcf76f2c700 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcf70068bf0 con 0x7fcf70130560 2023-09-30T01:32:52.143 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.125+0000 7fcf76f2c700 10 monclient(hunting): start opening mon connection 2023-09-30T01:32:52.143 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.125+0000 7fcf76f2c700 1 -- --> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] -- mon_getmap magic: 0 v1 -- 0x7fcf70068e60 con 0x7fcf70130930 2023-09-30T01:32:52.143 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.125+0000 7fcf76f2c700 10 monclient(hunting): start opening mon connection 2023-09-30T01:32:52.143 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.125+0000 7fcf76f2c700 1 -- --> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] -- mon_getmap magic: 0 v1 -- 0x7fcf700a1dc0 con 0x7fcf70130d60 2023-09-30T01:32:52.143 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.125+0000 7fcf76f2c700 10 monclient(hunting): _renew_subs 2023-09-30T01:32:52.143 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.125+0000 7fcf76f2c700 10 monclient(hunting): authenticate will time out at 855.551995s 2023-09-30T01:32:52.143 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.129+0000 7fcf75729700 1 --2- >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7fcf70130930 0x7fcf7009edf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:32:52.143 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.129+0000 7fcf75729700 1 --2- >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7fcf70130930 0x7fcf7009edf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3302/0 says I am v2:172.21.15.107:36244/0 (socket says 172.21.15.107:36244) 2023-09-30T01:32:52.143 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.129+0000 7fcf75729700 1 -- 172.21.15.107:0/206907488 learned_addr learned my addr 172.21.15.107:0/206907488 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-09-30T01:32:52.143 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.129+0000 7fcf75729700 10 monclient(hunting): get_auth_request con 0x7fcf70130930 auth_method 0 2023-09-30T01:32:52.143 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.129+0000 7fcf75729700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:32:52.144 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.129+0000 7fcf75729700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:32:52.144 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.129+0000 7fcf75729700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:32:52.144 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.129+0000 7fcf75f2a700 1 --2- 172.21.15.107:0/206907488 >> v2:172.21.15.88:3300/0 conn(0x7fcf70130560 0x7fcf700696c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:32:52.144 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.129+0000 7fcf7672b700 1 --2- 172.21.15.107:0/206907488 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7fcf70130d60 0x7fcf7009f330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:32:52.144 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.129+0000 7fcf7672b700 10 monclient(hunting): get_auth_request con 0x7fcf70130d60 auth_method 0 2023-09-30T01:32:52.144 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.129+0000 7fcf7672b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:32:52.144 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.129+0000 7fcf7672b700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:32:52.144 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.129+0000 7fcf7672b700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:32:52.144 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.129+0000 7fcf75f2a700 10 monclient(hunting): get_auth_request con 0x7fcf70130560 auth_method 0 2023-09-30T01:32:52.144 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.129+0000 7fcf75f2a700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:32:52.144 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.129+0000 7fcf75f2a700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:32:52.144 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.129+0000 7fcf75f2a700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:32:52.145 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.129+0000 7fcf75729700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:32:52.145 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.129+0000 7fcf75729700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:32:52.145 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.129+0000 7fcf75729700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:32:52.145 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.129+0000 7fcf7672b700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:32:52.145 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.129+0000 7fcf7672b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:32:52.145 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.129+0000 7fcf7672b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:32:52.145 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.129+0000 7fcf75f2a700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:32:52.145 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.129+0000 7fcf75f2a700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:32:52.145 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.129+0000 7fcf75f2a700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:32:52.145 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.129+0000 7fcf75729700 10 monclient(hunting): handle_auth_done global_id 4136 payload 274 2023-09-30T01:32:52.145 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.129+0000 7fcf75729700 1 -- 172.21.15.107:0/206907488 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7fcf70130d60 msgr2=0x7fcf7009f330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:32:52.145 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.129+0000 7fcf75729700 1 --2- 172.21.15.107:0/206907488 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7fcf70130d60 0x7fcf7009f330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:32:52.145 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.129+0000 7fcf75729700 1 -- 172.21.15.107:0/206907488 >> v2:172.21.15.88:3300/0 conn(0x7fcf70130560 msgr2=0x7fcf700696c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:32:52.146 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.129+0000 7fcf75729700 1 --2- 172.21.15.107:0/206907488 >> v2:172.21.15.88:3300/0 conn(0x7fcf70130560 0x7fcf700696c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:32:52.146 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.129+0000 7fcf75729700 10 monclient: _finish_hunting 0 2023-09-30T01:32:52.146 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.129+0000 7fcf75729700 1 monclient: found mon.noname-h 2023-09-30T01:32:52.146 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.129+0000 7fcf75729700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:32:52.146 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.129+0000 7fcf75729700 10 monclient: _send_mon_message to mon.noname-h at v2:172.21.15.107:3302/0 2023-09-30T01:32:52.146 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.129+0000 7fcf75729700 1 -- 172.21.15.107:0/206907488 --> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf7004bb20 con 0x7fcf70130930 2023-09-30T01:32:52.146 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.129+0000 7fcf75729700 1 --2- 172.21.15.107:0/206907488 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7fcf70130930 0x7fcf7009edf0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fcf600099b0 tx=0x7fcf60009250 comp rx=0 tx=0).ready entity=mon.5 client_cookie=d91a2adba58266c7 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:32:52.146 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.129+0000 7fcf74f28700 1 -- 172.21.15.107:0/206907488 <== mon.5 v2:172.21.15.107:3302/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7fcf6000bb00 con 0x7fcf70130930 2023-09-30T01:32:52.146 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.129+0000 7fcf74f28700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:32:52.146 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.129+0000 7fcf74f28700 10 monclient: got monmap 1 from mon.noname-h (according to old e1) 2023-09-30T01:32:52.146 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.129+0000 7fcf74f28700 10 monclient: dump: 2023-09-30T01:32:52.147 INFO:teuthology.orchestra.run.smithi107.stderr:epoch 1 2023-09-30T01:32:52.147 INFO:teuthology.orchestra.run.smithi107.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:32:52.147 INFO:teuthology.orchestra.run.smithi107.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:32:52.147 INFO:teuthology.orchestra.run.smithi107.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:32:52.147 INFO:teuthology.orchestra.run.smithi107.stderr:min_mon_release 17 (quincy) 2023-09-30T01:32:52.147 INFO:teuthology.orchestra.run.smithi107.stderr:election_strategy: 3 2023-09-30T01:32:52.147 INFO:teuthology.orchestra.run.smithi107.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:32:52.147 INFO:teuthology.orchestra.run.smithi107.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:32:52.147 INFO:teuthology.orchestra.run.smithi107.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:32:52.147 INFO:teuthology.orchestra.run.smithi107.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:32:52.147 INFO:teuthology.orchestra.run.smithi107.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:32:52.147 INFO:teuthology.orchestra.run.smithi107.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:32:52.147 INFO:teuthology.orchestra.run.smithi107.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:32:52.148 INFO:teuthology.orchestra.run.smithi107.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:32:52.148 INFO:teuthology.orchestra.run.smithi107.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:32:52.148 INFO:teuthology.orchestra.run.smithi107.stderr: 2023-09-30T01:32:52.148 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.129+0000 7fcf74f28700 10 monclient: _finish_auth 0 2023-09-30T01:32:52.148 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.129+0000 7fcf74f28700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:32:52.148 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.129+0000 7fcf74f28700 1 -- 172.21.15.107:0/206907488 <== mon.5 v2:172.21.15.107:3302/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf6000bc80 con 0x7fcf70130930 2023-09-30T01:32:52.148 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.129+0000 7fcf76f2c700 5 monclient: authenticate success, global_id 4136 2023-09-30T01:32:52.148 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.129+0000 7fcf76f2c700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:32:52.148 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.129+0000 7fcf74f28700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:32:52.148 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.129+0000 7fcf76f2c700 10 monclient: get_monmap_and_config success 2023-09-30T01:32:52.148 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.129+0000 7fcf76f2c700 4 set_mon_vals no callback set 2023-09-30T01:32:52.148 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.129+0000 7fcf74f28700 1 -- 172.21.15.107:0/206907488 <== mon.5 v2:172.21.15.107:3302/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7fcf6000bdf0 con 0x7fcf70130930 2023-09-30T01:32:52.149 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.129+0000 7fcf74f28700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:32:52.149 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.129+0000 7fcf74f28700 10 monclient: got monmap 1 from mon.h (according to old e1) 2023-09-30T01:32:52.149 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.129+0000 7fcf74f28700 10 monclient: dump: 2023-09-30T01:32:52.149 INFO:teuthology.orchestra.run.smithi107.stderr:epoch 1 2023-09-30T01:32:52.149 INFO:teuthology.orchestra.run.smithi107.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:32:52.149 INFO:teuthology.orchestra.run.smithi107.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:32:52.149 INFO:teuthology.orchestra.run.smithi107.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:32:52.149 INFO:teuthology.orchestra.run.smithi107.stderr:min_mon_release 17 (quincy) 2023-09-30T01:32:52.149 INFO:teuthology.orchestra.run.smithi107.stderr:election_strategy: 3 2023-09-30T01:32:52.149 INFO:teuthology.orchestra.run.smithi107.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:32:52.149 INFO:teuthology.orchestra.run.smithi107.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:32:52.149 INFO:teuthology.orchestra.run.smithi107.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:32:52.149 INFO:teuthology.orchestra.run.smithi107.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:32:52.150 INFO:teuthology.orchestra.run.smithi107.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:32:52.150 INFO:teuthology.orchestra.run.smithi107.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:32:52.150 INFO:teuthology.orchestra.run.smithi107.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:32:52.150 INFO:teuthology.orchestra.run.smithi107.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:32:52.150 INFO:teuthology.orchestra.run.smithi107.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:32:52.150 INFO:teuthology.orchestra.run.smithi107.stderr: 2023-09-30T01:32:52.150 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.129+0000 7fcf76f2c700 10 monclient: shutdown 2023-09-30T01:32:52.150 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.129+0000 7fcf76f2c700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:32:52.150 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.129+0000 7fcf76f2c700 1 -- 172.21.15.107:0/206907488 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7fcf70130930 msgr2=0x7fcf7009edf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:32:52.150 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.129+0000 7fcf76f2c700 1 --2- 172.21.15.107:0/206907488 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7fcf70130930 0x7fcf7009edf0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fcf600099b0 tx=0x7fcf60009250 comp rx=0 tx=0).stop 2023-09-30T01:32:52.150 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.129+0000 7fcf76f2c700 1 -- 172.21.15.107:0/206907488 shutdown_connections 2023-09-30T01:32:52.150 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.129+0000 7fcf76f2c700 1 --2- 172.21.15.107:0/206907488 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7fcf70130d60 0x7fcf7009f330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:32:52.151 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.129+0000 7fcf76f2c700 1 --2- 172.21.15.107:0/206907488 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7fcf70130930 0x7fcf7009edf0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:32:52.151 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.129+0000 7fcf76f2c700 1 --2- 172.21.15.107:0/206907488 >> v2:172.21.15.88:3300/0 conn(0x7fcf70130560 0x7fcf700696c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:32:52.151 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.129+0000 7fcf76f2c700 1 -- 172.21.15.107:0/206907488 >> 172.21.15.107:0/206907488 conn(0x7fcf70097a40 msgr2=0x7fcf700a0ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:32:52.151 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.129+0000 7fcf76f2c700 1 -- 172.21.15.107:0/206907488 shutdown_connections 2023-09-30T01:32:52.151 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.129+0000 7fcf76f2c700 1 -- 172.21.15.107:0/206907488 wait complete. 2023-09-30T01:32:52.151 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.129+0000 7fcf76f2c700 10 monclient: build_initial_monmap 2023-09-30T01:32:52.151 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.129+0000 7fcf76f2c700 10 monclient: monmap: 2023-09-30T01:32:52.151 INFO:teuthology.orchestra.run.smithi107.stderr:epoch 0 2023-09-30T01:32:52.151 INFO:teuthology.orchestra.run.smithi107.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:32:52.151 INFO:teuthology.orchestra.run.smithi107.stderr:last_changed 0.000000 2023-09-30T01:32:52.151 INFO:teuthology.orchestra.run.smithi107.stderr:created 0.000000 2023-09-30T01:32:52.151 INFO:teuthology.orchestra.run.smithi107.stderr:min_mon_release 0 (unknown) 2023-09-30T01:32:52.152 INFO:teuthology.orchestra.run.smithi107.stderr:election_strategy: 1 2023-09-30T01:32:52.152 INFO:teuthology.orchestra.run.smithi107.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.noname-a 2023-09-30T01:32:52.152 INFO:teuthology.orchestra.run.smithi107.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.noname-f 2023-09-30T01:32:52.152 INFO:teuthology.orchestra.run.smithi107.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.noname-b 2023-09-30T01:32:52.152 INFO:teuthology.orchestra.run.smithi107.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.noname-g 2023-09-30T01:32:52.152 INFO:teuthology.orchestra.run.smithi107.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.noname-c 2023-09-30T01:32:52.152 INFO:teuthology.orchestra.run.smithi107.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.noname-h 2023-09-30T01:32:52.152 INFO:teuthology.orchestra.run.smithi107.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.noname-d 2023-09-30T01:32:52.152 INFO:teuthology.orchestra.run.smithi107.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.noname-i 2023-09-30T01:32:52.152 INFO:teuthology.orchestra.run.smithi107.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.noname-e 2023-09-30T01:32:52.152 INFO:teuthology.orchestra.run.smithi107.stderr: 2023-09-30T01:32:52.152 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.129+0000 7fcf76f2c700 1 Processor -- start 2023-09-30T01:32:52.152 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.129+0000 7fcf76f2c700 1 -- start start 2023-09-30T01:32:52.153 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.129+0000 7fcf76f2c700 10 monclient: init 2023-09-30T01:32:52.153 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.133+0000 7fcf76f2c700 10 monclient: _reopen_session rank -1 2023-09-30T01:32:52.153 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.133+0000 7fcf76f2c700 10 monclient: _add_conns ranks=[0,6,1,7,4,5,2,3,8] 2023-09-30T01:32:52.153 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.133+0000 7fcf76f2c700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcf70130560 0x7fcf701424e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:32:52.153 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.133+0000 7fcf76f2c700 10 monclient(hunting): picked mon.noname-a con 0x7fcf70130560 addr [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] 2023-09-30T01:32:52.153 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.133+0000 7fcf76f2c700 1 --2- >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7fcf70130930 0x7fcf7012d2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:32:52.153 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.133+0000 7fcf76f2c700 10 monclient(hunting): picked mon.noname-d con 0x7fcf70130930 addr [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] 2023-09-30T01:32:52.153 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.133+0000 7fcf76f2c700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcf70130d60 0x7fcf70099140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:32:52.153 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.133+0000 7fcf76f2c700 10 monclient(hunting): picked mon.noname-f con 0x7fcf70130d60 addr [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] 2023-09-30T01:32:52.153 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.133+0000 7fcf76f2c700 10 monclient(hunting): start opening mon connection 2023-09-30T01:32:52.153 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.133+0000 7fcf76f2c700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf70069e90 con 0x7fcf70130560 2023-09-30T01:32:52.154 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.133+0000 7fcf76f2c700 10 monclient(hunting): start opening mon connection 2023-09-30T01:32:52.154 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.133+0000 7fcf76f2c700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf70120710 con 0x7fcf70130d60 2023-09-30T01:32:52.154 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.133+0000 7fcf76f2c700 10 monclient(hunting): start opening mon connection 2023-09-30T01:32:52.154 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.133+0000 7fcf76f2c700 1 -- --> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] -- mon_getmap magic: 0 v1 -- 0x7fcf7011f540 con 0x7fcf70130930 2023-09-30T01:32:52.154 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.133+0000 7fcf76f2c700 10 monclient(hunting): _renew_subs 2023-09-30T01:32:52.154 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.133+0000 7fcf76f2c700 10 monclient(hunting): authenticate will time out at 855.556140s 2023-09-30T01:32:52.154 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.133+0000 7fcf7672b700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcf70130d60 0x7fcf70099140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:32:52.154 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.133+0000 7fcf7672b700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcf70130d60 0x7fcf70099140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:48816/0 (socket says 172.21.15.107:48816) 2023-09-30T01:32:52.154 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.133+0000 7fcf7672b700 1 -- 172.21.15.107:0/1339158630 learned_addr learned my addr 172.21.15.107:0/1339158630 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-09-30T01:32:52.154 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.133+0000 7fcf7672b700 10 monclient(hunting): get_auth_request con 0x7fcf70130d60 auth_method 0 2023-09-30T01:32:52.154 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.133+0000 7fcf7672b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:32:52.154 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.133+0000 7fcf7672b700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:32:52.155 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.133+0000 7fcf7672b700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:32:52.155 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.133+0000 7fcf75729700 1 --2- 172.21.15.107:0/1339158630 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7fcf70130930 0x7fcf7012d2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:32:52.155 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.133+0000 7fcf75f2a700 1 --2- 172.21.15.107:0/1339158630 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcf70130560 0x7fcf701424e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:32:52.155 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.133+0000 7fcf7672b700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:32:52.155 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.133+0000 7fcf7672b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:32:52.155 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.133+0000 7fcf7672b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:32:52.155 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.133+0000 7fcf75729700 10 monclient(hunting): get_auth_request con 0x7fcf70130930 auth_method 0 2023-09-30T01:32:52.155 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.133+0000 7fcf75729700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:32:52.155 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.133+0000 7fcf75729700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:32:52.155 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.133+0000 7fcf75729700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:32:52.155 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.133+0000 7fcf75f2a700 10 monclient(hunting): get_auth_request con 0x7fcf70130560 auth_method 0 2023-09-30T01:32:52.155 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.133+0000 7fcf75f2a700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:32:52.156 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.133+0000 7fcf75f2a700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:32:52.156 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.133+0000 7fcf75f2a700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:32:52.156 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.133+0000 7fcf75f2a700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:32:52.156 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.133+0000 7fcf75f2a700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:32:52.156 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.133+0000 7fcf75f2a700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:32:52.156 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.133+0000 7fcf75729700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:32:52.156 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.133+0000 7fcf75729700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:32:52.156 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.133+0000 7fcf75729700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:32:52.156 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.133+0000 7fcf7672b700 10 monclient(hunting): handle_auth_done global_id 4132 payload 824 2023-09-30T01:32:52.156 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.133+0000 7fcf7672b700 1 -- 172.21.15.107:0/1339158630 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7fcf70130930 msgr2=0x7fcf7012d2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:32:52.156 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.133+0000 7fcf7672b700 1 --2- 172.21.15.107:0/1339158630 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7fcf70130930 0x7fcf7012d2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:32:52.156 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.133+0000 7fcf7672b700 1 -- 172.21.15.107:0/1339158630 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcf70130560 msgr2=0x7fcf701424e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:32:52.156 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.133+0000 7fcf7672b700 1 --2- 172.21.15.107:0/1339158630 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcf70130560 0x7fcf701424e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:32:52.157 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.133+0000 7fcf7672b700 10 monclient: _finish_hunting 0 2023-09-30T01:32:52.157 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.133+0000 7fcf7672b700 1 monclient: found mon.noname-f 2023-09-30T01:32:52.157 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.133+0000 7fcf7672b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:32:52.157 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.133+0000 7fcf7672b700 10 monclient: _send_mon_message to mon.noname-f at v2:172.21.15.107:3300/0 2023-09-30T01:32:52.157 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.133+0000 7fcf7672b700 1 -- 172.21.15.107:0/1339158630 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf600094f0 con 0x7fcf70130d60 2023-09-30T01:32:52.157 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.133+0000 7fcf7672b700 1 --2- 172.21.15.107:0/1339158630 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcf70130d60 0x7fcf70099140 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fcf68009e20 tx=0x7fcf6800b040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:32:52.157 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.133+0000 7fcf65ffb700 1 -- 172.21.15.107:0/1339158630 <== mon.1 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7fcf68014480 con 0x7fcf70130d60 2023-09-30T01:32:52.157 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.133+0000 7fcf65ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:32:52.157 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.133+0000 7fcf65ffb700 10 monclient: got monmap 1 from mon.noname-f (according to old e1) 2023-09-30T01:32:52.157 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.133+0000 7fcf65ffb700 10 monclient: dump: 2023-09-30T01:32:52.157 INFO:teuthology.orchestra.run.smithi107.stderr:epoch 1 2023-09-30T01:32:52.157 INFO:teuthology.orchestra.run.smithi107.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:32:52.158 INFO:teuthology.orchestra.run.smithi107.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:32:52.158 INFO:teuthology.orchestra.run.smithi107.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:32:52.158 INFO:teuthology.orchestra.run.smithi107.stderr:min_mon_release 17 (quincy) 2023-09-30T01:32:52.158 INFO:teuthology.orchestra.run.smithi107.stderr:election_strategy: 3 2023-09-30T01:32:52.158 INFO:teuthology.orchestra.run.smithi107.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:32:52.158 INFO:teuthology.orchestra.run.smithi107.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:32:52.158 INFO:teuthology.orchestra.run.smithi107.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:32:52.158 INFO:teuthology.orchestra.run.smithi107.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:32:52.158 INFO:teuthology.orchestra.run.smithi107.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:32:52.158 INFO:teuthology.orchestra.run.smithi107.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:32:52.158 INFO:teuthology.orchestra.run.smithi107.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:32:52.158 INFO:teuthology.orchestra.run.smithi107.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:32:52.158 INFO:teuthology.orchestra.run.smithi107.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:32:52.159 INFO:teuthology.orchestra.run.smithi107.stderr: 2023-09-30T01:32:52.159 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.133+0000 7fcf65ffb700 10 monclient: _finish_auth 0 2023-09-30T01:32:52.159 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.133+0000 7fcf65ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:32:52.159 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.133+0000 7fcf76f2c700 5 monclient: authenticate success, global_id 4132 2023-09-30T01:32:52.159 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.133+0000 7fcf65ffb700 1 -- 172.21.15.107:0/1339158630 <== mon.1 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf6800f040 con 0x7fcf70130d60 2023-09-30T01:32:52.159 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.133+0000 7fcf76f2c700 10 monclient: _renew_subs 2023-09-30T01:32:52.159 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.133+0000 7fcf76f2c700 10 monclient: _send_mon_message to mon.f at v2:172.21.15.107:3300/0 2023-09-30T01:32:52.159 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.133+0000 7fcf76f2c700 1 -- 172.21.15.107:0/1339158630 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf70120430 con 0x7fcf70130d60 2023-09-30T01:32:52.159 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.133+0000 7fcf65ffb700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:32:52.159 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.133+0000 7fcf65ffb700 1 -- 172.21.15.107:0/1339158630 <== mon.1 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7fcf68014aa0 con 0x7fcf70130d60 2023-09-30T01:32:52.159 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.133+0000 7fcf65ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:32:52.159 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.133+0000 7fcf65ffb700 10 monclient: got monmap 1 from mon.f (according to old e1) 2023-09-30T01:32:52.160 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.133+0000 7fcf67fff700 4 set_mon_vals no callback set 2023-09-30T01:32:52.160 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.133+0000 7fcf65ffb700 10 monclient: dump: 2023-09-30T01:32:52.160 INFO:teuthology.orchestra.run.smithi107.stderr:epoch 1 2023-09-30T01:32:52.160 INFO:teuthology.orchestra.run.smithi107.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:32:52.160 INFO:teuthology.orchestra.run.smithi107.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:32:52.160 INFO:teuthology.orchestra.run.smithi107.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:32:52.160 INFO:teuthology.orchestra.run.smithi107.stderr:min_mon_release 17 (quincy) 2023-09-30T01:32:52.160 INFO:teuthology.orchestra.run.smithi107.stderr:election_strategy: 3 2023-09-30T01:32:52.160 INFO:teuthology.orchestra.run.smithi107.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:32:52.160 INFO:teuthology.orchestra.run.smithi107.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:32:52.160 INFO:teuthology.orchestra.run.smithi107.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:32:52.160 INFO:teuthology.orchestra.run.smithi107.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:32:52.160 INFO:teuthology.orchestra.run.smithi107.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:32:52.161 INFO:teuthology.orchestra.run.smithi107.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:32:52.161 INFO:teuthology.orchestra.run.smithi107.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:32:52.161 INFO:teuthology.orchestra.run.smithi107.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:32:52.161 INFO:teuthology.orchestra.run.smithi107.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:32:52.161 INFO:teuthology.orchestra.run.smithi107.stderr: 2023-09-30T01:32:52.161 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.133+0000 7fcf76f2c700 10 monclient: _renew_subs 2023-09-30T01:32:52.161 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.133+0000 7fcf76f2c700 10 monclient: _send_mon_message to mon.f at v2:172.21.15.107:3300/0 2023-09-30T01:32:52.161 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.133+0000 7fcf76f2c700 1 -- 172.21.15.107:0/1339158630 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf7009abf0 con 0x7fcf70130d60 2023-09-30T01:32:52.161 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.133+0000 7fcf65ffb700 1 -- 172.21.15.107:0/1339158630 <== mon.1 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7fcf68014c00 con 0x7fcf70130d60 2023-09-30T01:32:52.161 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.133+0000 7fcf65ffb700 1 -- 172.21.15.107:0/1339158630 <== mon.1 v2:172.21.15.107:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 836+0+0 (secure 0 0 0) 0x7fcf6801d400 con 0x7fcf70130d60 2023-09-30T01:32:52.161 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.133+0000 7fcf76f2c700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:32:52.161 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.133+0000 7fcf76f2c700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:32:52.162 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.133+0000 7fcf76f2c700 10 monclient: _send_mon_message to mon.f at v2:172.21.15.107:3300/0 2023-09-30T01:32:52.162 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.133+0000 7fcf76f2c700 1 -- 172.21.15.107:0/1339158630 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcf7006b030 con 0x7fcf70130d60 2023-09-30T01:32:52.162 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.137+0000 7fcf65ffb700 1 -- 172.21.15.107:0/1339158630 <== mon.1 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7fcf6803b6d0 con 0x7fcf70130d60 2023-09-30T01:32:52.162 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.137+0000 7fcf65ffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:32:52.162 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.137+0000 7fcf65ffb700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:32:52.216 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.213+0000 7fcf76f2c700 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "e9831b75-399b-4d6d-bf97-9ef104adfdb5", "id": 1}] 2023-09-30T01:32:52.216 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.213+0000 7fcf76f2c700 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "e9831b75-399b-4d6d-bf97-9ef104adfdb5", "id": 1}] 2023-09-30T01:32:52.216 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.213+0000 7fcf76f2c700 10 monclient: _send_mon_message to mon.f at v2:172.21.15.107:3300/0 2023-09-30T01:32:52.216 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.213+0000 7fcf76f2c700 1 -- 172.21.15.107:0/1339158630 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd new", "uuid": "e9831b75-399b-4d6d-bf97-9ef104adfdb5", "id": 1} v 0) v1 -- 0x7fcf70120880 con 0x7fcf70130d60 2023-09-30T01:32:52.226 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.221+0000 7fcf65ffb700 1 -- 172.21.15.107:0/1339158630 <== mon.1 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "e9831b75-399b-4d6d-bf97-9ef104adfdb5", "id": 1}]=0 v4) v1 ==== 112+0+2 (secure 0 0 0) 0x7fcf6802a070 con 0x7fcf70130d60 2023-09-30T01:32:52.226 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.221+0000 7fcf65ffb700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "e9831b75-399b-4d6d-bf97-9ef104adfdb5", "id": 1}] 2023-09-30T01:32:52.226 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.221+0000 7fcf65ffb700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:32:52.226 INFO:teuthology.orchestra.run.smithi107.stdout:1 2023-09-30T01:32:52.227 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.221+0000 7fcf76f2c700 10 monclient: shutdown 2023-09-30T01:32:52.227 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.221+0000 7fcf76f2c700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:32:52.227 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.221+0000 7fcf76f2c700 1 -- 172.21.15.107:0/1339158630 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcf70130d60 msgr2=0x7fcf70099140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:32:52.227 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.221+0000 7fcf76f2c700 1 --2- 172.21.15.107:0/1339158630 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcf70130d60 0x7fcf70099140 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fcf68009e20 tx=0x7fcf6800b040 comp rx=0 tx=0).stop 2023-09-30T01:32:52.228 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.221+0000 7fcf76f2c700 1 -- 172.21.15.107:0/1339158630 shutdown_connections 2023-09-30T01:32:52.228 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.221+0000 7fcf76f2c700 1 --2- 172.21.15.107:0/1339158630 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcf70130d60 0x7fcf70099140 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:32:52.228 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.221+0000 7fcf76f2c700 1 --2- 172.21.15.107:0/1339158630 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7fcf70130930 0x7fcf7012d2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:32:52.228 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.221+0000 7fcf76f2c700 1 --2- 172.21.15.107:0/1339158630 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcf70130560 0x7fcf701424e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:32:52.228 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.221+0000 7fcf76f2c700 1 -- 172.21.15.107:0/1339158630 >> 172.21.15.107:0/1339158630 conn(0x7fcf70097a40 msgr2=0x7fcf7009d7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:32:52.228 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.221+0000 7fcf76f2c700 1 -- 172.21.15.107:0/1339158630 shutdown_connections 2023-09-30T01:32:52.228 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.221+0000 7fcf76f2c700 1 -- 172.21.15.107:0/1339158630 wait complete. 2023-09-30T01:32:52.238 DEBUG:teuthology.orchestra.run.smithi107:> sudo ceph --cluster ceph osd new fbc7be74-cd43-4c72-89c1-469c17055b85 2 2023-09-30T01:32:52.339 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.333+0000 7f498ffff700 10 monclient: get_monmap_and_config 2023-09-30T01:32:52.340 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.333+0000 7f498ffff700 10 monclient: build_initial_monmap 2023-09-30T01:32:52.340 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.333+0000 7f498ffff700 10 monclient: monmap: 2023-09-30T01:32:52.340 INFO:teuthology.orchestra.run.smithi107.stderr:epoch 0 2023-09-30T01:32:52.340 INFO:teuthology.orchestra.run.smithi107.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:32:52.340 INFO:teuthology.orchestra.run.smithi107.stderr:last_changed 2023-09-30T01:32:52.339656+0000 2023-09-30T01:32:52.340 INFO:teuthology.orchestra.run.smithi107.stderr:created 2023-09-30T01:32:52.339656+0000 2023-09-30T01:32:52.340 INFO:teuthology.orchestra.run.smithi107.stderr:min_mon_release 0 (unknown) 2023-09-30T01:32:52.340 INFO:teuthology.orchestra.run.smithi107.stderr:election_strategy: 3 2023-09-30T01:32:52.340 INFO:teuthology.orchestra.run.smithi107.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a-legacy 2023-09-30T01:32:52.341 INFO:teuthology.orchestra.run.smithi107.stderr:1: v1:172.21.15.107:6789/0 mon.noname-f-legacy 2023-09-30T01:32:52.341 INFO:teuthology.orchestra.run.smithi107.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.noname-b 2023-09-30T01:32:52.341 INFO:teuthology.orchestra.run.smithi107.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.noname-g 2023-09-30T01:32:52.341 INFO:teuthology.orchestra.run.smithi107.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.noname-c 2023-09-30T01:32:52.341 INFO:teuthology.orchestra.run.smithi107.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.noname-h 2023-09-30T01:32:52.341 INFO:teuthology.orchestra.run.smithi107.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.noname-d 2023-09-30T01:32:52.341 INFO:teuthology.orchestra.run.smithi107.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.noname-i 2023-09-30T01:32:52.341 INFO:teuthology.orchestra.run.smithi107.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.noname-e 2023-09-30T01:32:52.341 INFO:teuthology.orchestra.run.smithi107.stderr:9: v2:172.21.15.88:3300/0 mon.noname-a 2023-09-30T01:32:52.341 INFO:teuthology.orchestra.run.smithi107.stderr:10: v2:172.21.15.107:3300/0 mon.noname-f 2023-09-30T01:32:52.341 INFO:teuthology.orchestra.run.smithi107.stderr: 2023-09-30T01:32:52.351 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.337+0000 7f498ffff700 1 Processor -- start 2023-09-30T01:32:52.351 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.337+0000 7f498ffff700 1 -- start start 2023-09-30T01:32:52.351 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.337+0000 7f498ffff700 10 monclient: init 2023-09-30T01:32:52.352 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.337+0000 7f498ffff700 10 monclient: _reopen_session rank -1 2023-09-30T01:32:52.352 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.337+0000 7f498ffff700 10 monclient: _add_conns ranks=[5,3,4,8,1,9,2,0,6,7,10] 2023-09-30T01:32:52.352 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.337+0000 7f498ffff700 1 --2- >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7f4990097a40 0x7f4990132700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:32:52.352 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.337+0000 7f498ffff700 10 monclient(hunting): picked mon.noname-h con 0x7f4990097a40 addr [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] 2023-09-30T01:32:52.352 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.337+0000 7f498ffff700 1 --2- >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7f49901311c0 0x7f4990132c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:32:52.352 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.337+0000 7f498ffff700 10 monclient(hunting): picked mon.noname-g con 0x7f49901311c0 addr [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] 2023-09-30T01:32:52.352 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.337+0000 7f498ffff700 1 --2- >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f49901332e0 0x7f4990133710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:32:52.352 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.337+0000 7f498ffff700 10 monclient(hunting): picked mon.noname-c con 0x7f49901332e0 addr [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] 2023-09-30T01:32:52.352 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.337+0000 7f498ffff700 10 monclient(hunting): start opening mon connection 2023-09-30T01:32:52.352 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.337+0000 7f498ffff700 1 -- --> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4990131590 con 0x7f49901311c0 2023-09-30T01:32:52.353 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.337+0000 7f498ffff700 10 monclient(hunting): start opening mon connection 2023-09-30T01:32:52.353 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.337+0000 7f498ffff700 1 -- --> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] -- mon_getmap magic: 0 v1 -- 0x7f499009a5d0 con 0x7f49901332e0 2023-09-30T01:32:52.353 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.337+0000 7f498ffff700 10 monclient(hunting): start opening mon connection 2023-09-30T01:32:52.353 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.337+0000 7f498ffff700 1 -- --> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] -- mon_getmap magic: 0 v1 -- 0x7f4990097e10 con 0x7f4990097a40 2023-09-30T01:32:52.353 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.337+0000 7f498ffff700 10 monclient(hunting): _renew_subs 2023-09-30T01:32:52.353 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.337+0000 7f498ffff700 10 monclient(hunting): authenticate will time out at 855.761760s 2023-09-30T01:32:52.353 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.337+0000 7f498effd700 1 --2- >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7f4990097a40 0x7f4990132700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:32:52.353 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.337+0000 7f498e7fc700 1 --2- >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7f49901311c0 0x7f4990132c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:32:52.353 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.337+0000 7f498effd700 1 --2- >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7f4990097a40 0x7f4990132700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3302/0 says I am v2:172.21.15.107:36248/0 (socket says 172.21.15.107:36248) 2023-09-30T01:32:52.353 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.337+0000 7f498effd700 1 -- 172.21.15.107:0/200783321 learned_addr learned my addr 172.21.15.107:0/200783321 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-09-30T01:32:52.354 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.337+0000 7f498effd700 10 monclient(hunting): get_auth_request con 0x7f4990097a40 auth_method 0 2023-09-30T01:32:52.354 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.337+0000 7f498effd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:32:52.354 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.337+0000 7f498effd700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:32:52.354 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.337+0000 7f498effd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:32:52.354 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.337+0000 7f498e7fc700 10 monclient(hunting): get_auth_request con 0x7f49901311c0 auth_method 0 2023-09-30T01:32:52.354 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.337+0000 7f498e7fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:32:52.354 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.337+0000 7f498e7fc700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:32:52.354 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.337+0000 7f498e7fc700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:32:52.354 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.337+0000 7f498effd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:32:52.354 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.337+0000 7f498effd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:32:52.354 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.337+0000 7f498effd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:32:52.355 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.337+0000 7f498e7fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:32:52.355 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.337+0000 7f498e7fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:32:52.355 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.337+0000 7f498e7fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:32:52.355 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.337+0000 7f498f7fe700 1 --2- 172.21.15.107:0/200783321 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f49901332e0 0x7f4990133710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:32:52.355 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.337+0000 7f498f7fe700 10 monclient(hunting): get_auth_request con 0x7f49901332e0 auth_method 0 2023-09-30T01:32:52.355 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.337+0000 7f498f7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:32:52.355 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.337+0000 7f498f7fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:32:52.355 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.337+0000 7f498f7fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:32:52.355 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.337+0000 7f498effd700 10 monclient(hunting): handle_auth_done global_id 4145 payload 274 2023-09-30T01:32:52.355 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.337+0000 7f498effd700 1 -- 172.21.15.107:0/200783321 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f49901332e0 msgr2=0x7f4990133710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:32:52.355 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.337+0000 7f498effd700 1 --2- 172.21.15.107:0/200783321 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f49901332e0 0x7f4990133710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:32:52.356 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.337+0000 7f498effd700 1 -- 172.21.15.107:0/200783321 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7f49901311c0 msgr2=0x7f4990132c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:32:52.356 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.337+0000 7f498effd700 1 --2- 172.21.15.107:0/200783321 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7f49901311c0 0x7f4990132c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:32:52.356 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.337+0000 7f498effd700 10 monclient: _finish_hunting 0 2023-09-30T01:32:52.356 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.337+0000 7f498effd700 1 monclient: found mon.noname-h 2023-09-30T01:32:52.357 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.337+0000 7f498effd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:32:52.357 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.337+0000 7f498effd700 10 monclient: _send_mon_message to mon.noname-h at v2:172.21.15.107:3302/0 2023-09-30T01:32:52.357 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.337+0000 7f498effd700 1 -- 172.21.15.107:0/200783321 --> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f499004bb20 con 0x7f4990097a40 2023-09-30T01:32:52.358 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.337+0000 7f498e7fc700 1 --2- 172.21.15.107:0/200783321 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7f49901311c0 0x7f4990132c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:32:52.358 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.337+0000 7f498effd700 1 --2- 172.21.15.107:0/200783321 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7f4990097a40 0x7f4990132700 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f49800099e0 tx=0x7f4980009250 comp rx=0 tx=0).ready entity=mon.5 client_cookie=4bd99fd2c9b23ebf server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:32:52.358 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.337+0000 7f498dffb700 1 -- 172.21.15.107:0/200783321 <== mon.5 v2:172.21.15.107:3302/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f498000bce0 con 0x7f4990097a40 2023-09-30T01:32:52.358 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.337+0000 7f498dffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:32:52.358 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.337+0000 7f498dffb700 10 monclient: got monmap 1 from mon.noname-h (according to old e1) 2023-09-30T01:32:52.358 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.337+0000 7f498dffb700 10 monclient: dump: 2023-09-30T01:32:52.358 INFO:teuthology.orchestra.run.smithi107.stderr:epoch 1 2023-09-30T01:32:52.358 INFO:teuthology.orchestra.run.smithi107.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:32:52.358 INFO:teuthology.orchestra.run.smithi107.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:32:52.358 INFO:teuthology.orchestra.run.smithi107.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:32:52.359 INFO:teuthology.orchestra.run.smithi107.stderr:min_mon_release 17 (quincy) 2023-09-30T01:32:52.359 INFO:teuthology.orchestra.run.smithi107.stderr:election_strategy: 3 2023-09-30T01:32:52.359 INFO:teuthology.orchestra.run.smithi107.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:32:52.359 INFO:teuthology.orchestra.run.smithi107.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:32:52.359 INFO:teuthology.orchestra.run.smithi107.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:32:52.359 INFO:teuthology.orchestra.run.smithi107.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:32:52.359 INFO:teuthology.orchestra.run.smithi107.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:32:52.359 INFO:teuthology.orchestra.run.smithi107.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:32:52.359 INFO:teuthology.orchestra.run.smithi107.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:32:52.359 INFO:teuthology.orchestra.run.smithi107.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:32:52.360 INFO:teuthology.orchestra.run.smithi107.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:32:52.360 INFO:teuthology.orchestra.run.smithi107.stderr: 2023-09-30T01:32:52.360 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.337+0000 7f498dffb700 10 monclient: _finish_auth 0 2023-09-30T01:32:52.360 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.337+0000 7f498dffb700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:32:52.360 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.337+0000 7f498dffb700 1 -- 172.21.15.107:0/200783321 <== mon.5 v2:172.21.15.107:3302/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f498000be60 con 0x7f4990097a40 2023-09-30T01:32:52.360 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.337+0000 7f498ffff700 5 monclient: authenticate success, global_id 4145 2023-09-30T01:32:52.360 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.337+0000 7f498ffff700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:32:52.360 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.337+0000 7f498dffb700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:32:52.360 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.337+0000 7f498ffff700 10 monclient: get_monmap_and_config success 2023-09-30T01:32:52.360 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.337+0000 7f498dffb700 1 -- 172.21.15.107:0/200783321 <== mon.5 v2:172.21.15.107:3302/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f49800118a0 con 0x7f4990097a40 2023-09-30T01:32:52.360 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.337+0000 7f498ffff700 4 set_mon_vals no callback set 2023-09-30T01:32:52.361 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.337+0000 7f498dffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:32:52.361 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.337+0000 7f498dffb700 10 monclient: got monmap 1 from mon.h (according to old e1) 2023-09-30T01:32:52.361 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.337+0000 7f498dffb700 10 monclient: dump: 2023-09-30T01:32:52.361 INFO:teuthology.orchestra.run.smithi107.stderr:epoch 1 2023-09-30T01:32:52.361 INFO:teuthology.orchestra.run.smithi107.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:32:52.361 INFO:teuthology.orchestra.run.smithi107.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:32:52.361 INFO:teuthology.orchestra.run.smithi107.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:32:52.361 INFO:teuthology.orchestra.run.smithi107.stderr:min_mon_release 17 (quincy) 2023-09-30T01:32:52.361 INFO:teuthology.orchestra.run.smithi107.stderr:election_strategy: 3 2023-09-30T01:32:52.362 INFO:teuthology.orchestra.run.smithi107.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:32:52.362 INFO:teuthology.orchestra.run.smithi107.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:32:52.362 INFO:teuthology.orchestra.run.smithi107.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:32:52.362 INFO:teuthology.orchestra.run.smithi107.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:32:52.362 INFO:teuthology.orchestra.run.smithi107.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:32:52.362 INFO:teuthology.orchestra.run.smithi107.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:32:52.362 INFO:teuthology.orchestra.run.smithi107.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:32:52.362 INFO:teuthology.orchestra.run.smithi107.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:32:52.362 INFO:teuthology.orchestra.run.smithi107.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:32:52.362 INFO:teuthology.orchestra.run.smithi107.stderr: 2023-09-30T01:32:52.362 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.337+0000 7f498ffff700 10 monclient: shutdown 2023-09-30T01:32:52.363 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.337+0000 7f498ffff700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:32:52.363 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.337+0000 7f498ffff700 1 -- 172.21.15.107:0/200783321 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7f4990097a40 msgr2=0x7f4990132700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:32:52.363 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.337+0000 7f498ffff700 1 --2- 172.21.15.107:0/200783321 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7f4990097a40 0x7f4990132700 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f49800099e0 tx=0x7f4980009250 comp rx=0 tx=0).stop 2023-09-30T01:32:52.363 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.341+0000 7f498ffff700 1 -- 172.21.15.107:0/200783321 shutdown_connections 2023-09-30T01:32:52.363 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.341+0000 7f498ffff700 1 --2- 172.21.15.107:0/200783321 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f49901332e0 0x7f4990133710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:32:52.363 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.341+0000 7f498ffff700 1 --2- 172.21.15.107:0/200783321 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7f49901311c0 0x7f4990132c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:32:52.363 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.341+0000 7f498ffff700 1 --2- 172.21.15.107:0/200783321 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7f4990097a40 0x7f4990132700 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:32:52.363 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.341+0000 7f498ffff700 1 -- 172.21.15.107:0/200783321 >> 172.21.15.107:0/200783321 conn(0x7f499006e5e0 msgr2=0x7f499006e9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:32:52.363 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.341+0000 7f498ffff700 1 -- 172.21.15.107:0/200783321 shutdown_connections 2023-09-30T01:32:52.363 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.341+0000 7f498ffff700 1 -- 172.21.15.107:0/200783321 wait complete. 2023-09-30T01:32:52.364 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.341+0000 7f498ffff700 10 monclient: build_initial_monmap 2023-09-30T01:32:52.364 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.341+0000 7f498ffff700 10 monclient: monmap: 2023-09-30T01:32:52.364 INFO:teuthology.orchestra.run.smithi107.stderr:epoch 0 2023-09-30T01:32:52.364 INFO:teuthology.orchestra.run.smithi107.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:32:52.364 INFO:teuthology.orchestra.run.smithi107.stderr:last_changed 0.000000 2023-09-30T01:32:52.364 INFO:teuthology.orchestra.run.smithi107.stderr:created 0.000000 2023-09-30T01:32:52.364 INFO:teuthology.orchestra.run.smithi107.stderr:min_mon_release 0 (unknown) 2023-09-30T01:32:52.364 INFO:teuthology.orchestra.run.smithi107.stderr:election_strategy: 1 2023-09-30T01:32:52.364 INFO:teuthology.orchestra.run.smithi107.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.noname-a 2023-09-30T01:32:52.364 INFO:teuthology.orchestra.run.smithi107.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.noname-f 2023-09-30T01:32:52.364 INFO:teuthology.orchestra.run.smithi107.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.noname-b 2023-09-30T01:32:52.365 INFO:teuthology.orchestra.run.smithi107.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.noname-g 2023-09-30T01:32:52.365 INFO:teuthology.orchestra.run.smithi107.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.noname-c 2023-09-30T01:32:52.365 INFO:teuthology.orchestra.run.smithi107.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.noname-h 2023-09-30T01:32:52.365 INFO:teuthology.orchestra.run.smithi107.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.noname-d 2023-09-30T01:32:52.365 INFO:teuthology.orchestra.run.smithi107.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.noname-i 2023-09-30T01:32:52.365 INFO:teuthology.orchestra.run.smithi107.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.noname-e 2023-09-30T01:32:52.365 INFO:teuthology.orchestra.run.smithi107.stderr: 2023-09-30T01:32:52.365 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.341+0000 7f498ffff700 1 Processor -- start 2023-09-30T01:32:52.365 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.341+0000 7f498ffff700 1 -- start start 2023-09-30T01:32:52.365 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.341+0000 7f498ffff700 10 monclient: init 2023-09-30T01:32:52.365 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.341+0000 7f498ffff700 10 monclient: _reopen_session rank -1 2023-09-30T01:32:52.366 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.341+0000 7f498ffff700 10 monclient: _add_conns ranks=[4,5,8,3,0,7,2,1,6] 2023-09-30T01:32:52.366 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.341+0000 7f498ffff700 1 --2- >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f4990097a40 0x7f49900a1b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:32:52.366 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.341+0000 7f498ffff700 10 monclient(hunting): picked mon.noname-c con 0x7f4990097a40 addr [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] 2023-09-30T01:32:52.366 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.341+0000 7f498ffff700 1 --2- >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7f49901311c0 0x7f49900a2040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:32:52.366 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.341+0000 7f498ffff700 10 monclient(hunting): picked mon.noname-h con 0x7f49901311c0 addr [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] 2023-09-30T01:32:52.366 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.341+0000 7f498ffff700 1 --2- >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f49901332e0 0x7f49900a2580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:32:52.366 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.341+0000 7f498ffff700 10 monclient(hunting): picked mon.noname-e con 0x7f49901332e0 addr [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] 2023-09-30T01:32:52.367 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.341+0000 7f498ffff700 10 monclient(hunting): start opening mon connection 2023-09-30T01:32:52.367 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.341+0000 7f498ffff700 1 -- --> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] -- mon_getmap magic: 0 v1 -- 0x7f499011eff0 con 0x7f4990097a40 2023-09-30T01:32:52.367 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.341+0000 7f498ffff700 10 monclient(hunting): start opening mon connection 2023-09-30T01:32:52.367 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.341+0000 7f498ffff700 1 -- --> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] -- mon_getmap magic: 0 v1 -- 0x7f4990099c30 con 0x7f49901311c0 2023-09-30T01:32:52.367 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.341+0000 7f498ffff700 10 monclient(hunting): start opening mon connection 2023-09-30T01:32:52.367 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.341+0000 7f498ffff700 1 -- --> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] -- mon_getmap magic: 0 v1 -- 0x7f49900996b0 con 0x7f49901332e0 2023-09-30T01:32:52.367 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.341+0000 7f498ffff700 10 monclient(hunting): _renew_subs 2023-09-30T01:32:52.368 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.341+0000 7f498ffff700 10 monclient(hunting): authenticate will time out at 855.766291s 2023-09-30T01:32:52.368 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.341+0000 7f498e7fc700 1 --2- >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7f49901311c0 0x7f49900a2040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:32:52.368 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.341+0000 7f498e7fc700 1 --2- >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7f49901311c0 0x7f49900a2040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3302/0 says I am v2:172.21.15.107:36252/0 (socket says 172.21.15.107:36252) 2023-09-30T01:32:52.368 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.341+0000 7f498e7fc700 1 -- 172.21.15.107:0/2838193202 learned_addr learned my addr 172.21.15.107:0/2838193202 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-09-30T01:32:52.368 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.341+0000 7f498e7fc700 10 monclient(hunting): get_auth_request con 0x7f49901311c0 auth_method 0 2023-09-30T01:32:52.368 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.341+0000 7f498e7fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:32:52.368 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.341+0000 7f498e7fc700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:32:52.368 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.341+0000 7f498e7fc700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:32:52.368 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.341+0000 7f498e7fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:32:52.368 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.341+0000 7f498e7fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:32:52.369 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.341+0000 7f498e7fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:32:52.369 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.341+0000 7f498f7fe700 1 --2- 172.21.15.107:0/2838193202 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f49901332e0 0x7f49900a2580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:32:52.369 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.341+0000 7f498effd700 1 --2- 172.21.15.107:0/2838193202 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f4990097a40 0x7f49900a1b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:32:52.369 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.341+0000 7f498effd700 10 monclient(hunting): get_auth_request con 0x7f4990097a40 auth_method 0 2023-09-30T01:32:52.369 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.341+0000 7f498effd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:32:52.369 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.341+0000 7f498effd700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:32:52.369 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.341+0000 7f498effd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:32:52.369 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.341+0000 7f498f7fe700 10 monclient(hunting): get_auth_request con 0x7f49901332e0 auth_method 0 2023-09-30T01:32:52.369 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.341+0000 7f498f7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:32:52.369 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.341+0000 7f498f7fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:32:52.370 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.341+0000 7f498f7fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:32:52.370 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.341+0000 7f498e7fc700 10 monclient(hunting): handle_auth_done global_id 4154 payload 824 2023-09-30T01:32:52.370 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.341+0000 7f498e7fc700 1 -- 172.21.15.107:0/2838193202 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f49901332e0 msgr2=0x7f49900a2580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:32:52.370 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.341+0000 7f498e7fc700 1 --2- 172.21.15.107:0/2838193202 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f49901332e0 0x7f49900a2580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:32:52.370 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.341+0000 7f498e7fc700 1 -- 172.21.15.107:0/2838193202 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f4990097a40 msgr2=0x7f49900a1b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:32:52.370 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.341+0000 7f498e7fc700 1 --2- 172.21.15.107:0/2838193202 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f4990097a40 0x7f49900a1b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:32:52.370 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.341+0000 7f498e7fc700 10 monclient: _finish_hunting 0 2023-09-30T01:32:52.370 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.341+0000 7f498e7fc700 1 monclient: found mon.noname-h 2023-09-30T01:32:52.370 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.341+0000 7f498e7fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:32:52.371 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.341+0000 7f498e7fc700 10 monclient: _send_mon_message to mon.noname-h at v2:172.21.15.107:3302/0 2023-09-30T01:32:52.371 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.341+0000 7f498e7fc700 1 -- 172.21.15.107:0/2838193202 --> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4984009420 con 0x7f49901311c0 2023-09-30T01:32:52.371 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.341+0000 7f498effd700 1 --2- 172.21.15.107:0/2838193202 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f4990097a40 0x7f49900a1b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-30T01:32:52.371 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.341+0000 7f498e7fc700 1 --2- 172.21.15.107:0/2838193202 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7f49901311c0 0x7f49900a2040 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f498400ece0 tx=0x7f49840091a0 comp rx=0 tx=0).ready entity=mon.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:32:52.371 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.341+0000 7f4976ffd700 1 -- 172.21.15.107:0/2838193202 <== mon.5 v2:172.21.15.107:3302/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f4984009780 con 0x7f49901311c0 2023-09-30T01:32:52.371 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.341+0000 7f4976ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:32:52.371 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.341+0000 7f4976ffd700 10 monclient: got monmap 1 from mon.noname-h (according to old e1) 2023-09-30T01:32:52.371 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.341+0000 7f4976ffd700 10 monclient: dump: 2023-09-30T01:32:52.371 INFO:teuthology.orchestra.run.smithi107.stderr:epoch 1 2023-09-30T01:32:52.371 INFO:teuthology.orchestra.run.smithi107.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:32:52.372 INFO:teuthology.orchestra.run.smithi107.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:32:52.372 INFO:teuthology.orchestra.run.smithi107.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:32:52.372 INFO:teuthology.orchestra.run.smithi107.stderr:min_mon_release 17 (quincy) 2023-09-30T01:32:52.372 INFO:teuthology.orchestra.run.smithi107.stderr:election_strategy: 3 2023-09-30T01:32:52.372 INFO:teuthology.orchestra.run.smithi107.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:32:52.372 INFO:teuthology.orchestra.run.smithi107.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:32:52.372 INFO:teuthology.orchestra.run.smithi107.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:32:52.372 INFO:teuthology.orchestra.run.smithi107.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:32:52.372 INFO:teuthology.orchestra.run.smithi107.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:32:52.372 INFO:teuthology.orchestra.run.smithi107.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:32:52.373 INFO:teuthology.orchestra.run.smithi107.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:32:52.373 INFO:teuthology.orchestra.run.smithi107.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:32:52.373 INFO:teuthology.orchestra.run.smithi107.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:32:52.373 INFO:teuthology.orchestra.run.smithi107.stderr: 2023-09-30T01:32:52.373 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.341+0000 7f4976ffd700 10 monclient: _finish_auth 0 2023-09-30T01:32:52.373 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.341+0000 7f4976ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:32:52.373 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.341+0000 7f4976ffd700 1 -- 172.21.15.107:0/2838193202 <== mon.5 v2:172.21.15.107:3302/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f498400cbe0 con 0x7f49901311c0 2023-09-30T01:32:52.373 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.341+0000 7f498ffff700 5 monclient: authenticate success, global_id 4154 2023-09-30T01:32:52.374 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.341+0000 7f498ffff700 10 monclient: _renew_subs 2023-09-30T01:32:52.374 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.341+0000 7f498ffff700 10 monclient: _send_mon_message to mon.h at v2:172.21.15.107:3302/0 2023-09-30T01:32:52.374 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.341+0000 7f498ffff700 1 -- 172.21.15.107:0/2838193202 --> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f49800094f0 con 0x7f49901311c0 2023-09-30T01:32:52.374 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.341+0000 7f4976ffd700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:32:52.374 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.341+0000 7f4976ffd700 1 -- 172.21.15.107:0/2838193202 <== mon.5 v2:172.21.15.107:3302/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f4984013d70 con 0x7f49901311c0 2023-09-30T01:32:52.374 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.341+0000 7f4976ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:32:52.374 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.341+0000 7f4976ffd700 10 monclient: got monmap 1 from mon.h (according to old e1) 2023-09-30T01:32:52.374 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.341+0000 7f4976ffd700 10 monclient: dump: 2023-09-30T01:32:52.374 INFO:teuthology.orchestra.run.smithi107.stderr:epoch 1 2023-09-30T01:32:52.374 INFO:teuthology.orchestra.run.smithi107.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:32:52.375 INFO:teuthology.orchestra.run.smithi107.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:32:52.375 INFO:teuthology.orchestra.run.smithi107.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:32:52.375 INFO:teuthology.orchestra.run.smithi107.stderr:min_mon_release 17 (quincy) 2023-09-30T01:32:52.375 INFO:teuthology.orchestra.run.smithi107.stderr:election_strategy: 3 2023-09-30T01:32:52.375 INFO:teuthology.orchestra.run.smithi107.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:32:52.375 INFO:teuthology.orchestra.run.smithi107.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:32:52.375 INFO:teuthology.orchestra.run.smithi107.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:32:52.375 INFO:teuthology.orchestra.run.smithi107.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:32:52.375 INFO:teuthology.orchestra.run.smithi107.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:32:52.375 INFO:teuthology.orchestra.run.smithi107.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:32:52.376 INFO:teuthology.orchestra.run.smithi107.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:32:52.376 INFO:teuthology.orchestra.run.smithi107.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:32:52.376 INFO:teuthology.orchestra.run.smithi107.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:32:52.376 INFO:teuthology.orchestra.run.smithi107.stderr: 2023-09-30T01:32:52.376 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.341+0000 7f498cff9700 4 set_mon_vals no callback set 2023-09-30T01:32:52.376 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.341+0000 7f498ffff700 10 monclient: _renew_subs 2023-09-30T01:32:52.377 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.341+0000 7f498ffff700 10 monclient: _send_mon_message to mon.h at v2:172.21.15.107:3302/0 2023-09-30T01:32:52.377 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.341+0000 7f498ffff700 1 -- 172.21.15.107:0/2838193202 --> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f499011d1e0 con 0x7f49901311c0 2023-09-30T01:32:52.377 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.341+0000 7f4976ffd700 1 -- 172.21.15.107:0/2838193202 <== mon.5 v2:172.21.15.107:3302/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7f4984013470 con 0x7f49901311c0 2023-09-30T01:32:52.377 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.345+0000 7f4976ffd700 1 -- 172.21.15.107:0/2838193202 <== mon.5 v2:172.21.15.107:3302/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 951+0+0 (secure 0 0 0) 0x7f4984013b50 con 0x7f49901311c0 2023-09-30T01:32:52.377 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.345+0000 7f498ffff700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:32:52.377 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.345+0000 7f498ffff700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:32:52.377 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.345+0000 7f498ffff700 10 monclient: _send_mon_message to mon.h at v2:172.21.15.107:3302/0 2023-09-30T01:32:52.377 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.345+0000 7f498ffff700 1 -- 172.21.15.107:0/2838193202 --> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4990071960 con 0x7f49901311c0 2023-09-30T01:32:52.378 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.349+0000 7f4976ffd700 1 -- 172.21.15.107:0/2838193202 <== mon.5 v2:172.21.15.107:3302/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7f4984017070 con 0x7f49901311c0 2023-09-30T01:32:52.378 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.349+0000 7f4976ffd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:32:52.379 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.349+0000 7f4976ffd700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:32:52.379 INFO:tasks.ceph.mgr.x.smithi107.stderr:2023-09-30T01:32:52.361+0000 7f46b28dd000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-09-30T01:32:52.420 INFO:tasks.ceph.mgr.x.smithi107.stderr:2023-09-30T01:32:52.417+0000 7f46b28dd000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-09-30T01:32:52.428 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.425+0000 7f498ffff700 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "fbc7be74-cd43-4c72-89c1-469c17055b85", "id": 2}] 2023-09-30T01:32:52.428 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.425+0000 7f498ffff700 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "fbc7be74-cd43-4c72-89c1-469c17055b85", "id": 2}] 2023-09-30T01:32:52.428 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.425+0000 7f498ffff700 10 monclient: _send_mon_message to mon.h at v2:172.21.15.107:3302/0 2023-09-30T01:32:52.428 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.425+0000 7f498ffff700 1 -- 172.21.15.107:0/2838193202 --> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] -- mon_command({"prefix": "osd new", "uuid": "fbc7be74-cd43-4c72-89c1-469c17055b85", "id": 2} v 0) v1 -- 0x7f4990067b60 con 0x7f49901311c0 2023-09-30T01:32:52.438 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.433+0000 7f4976ffd700 1 -- 172.21.15.107:0/2838193202 <== mon.5 v2:172.21.15.107:3302/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "fbc7be74-cd43-4c72-89c1-469c17055b85", "id": 2}]=0 v5) v1 ==== 112+0+2 (secure 0 0 0) 0x7f4984029020 con 0x7f49901311c0 2023-09-30T01:32:52.438 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.433+0000 7f4976ffd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "fbc7be74-cd43-4c72-89c1-469c17055b85", "id": 2}] 2023-09-30T01:32:52.438 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.433+0000 7f4976ffd700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:32:52.438 INFO:teuthology.orchestra.run.smithi107.stdout:2 2023-09-30T01:32:52.439 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.433+0000 7f498ffff700 10 monclient: shutdown 2023-09-30T01:32:52.440 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.433+0000 7f498ffff700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:32:52.440 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.433+0000 7f498ffff700 1 -- 172.21.15.107:0/2838193202 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7f49901311c0 msgr2=0x7f49900a2040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:32:52.440 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.433+0000 7f498ffff700 1 --2- 172.21.15.107:0/2838193202 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7f49901311c0 0x7f49900a2040 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f498400ece0 tx=0x7f49840091a0 comp rx=0 tx=0).stop 2023-09-30T01:32:52.440 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.433+0000 7f498ffff700 1 -- 172.21.15.107:0/2838193202 shutdown_connections 2023-09-30T01:32:52.440 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.433+0000 7f498ffff700 1 --2- 172.21.15.107:0/2838193202 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7f49901311c0 0x7f49900a2040 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:32:52.440 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.433+0000 7f498ffff700 1 --2- 172.21.15.107:0/2838193202 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f49901332e0 0x7f49900a2580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:32:52.440 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.433+0000 7f498ffff700 1 --2- 172.21.15.107:0/2838193202 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f4990097a40 0x7f49900a1b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:32:52.440 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.433+0000 7f498ffff700 1 -- 172.21.15.107:0/2838193202 >> 172.21.15.107:0/2838193202 conn(0x7f499006e5e0 msgr2=0x7f49901327b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:32:52.440 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.433+0000 7f498ffff700 1 -- 172.21.15.107:0/2838193202 shutdown_connections 2023-09-30T01:32:52.440 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.433+0000 7f498ffff700 1 -- 172.21.15.107:0/2838193202 wait complete. 2023-09-30T01:32:52.450 DEBUG:teuthology.orchestra.run.smithi107:> sudo ceph --cluster ceph osd new ca888e00-8f75-4dc4-b70c-6cd46880fe74 3 2023-09-30T01:32:52.541 INFO:tasks.ceph.mgr.x.smithi107.stderr:2023-09-30T01:32:52.537+0000 7f46b28dd000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-09-30T01:32:52.555 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.549+0000 7f10d0733700 10 monclient: get_monmap_and_config 2023-09-30T01:32:52.555 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.549+0000 7f10d0733700 10 monclient: build_initial_monmap 2023-09-30T01:32:52.555 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.549+0000 7f10d0733700 10 monclient: monmap: 2023-09-30T01:32:52.555 INFO:teuthology.orchestra.run.smithi107.stderr:epoch 0 2023-09-30T01:32:52.555 INFO:teuthology.orchestra.run.smithi107.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:32:52.555 INFO:teuthology.orchestra.run.smithi107.stderr:last_changed 2023-09-30T01:32:52.554735+0000 2023-09-30T01:32:52.556 INFO:teuthology.orchestra.run.smithi107.stderr:created 2023-09-30T01:32:52.554735+0000 2023-09-30T01:32:52.556 INFO:teuthology.orchestra.run.smithi107.stderr:min_mon_release 0 (unknown) 2023-09-30T01:32:52.556 INFO:teuthology.orchestra.run.smithi107.stderr:election_strategy: 3 2023-09-30T01:32:52.556 INFO:teuthology.orchestra.run.smithi107.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a-legacy 2023-09-30T01:32:52.556 INFO:teuthology.orchestra.run.smithi107.stderr:1: v1:172.21.15.107:6789/0 mon.noname-f-legacy 2023-09-30T01:32:52.556 INFO:teuthology.orchestra.run.smithi107.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.noname-b 2023-09-30T01:32:52.556 INFO:teuthology.orchestra.run.smithi107.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.noname-g 2023-09-30T01:32:52.556 INFO:teuthology.orchestra.run.smithi107.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.noname-c 2023-09-30T01:32:52.556 INFO:teuthology.orchestra.run.smithi107.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.noname-h 2023-09-30T01:32:52.557 INFO:teuthology.orchestra.run.smithi107.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.noname-d 2023-09-30T01:32:52.557 INFO:teuthology.orchestra.run.smithi107.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.noname-i 2023-09-30T01:32:52.557 INFO:teuthology.orchestra.run.smithi107.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.noname-e 2023-09-30T01:32:52.557 INFO:teuthology.orchestra.run.smithi107.stderr:9: v2:172.21.15.88:3300/0 mon.noname-a 2023-09-30T01:32:52.557 INFO:teuthology.orchestra.run.smithi107.stderr:10: v2:172.21.15.107:3300/0 mon.noname-f 2023-09-30T01:32:52.557 INFO:teuthology.orchestra.run.smithi107.stderr: 2023-09-30T01:32:52.557 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.549+0000 7f10d0733700 1 Processor -- start 2023-09-30T01:32:52.557 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.549+0000 7f10d0733700 1 -- start start 2023-09-30T01:32:52.558 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.549+0000 7f10d0733700 10 monclient: init 2023-09-30T01:32:52.569 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.553+0000 7f10d0733700 10 monclient: _reopen_session rank -1 2023-09-30T01:32:52.569 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.553+0000 7f10d0733700 10 monclient: _add_conns ranks=[8,1,9,6,2,10,3,4,0,5,7] 2023-09-30T01:32:52.569 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.553+0000 7f10d0733700 1 --2- >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f10c80a3270 0x7f10c80a0160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:32:52.569 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.553+0000 7f10d0733700 10 monclient(hunting): picked mon.noname-e con 0x7f10c80a3270 addr [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] 2023-09-30T01:32:52.569 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.553+0000 7f10d0733700 10 monclient(hunting): picked mon.noname-f-legacy con 0x7f10c80a4b00 addr v1:172.21.15.107:6789/0 2023-09-30T01:32:52.569 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.553+0000 7f10d0733700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f10c8138750 0x7f10c813ab30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:32:52.569 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.553+0000 7f10d0733700 10 monclient(hunting): picked mon.noname-a con 0x7f10c8138750 addr v2:172.21.15.88:3300/0 2023-09-30T01:32:52.570 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.553+0000 7f10d0733700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10c805ecb0 con 0x7f10c80a4b00 2023-09-30T01:32:52.570 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.553+0000 7f10d0733700 10 monclient(hunting): start opening mon connection 2023-09-30T01:32:52.570 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.553+0000 7f10d0733700 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f10c80a4ed0 con 0x7f10c8138750 2023-09-30T01:32:52.570 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.553+0000 7f10d0733700 10 monclient(hunting): start opening mon connection 2023-09-30T01:32:52.570 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.553+0000 7f10d0733700 1 -- --> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] -- mon_getmap magic: 0 v1 -- 0x7f10c80a6330 con 0x7f10c80a3270 2023-09-30T01:32:52.570 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.553+0000 7f10d0733700 10 monclient(hunting): _renew_subs 2023-09-30T01:32:52.570 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.553+0000 7f10d0733700 10 monclient(hunting): authenticate will time out at 855.976164s 2023-09-30T01:32:52.570 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.553+0000 7f10cdcce700 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f10c80a4b00 0x7f10c80a06a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:59722/0 (socket says 172.21.15.107:59722) 2023-09-30T01:32:52.570 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.553+0000 7f10cdcce700 1 -- 172.21.15.107:0/3278274572 learned_addr learned my addr 172.21.15.107:0/3278274572 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-09-30T01:32:52.570 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.553+0000 7f10ce4cf700 1 --2- 172.21.15.107:0/3278274572 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f10c80a3270 0x7f10c80a0160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:32:52.571 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.553+0000 7f10cecd0700 1 --2- 172.21.15.107:0/3278274572 >> v2:172.21.15.88:3300/0 conn(0x7f10c8138750 0x7f10c813ab30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:32:52.571 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.553+0000 7f10ce4cf700 10 monclient(hunting): get_auth_request con 0x7f10c80a3270 auth_method 0 2023-09-30T01:32:52.571 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.553+0000 7f10ce4cf700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:32:52.571 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.553+0000 7f10ce4cf700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:32:52.571 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.553+0000 7f10ce4cf700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:32:52.571 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.553+0000 7f10cecd0700 10 monclient(hunting): get_auth_request con 0x7f10c8138750 auth_method 0 2023-09-30T01:32:52.571 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.553+0000 7f10cecd0700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:32:52.571 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.553+0000 7f10cecd0700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:32:52.571 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.553+0000 7f10cecd0700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:32:52.571 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.553+0000 7f10cd4cd700 1 -- 172.21.15.107:0/3278274572 <== mon.1 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 39584027 0 0) 0x7f10c805ecb0 con 0x7f10c80a4b00 2023-09-30T01:32:52.572 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.553+0000 7f10cd4cd700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:32:52.572 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.553+0000 7f10cd4cd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:32:52.572 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.553+0000 7f10cd4cd700 10 monclient(hunting): my global_id is 4141 2023-09-30T01:32:52.572 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.553+0000 7f10cd4cd700 1 -- 172.21.15.107:0/3278274572 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f10b4003570 con 0x7f10c80a4b00 2023-09-30T01:32:52.572 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.553+0000 7f10ce4cf700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:32:52.572 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.553+0000 7f10ce4cf700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:32:52.572 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.553+0000 7f10ce4cf700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:32:52.572 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.553+0000 7f10cecd0700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:32:52.572 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.553+0000 7f10cecd0700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:32:52.572 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.553+0000 7f10cecd0700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:32:52.573 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.553+0000 7f10cd4cd700 1 -- 172.21.15.107:0/3278274572 <== mon.1 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 148286910 0 0) 0x7f10b4003570 con 0x7f10c80a4b00 2023-09-30T01:32:52.573 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.553+0000 7f10cd4cd700 1 -- 172.21.15.107:0/3278274572 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f10c80a3270 msgr2=0x7f10c80a0160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:32:52.573 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.553+0000 7f10cd4cd700 1 --2- 172.21.15.107:0/3278274572 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f10c80a3270 0x7f10c80a0160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:32:52.573 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.553+0000 7f10cd4cd700 1 -- 172.21.15.107:0/3278274572 >> v2:172.21.15.88:3300/0 conn(0x7f10c8138750 msgr2=0x7f10c813ab30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:32:52.573 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.553+0000 7f10cd4cd700 1 --2- 172.21.15.107:0/3278274572 >> v2:172.21.15.88:3300/0 conn(0x7f10c8138750 0x7f10c813ab30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:32:52.573 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.553+0000 7f10cd4cd700 10 monclient: _finish_hunting 0 2023-09-30T01:32:52.573 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.553+0000 7f10cd4cd700 1 monclient: found mon.noname-f-legacy 2023-09-30T01:32:52.573 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.553+0000 7f10cd4cd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:32:52.573 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.553+0000 7f10cd4cd700 10 monclient: _send_mon_message to mon.noname-f-legacy at v1:172.21.15.107:6789/0 2023-09-30T01:32:52.573 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.553+0000 7f10cd4cd700 1 -- 172.21.15.107:0/3278274572 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10c8098610 con 0x7f10c80a4b00 2023-09-30T01:32:52.574 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.553+0000 7f10cd4cd700 10 monclient: _finish_auth 0 2023-09-30T01:32:52.574 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.553+0000 7f10cd4cd700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:32:52.574 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.553+0000 7f10d0733700 5 monclient: authenticate success, global_id 4141 2023-09-30T01:32:52.574 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.553+0000 7f10d0733700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:32:52.574 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.553+0000 7f10cd4cd700 1 -- 172.21.15.107:0/3278274572 <== mon.1 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (unknown 176869241 0 0) 0x7f10b80018f0 con 0x7f10c80a4b00 2023-09-30T01:32:52.574 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.553+0000 7f10cd4cd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:32:52.574 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.553+0000 7f10cd4cd700 10 monclient: got monmap 1 from mon.noname-f-legacy (according to old e1) 2023-09-30T01:32:52.574 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.553+0000 7f10cd4cd700 10 monclient: dump: 2023-09-30T01:32:52.574 INFO:teuthology.orchestra.run.smithi107.stderr:epoch 1 2023-09-30T01:32:52.574 INFO:teuthology.orchestra.run.smithi107.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:32:52.574 INFO:teuthology.orchestra.run.smithi107.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:32:52.575 INFO:teuthology.orchestra.run.smithi107.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:32:52.575 INFO:teuthology.orchestra.run.smithi107.stderr:min_mon_release 17 (quincy) 2023-09-30T01:32:52.575 INFO:teuthology.orchestra.run.smithi107.stderr:election_strategy: 3 2023-09-30T01:32:52.575 INFO:teuthology.orchestra.run.smithi107.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:32:52.575 INFO:teuthology.orchestra.run.smithi107.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:32:52.575 INFO:teuthology.orchestra.run.smithi107.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:32:52.575 INFO:teuthology.orchestra.run.smithi107.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:32:52.575 INFO:teuthology.orchestra.run.smithi107.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:32:52.575 INFO:teuthology.orchestra.run.smithi107.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:32:52.575 INFO:teuthology.orchestra.run.smithi107.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:32:52.576 INFO:teuthology.orchestra.run.smithi107.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:32:52.576 INFO:teuthology.orchestra.run.smithi107.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:32:52.576 INFO:teuthology.orchestra.run.smithi107.stderr: 2023-09-30T01:32:52.576 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.553+0000 7f10cd4cd700 1 monclient: mon.f has (v2) addrs [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] but i'm connected to v1:172.21.15.107:6789/0, reconnecting 2023-09-30T01:32:52.576 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.553+0000 7f10cd4cd700 10 monclient: _reopen_session rank -1 2023-09-30T01:32:52.576 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.553+0000 7f10cd4cd700 1 -- 172.21.15.107:0/3278274572 >> v1:172.21.15.107:6789/0 conn(0x7f10c80a4b00 legacy=0x7f10c80a06a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:32:52.576 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.553+0000 7f10cd4cd700 10 monclient: _add_conns ranks=[3,8,4,7,1,0,5,2,6] 2023-09-30T01:32:52.576 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.553+0000 7f10cd4cd700 1 --2- 172.21.15.107:0/3278274572 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7f10b40042e0 0x7f10b4004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:32:52.576 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.553+0000 7f10cd4cd700 10 monclient(hunting): picked mon.g con 0x7f10b40042e0 addr [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] 2023-09-30T01:32:52.576 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.553+0000 7f10cd4cd700 1 --2- 172.21.15.107:0/3278274572 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f10c80a3270 0x7f10b4004e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:32:52.577 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.553+0000 7f10cd4cd700 10 monclient(hunting): picked mon.e con 0x7f10c80a3270 addr [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] 2023-09-30T01:32:52.577 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.553+0000 7f10cd4cd700 1 --2- 172.21.15.107:0/3278274572 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f10b40053d0 0x7f10b400ba80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:32:52.577 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.553+0000 7f10cd4cd700 10 monclient(hunting): picked mon.c con 0x7f10b40053d0 addr [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] 2023-09-30T01:32:52.577 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.553+0000 7f10cd4cd700 10 monclient(hunting): start opening mon connection 2023-09-30T01:32:52.577 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.553+0000 7f10cd4cd700 1 -- 172.21.15.107:0/3278274572 --> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10c80a4ed0 con 0x7f10b40042e0 2023-09-30T01:32:52.577 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.553+0000 7f10cd4cd700 10 monclient(hunting): start opening mon connection 2023-09-30T01:32:52.577 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.553+0000 7f10cd4cd700 1 -- 172.21.15.107:0/3278274572 --> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] -- mon_getmap magic: 0 v1 -- 0x7f10c80a6330 con 0x7f10b40053d0 2023-09-30T01:32:52.577 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.553+0000 7f10cd4cd700 10 monclient(hunting): start opening mon connection 2023-09-30T01:32:52.577 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.553+0000 7f10cd4cd700 1 -- 172.21.15.107:0/3278274572 --> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] -- mon_getmap magic: 0 v1 -- 0x7f10b400c300 con 0x7f10c80a3270 2023-09-30T01:32:52.577 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.553+0000 7f10cd4cd700 10 monclient(hunting): _renew_subs 2023-09-30T01:32:52.578 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.553+0000 7f10cd4cd700 10 monclient(hunting): _finish_auth 0 2023-09-30T01:32:52.578 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.553+0000 7f10d0733700 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2023-09-30T01:32:52.578 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.553+0000 7f10ce4cf700 1 --2- 172.21.15.107:0/3278274572 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7f10b40042e0 0x7f10b4004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:32:52.578 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.553+0000 7f10ce4cf700 10 monclient(hunting): get_auth_request con 0x7f10b40042e0 auth_method 0 2023-09-30T01:32:52.578 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.553+0000 7f10ce4cf700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:32:52.578 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.553+0000 7f10ce4cf700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:32:52.578 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.553+0000 7f10ce4cf700 10 monclient(hunting): _init_auth already have auth, reseting 2023-09-30T01:32:52.578 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.553+0000 7f10cdcce700 1 --2- 172.21.15.107:0/3278274572 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f10c80a3270 0x7f10b4004e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:32:52.578 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.553+0000 7f10cdcce700 10 monclient(hunting): get_auth_request con 0x7f10c80a3270 auth_method 0 2023-09-30T01:32:52.578 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.553+0000 7f10cdcce700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:32:52.579 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.553+0000 7f10cdcce700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:32:52.579 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.553+0000 7f10cdcce700 10 monclient(hunting): _init_auth already have auth, reseting 2023-09-30T01:32:52.579 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.553+0000 7f10ce4cf700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:32:52.579 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.553+0000 7f10ce4cf700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:32:52.579 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.553+0000 7f10ce4cf700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-09-30T01:32:52.579 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.553+0000 7f10cecd0700 1 --2- 172.21.15.107:0/3278274572 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f10b40053d0 0x7f10b400ba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:32:52.579 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.553+0000 7f10cecd0700 10 monclient(hunting): get_auth_request con 0x7f10b40053d0 auth_method 0 2023-09-30T01:32:52.579 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.553+0000 7f10cecd0700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:32:52.579 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.553+0000 7f10cecd0700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:32:52.579 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.553+0000 7f10cecd0700 10 monclient(hunting): _init_auth already have auth, reseting 2023-09-30T01:32:52.580 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.553+0000 7f10cdcce700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:32:52.580 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.553+0000 7f10cdcce700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:32:52.580 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.553+0000 7f10cdcce700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-09-30T01:32:52.580 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.553+0000 7f10ce4cf700 10 monclient(hunting): handle_auth_done global_id 4141 payload 293 2023-09-30T01:32:52.580 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.553+0000 7f10ce4cf700 1 -- 172.21.15.107:0/3278274572 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f10c80a3270 msgr2=0x7f10b4004e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:32:52.580 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.553+0000 7f10ce4cf700 1 --2- 172.21.15.107:0/3278274572 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f10c80a3270 0x7f10b4004e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:32:52.580 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.553+0000 7f10ce4cf700 1 -- 172.21.15.107:0/3278274572 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f10b40053d0 msgr2=0x7f10b400ba80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:32:52.580 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.553+0000 7f10ce4cf700 1 --2- 172.21.15.107:0/3278274572 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f10b40053d0 0x7f10b400ba80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:32:52.581 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.553+0000 7f10ce4cf700 10 monclient: _finish_hunting 0 2023-09-30T01:32:52.581 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.553+0000 7f10ce4cf700 1 monclient: found mon.g 2023-09-30T01:32:52.581 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.553+0000 7f10ce4cf700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:32:52.581 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.553+0000 7f10ce4cf700 10 monclient: _send_mon_message to mon.g at v2:172.21.15.107:3301/0 2023-09-30T01:32:52.581 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.553+0000 7f10ce4cf700 1 -- 172.21.15.107:0/3278274572 --> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10b4003570 con 0x7f10b40042e0 2023-09-30T01:32:52.581 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.553+0000 7f10ce4cf700 10 monclient: _finish_auth 0 2023-09-30T01:32:52.581 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.553+0000 7f10ce4cf700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:32:52.581 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.553+0000 7f10ce4cf700 1 --2- 172.21.15.107:0/3278274572 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7f10b40042e0 0x7f10b4004810 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f10c4006f00 tx=0x7f10c40091a0 comp rx=0 tx=0).ready entity=mon.3 client_cookie=1a7d708fd8dc9673 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:32:52.581 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.553+0000 7f10cd4cd700 1 -- 172.21.15.107:0/3278274572 <== mon.3 v2:172.21.15.107:3301/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f10c400eb90 con 0x7f10b40042e0 2023-09-30T01:32:52.581 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.553+0000 7f10cd4cd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:32:52.582 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.553+0000 7f10cd4cd700 10 monclient: got monmap 1 from mon.g (according to old e1) 2023-09-30T01:32:52.582 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.553+0000 7f10cd4cd700 10 monclient: dump: 2023-09-30T01:32:52.582 INFO:teuthology.orchestra.run.smithi107.stderr:epoch 1 2023-09-30T01:32:52.582 INFO:teuthology.orchestra.run.smithi107.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:32:52.582 INFO:teuthology.orchestra.run.smithi107.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:32:52.582 INFO:teuthology.orchestra.run.smithi107.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:32:52.582 INFO:teuthology.orchestra.run.smithi107.stderr:min_mon_release 17 (quincy) 2023-09-30T01:32:52.582 INFO:teuthology.orchestra.run.smithi107.stderr:election_strategy: 3 2023-09-30T01:32:52.582 INFO:teuthology.orchestra.run.smithi107.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:32:52.583 INFO:teuthology.orchestra.run.smithi107.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:32:52.583 INFO:teuthology.orchestra.run.smithi107.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:32:52.583 INFO:teuthology.orchestra.run.smithi107.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:32:52.583 INFO:teuthology.orchestra.run.smithi107.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:32:52.583 INFO:teuthology.orchestra.run.smithi107.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:32:52.583 INFO:teuthology.orchestra.run.smithi107.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:32:52.583 INFO:teuthology.orchestra.run.smithi107.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:32:52.583 INFO:teuthology.orchestra.run.smithi107.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:32:52.583 INFO:teuthology.orchestra.run.smithi107.stderr: 2023-09-30T01:32:52.583 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.553+0000 7f10d0733700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:32:52.583 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.553+0000 7f10cd4cd700 1 -- 172.21.15.107:0/3278274572 <== mon.3 v2:172.21.15.107:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f10c400ed30 con 0x7f10b40042e0 2023-09-30T01:32:52.584 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.553+0000 7f10cd4cd700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:32:52.584 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.553+0000 7f10d0733700 10 monclient: get_monmap_and_config success 2023-09-30T01:32:52.584 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.553+0000 7f10cd4cd700 1 -- 172.21.15.107:0/3278274572 <== mon.3 v2:172.21.15.107:3301/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f10c400eea0 con 0x7f10b40042e0 2023-09-30T01:32:52.584 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.553+0000 7f10d0733700 4 set_mon_vals no callback set 2023-09-30T01:32:52.584 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.553+0000 7f10cd4cd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:32:52.584 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.553+0000 7f10cd4cd700 10 monclient: got monmap 1 from mon.g (according to old e1) 2023-09-30T01:32:52.584 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.553+0000 7f10cd4cd700 10 monclient: dump: 2023-09-30T01:32:52.584 INFO:teuthology.orchestra.run.smithi107.stderr:epoch 1 2023-09-30T01:32:52.584 INFO:teuthology.orchestra.run.smithi107.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:32:52.584 INFO:teuthology.orchestra.run.smithi107.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:32:52.585 INFO:teuthology.orchestra.run.smithi107.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:32:52.585 INFO:teuthology.orchestra.run.smithi107.stderr:min_mon_release 17 (quincy) 2023-09-30T01:32:52.585 INFO:teuthology.orchestra.run.smithi107.stderr:election_strategy: 3 2023-09-30T01:32:52.585 INFO:teuthology.orchestra.run.smithi107.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:32:52.585 INFO:teuthology.orchestra.run.smithi107.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:32:52.585 INFO:teuthology.orchestra.run.smithi107.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:32:52.585 INFO:teuthology.orchestra.run.smithi107.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:32:52.585 INFO:teuthology.orchestra.run.smithi107.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:32:52.585 INFO:teuthology.orchestra.run.smithi107.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:32:52.585 INFO:teuthology.orchestra.run.smithi107.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:32:52.585 INFO:teuthology.orchestra.run.smithi107.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:32:52.586 INFO:teuthology.orchestra.run.smithi107.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:32:52.586 INFO:teuthology.orchestra.run.smithi107.stderr: 2023-09-30T01:32:52.586 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.553+0000 7f10d0733700 10 monclient: shutdown 2023-09-30T01:32:52.586 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.553+0000 7f10d0733700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:32:52.586 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.553+0000 7f10d0733700 1 -- 172.21.15.107:0/3278274572 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7f10b40042e0 msgr2=0x7f10b4004810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:32:52.586 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.553+0000 7f10d0733700 1 --2- 172.21.15.107:0/3278274572 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7f10b40042e0 0x7f10b4004810 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f10c4006f00 tx=0x7f10c40091a0 comp rx=0 tx=0).stop 2023-09-30T01:32:52.586 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.553+0000 7f10cecd0700 1 -- 172.21.15.107:0/3278274572 reap_dead start 2023-09-30T01:32:52.586 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.553+0000 7f10d0733700 1 -- 172.21.15.107:0/3278274572 shutdown_connections 2023-09-30T01:32:52.586 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.553+0000 7f10d0733700 1 -- 172.21.15.107:0/3278274572 >> 172.21.15.107:0/3278274572 conn(0x7f10c80a67f0 msgr2=0x7f10c80a6bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:32:52.586 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.553+0000 7f10d0733700 1 -- 172.21.15.107:0/3278274572 shutdown_connections 2023-09-30T01:32:52.587 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.553+0000 7f10d0733700 1 -- 172.21.15.107:0/3278274572 wait complete. 2023-09-30T01:32:52.587 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.557+0000 7f10d0733700 10 monclient: build_initial_monmap 2023-09-30T01:32:52.587 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.557+0000 7f10d0733700 10 monclient: monmap: 2023-09-30T01:32:52.587 INFO:teuthology.orchestra.run.smithi107.stderr:epoch 0 2023-09-30T01:32:52.587 INFO:teuthology.orchestra.run.smithi107.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:32:52.587 INFO:teuthology.orchestra.run.smithi107.stderr:last_changed 0.000000 2023-09-30T01:32:52.587 INFO:teuthology.orchestra.run.smithi107.stderr:created 0.000000 2023-09-30T01:32:52.587 INFO:teuthology.orchestra.run.smithi107.stderr:min_mon_release 0 (unknown) 2023-09-30T01:32:52.587 INFO:teuthology.orchestra.run.smithi107.stderr:election_strategy: 1 2023-09-30T01:32:52.587 INFO:teuthology.orchestra.run.smithi107.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.noname-a 2023-09-30T01:32:52.588 INFO:teuthology.orchestra.run.smithi107.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.noname-f 2023-09-30T01:32:52.588 INFO:teuthology.orchestra.run.smithi107.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.noname-b 2023-09-30T01:32:52.588 INFO:teuthology.orchestra.run.smithi107.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.noname-g 2023-09-30T01:32:52.588 INFO:teuthology.orchestra.run.smithi107.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.noname-c 2023-09-30T01:32:52.588 INFO:teuthology.orchestra.run.smithi107.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.noname-h 2023-09-30T01:32:52.588 INFO:teuthology.orchestra.run.smithi107.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.noname-d 2023-09-30T01:32:52.588 INFO:teuthology.orchestra.run.smithi107.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.noname-i 2023-09-30T01:32:52.588 INFO:teuthology.orchestra.run.smithi107.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.noname-e 2023-09-30T01:32:52.588 INFO:teuthology.orchestra.run.smithi107.stderr: 2023-09-30T01:32:52.588 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.557+0000 7f10d0733700 1 Processor -- start 2023-09-30T01:32:52.588 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.557+0000 7f10d0733700 1 -- start start 2023-09-30T01:32:52.589 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.557+0000 7f10d0733700 10 monclient: init 2023-09-30T01:32:52.589 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.557+0000 7f10d0733700 10 monclient: _reopen_session rank -1 2023-09-30T01:32:52.589 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.557+0000 7f10d0733700 10 monclient: _add_conns ranks=[5,8,1,0,7,4,2,6,3] 2023-09-30T01:32:52.589 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.557+0000 7f10d0733700 1 --2- >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7f10b40042e0 0x7f10c8142d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:32:52.589 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.557+0000 7f10d0733700 10 monclient(hunting): picked mon.noname-h con 0x7f10b40042e0 addr [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] 2023-09-30T01:32:52.589 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.557+0000 7f10d0733700 1 --2- >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f10c81432d0 0x7f10c8135090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:32:52.589 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.557+0000 7f10d0733700 10 monclient(hunting): picked mon.noname-e con 0x7f10c81432d0 addr [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] 2023-09-30T01:32:52.589 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.557+0000 7f10d0733700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f10c81436a0 0x7f10c81355d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:32:52.589 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.557+0000 7f10d0733700 10 monclient(hunting): picked mon.noname-f con 0x7f10c81436a0 addr [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] 2023-09-30T01:32:52.589 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.557+0000 7f10d0733700 10 monclient(hunting): start opening mon connection 2023-09-30T01:32:52.590 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.557+0000 7f10d0733700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10c8143a90 con 0x7f10c81436a0 2023-09-30T01:32:52.590 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.557+0000 7f10d0733700 10 monclient(hunting): start opening mon connection 2023-09-30T01:32:52.590 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.557+0000 7f10d0733700 1 -- --> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] -- mon_getmap magic: 0 v1 -- 0x7f10c8097c70 con 0x7f10b40042e0 2023-09-30T01:32:52.590 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.557+0000 7f10d0733700 10 monclient(hunting): start opening mon connection 2023-09-30T01:32:52.590 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.557+0000 7f10d0733700 1 -- --> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] -- mon_getmap magic: 0 v1 -- 0x7f10c80a0e80 con 0x7f10c81432d0 2023-09-30T01:32:52.590 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.557+0000 7f10d0733700 10 monclient(hunting): _renew_subs 2023-09-30T01:32:52.590 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.557+0000 7f10d0733700 10 monclient(hunting): authenticate will time out at 855.981952s 2023-09-30T01:32:52.590 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.557+0000 7f10ce4cf700 1 --2- >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7f10b40042e0 0x7f10c8142d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:32:52.590 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.557+0000 7f10ce4cf700 1 --2- >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7f10b40042e0 0x7f10c8142d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3302/0 says I am v2:172.21.15.107:36260/0 (socket says 172.21.15.107:36260) 2023-09-30T01:32:52.590 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.557+0000 7f10cecd0700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f10c81436a0 0x7f10c81355d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:32:52.591 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.557+0000 7f10ce4cf700 1 -- 172.21.15.107:0/2046613534 learned_addr learned my addr 172.21.15.107:0/2046613534 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-09-30T01:32:52.591 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.557+0000 7f10ce4cf700 10 monclient(hunting): get_auth_request con 0x7f10b40042e0 auth_method 0 2023-09-30T01:32:52.591 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.557+0000 7f10ce4cf700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:32:52.591 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.557+0000 7f10ce4cf700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:32:52.591 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.557+0000 7f10ce4cf700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:32:52.591 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.557+0000 7f10cdcce700 1 --2- 172.21.15.107:0/2046613534 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f10c81432d0 0x7f10c8135090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:32:52.591 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.557+0000 7f10cecd0700 10 monclient(hunting): get_auth_request con 0x7f10c81436a0 auth_method 0 2023-09-30T01:32:52.591 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.557+0000 7f10cecd0700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:32:52.591 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.557+0000 7f10cecd0700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:32:52.591 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.557+0000 7f10cecd0700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:32:52.592 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.557+0000 7f10cdcce700 10 monclient(hunting): get_auth_request con 0x7f10c81432d0 auth_method 0 2023-09-30T01:32:52.592 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.557+0000 7f10cdcce700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:32:52.592 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.557+0000 7f10cdcce700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:32:52.592 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.557+0000 7f10cdcce700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:32:52.592 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.557+0000 7f10ce4cf700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:32:52.592 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.557+0000 7f10ce4cf700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:32:52.592 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.557+0000 7f10ce4cf700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:32:52.592 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.557+0000 7f10cdcce700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:32:52.592 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.557+0000 7f10cdcce700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:32:52.592 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.557+0000 7f10cdcce700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:32:52.593 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.557+0000 7f10cecd0700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:32:52.593 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.557+0000 7f10cecd0700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:32:52.593 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.557+0000 7f10cecd0700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:32:52.593 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.557+0000 7f10ce4cf700 10 monclient(hunting): handle_auth_done global_id 4163 payload 824 2023-09-30T01:32:52.593 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.557+0000 7f10ce4cf700 1 -- 172.21.15.107:0/2046613534 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f10c81432d0 msgr2=0x7f10c8135090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:32:52.593 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.557+0000 7f10ce4cf700 1 --2- 172.21.15.107:0/2046613534 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f10c81432d0 0x7f10c8135090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:32:52.593 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.557+0000 7f10ce4cf700 1 -- 172.21.15.107:0/2046613534 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f10c81436a0 msgr2=0x7f10c81355d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:32:52.593 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.557+0000 7f10ce4cf700 1 --2- 172.21.15.107:0/2046613534 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f10c81436a0 0x7f10c81355d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:32:52.593 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.557+0000 7f10ce4cf700 10 monclient: _finish_hunting 0 2023-09-30T01:32:52.593 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.557+0000 7f10ce4cf700 1 monclient: found mon.noname-h 2023-09-30T01:32:52.594 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.557+0000 7f10ce4cf700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:32:52.594 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.557+0000 7f10ce4cf700 10 monclient: _send_mon_message to mon.noname-h at v2:172.21.15.107:3302/0 2023-09-30T01:32:52.594 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.557+0000 7f10ce4cf700 1 -- 172.21.15.107:0/2046613534 --> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10c8099710 con 0x7f10b40042e0 2023-09-30T01:32:52.594 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.557+0000 7f10cecd0700 1 --2- 172.21.15.107:0/2046613534 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f10c81436a0 0x7f10c81355d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:32:52.594 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.557+0000 7f10cdcce700 1 --2- 172.21.15.107:0/2046613534 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f10c81432d0 0x7f10c8135090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:32:52.594 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.557+0000 7f10ce4cf700 1 --2- 172.21.15.107:0/2046613534 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7f10b40042e0 0x7f10c8142d90 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f10c40091e0 tx=0x7f10c401a000 comp rx=0 tx=0).ready entity=mon.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:32:52.594 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.557+0000 7f10be7fc700 1 -- 172.21.15.107:0/2046613534 <== mon.5 v2:172.21.15.107:3302/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f10c4022070 con 0x7f10b40042e0 2023-09-30T01:32:52.594 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.557+0000 7f10be7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:32:52.594 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.557+0000 7f10be7fc700 10 monclient: got monmap 1 from mon.noname-h (according to old e1) 2023-09-30T01:32:52.594 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.557+0000 7f10be7fc700 10 monclient: dump: 2023-09-30T01:32:52.595 INFO:teuthology.orchestra.run.smithi107.stderr:epoch 1 2023-09-30T01:32:52.595 INFO:teuthology.orchestra.run.smithi107.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:32:52.595 INFO:teuthology.orchestra.run.smithi107.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:32:52.595 INFO:teuthology.orchestra.run.smithi107.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:32:52.595 INFO:teuthology.orchestra.run.smithi107.stderr:min_mon_release 17 (quincy) 2023-09-30T01:32:52.595 INFO:teuthology.orchestra.run.smithi107.stderr:election_strategy: 3 2023-09-30T01:32:52.595 INFO:teuthology.orchestra.run.smithi107.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:32:52.595 INFO:teuthology.orchestra.run.smithi107.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:32:52.595 INFO:teuthology.orchestra.run.smithi107.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:32:52.595 INFO:teuthology.orchestra.run.smithi107.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:32:52.596 INFO:teuthology.orchestra.run.smithi107.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:32:52.596 INFO:teuthology.orchestra.run.smithi107.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:32:52.596 INFO:teuthology.orchestra.run.smithi107.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:32:52.596 INFO:teuthology.orchestra.run.smithi107.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:32:52.596 INFO:teuthology.orchestra.run.smithi107.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:32:52.596 INFO:teuthology.orchestra.run.smithi107.stderr: 2023-09-30T01:32:52.596 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.557+0000 7f10be7fc700 10 monclient: _finish_auth 0 2023-09-30T01:32:52.596 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.557+0000 7f10be7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:32:52.596 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.557+0000 7f10be7fc700 1 -- 172.21.15.107:0/2046613534 <== mon.5 v2:172.21.15.107:3302/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f10c4013e80 con 0x7f10b40042e0 2023-09-30T01:32:52.596 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.557+0000 7f10d0733700 5 monclient: authenticate success, global_id 4163 2023-09-30T01:32:52.596 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.557+0000 7f10be7fc700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:32:52.597 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.557+0000 7f10d0733700 10 monclient: _renew_subs 2023-09-30T01:32:52.597 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.557+0000 7f10be7fc700 1 -- 172.21.15.107:0/2046613534 <== mon.5 v2:172.21.15.107:3302/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f10c400eea0 con 0x7f10b40042e0 2023-09-30T01:32:52.597 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.557+0000 7f10d0733700 10 monclient: _send_mon_message to mon.h at v2:172.21.15.107:3302/0 2023-09-30T01:32:52.597 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.557+0000 7f10d0733700 1 -- 172.21.15.107:0/2046613534 --> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f10c81378a0 con 0x7f10b40042e0 2023-09-30T01:32:52.597 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.557+0000 7f10be7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:32:52.597 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.557+0000 7f10be7fc700 10 monclient: got monmap 1 from mon.h (according to old e1) 2023-09-30T01:32:52.597 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.557+0000 7f10ccccc700 4 set_mon_vals no callback set 2023-09-30T01:32:52.597 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.557+0000 7f10be7fc700 10 monclient: dump: 2023-09-30T01:32:52.597 INFO:teuthology.orchestra.run.smithi107.stderr:epoch 1 2023-09-30T01:32:52.597 INFO:teuthology.orchestra.run.smithi107.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:32:52.598 INFO:teuthology.orchestra.run.smithi107.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:32:52.598 INFO:teuthology.orchestra.run.smithi107.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:32:52.598 INFO:teuthology.orchestra.run.smithi107.stderr:min_mon_release 17 (quincy) 2023-09-30T01:32:52.598 INFO:teuthology.orchestra.run.smithi107.stderr:election_strategy: 3 2023-09-30T01:32:52.598 INFO:teuthology.orchestra.run.smithi107.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:32:52.598 INFO:teuthology.orchestra.run.smithi107.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:32:52.598 INFO:teuthology.orchestra.run.smithi107.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:32:52.598 INFO:teuthology.orchestra.run.smithi107.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:32:52.598 INFO:teuthology.orchestra.run.smithi107.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:32:52.598 INFO:teuthology.orchestra.run.smithi107.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:32:52.598 INFO:teuthology.orchestra.run.smithi107.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:32:52.599 INFO:teuthology.orchestra.run.smithi107.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:32:52.599 INFO:teuthology.orchestra.run.smithi107.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:32:52.599 INFO:teuthology.orchestra.run.smithi107.stderr: 2023-09-30T01:32:52.599 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.557+0000 7f10d0733700 10 monclient: _renew_subs 2023-09-30T01:32:52.599 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.557+0000 7f10d0733700 10 monclient: _send_mon_message to mon.h at v2:172.21.15.107:3302/0 2023-09-30T01:32:52.599 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.557+0000 7f10d0733700 1 -- 172.21.15.107:0/2046613534 --> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f10c81373d0 con 0x7f10b40042e0 2023-09-30T01:32:52.599 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.557+0000 7f10be7fc700 1 -- 172.21.15.107:0/2046613534 <== mon.5 v2:172.21.15.107:3302/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7f10c401c6d0 con 0x7f10b40042e0 2023-09-30T01:32:52.599 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.557+0000 7f10be7fc700 1 -- 172.21.15.107:0/2046613534 <== mon.5 v2:172.21.15.107:3302/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1066+0+0 (secure 0 0 0) 0x7f10c40268b0 con 0x7f10b40042e0 2023-09-30T01:32:52.599 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.561+0000 7f10d0733700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:32:52.599 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.561+0000 7f10d0733700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:32:52.600 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.561+0000 7f10d0733700 10 monclient: _send_mon_message to mon.h at v2:172.21.15.107:3302/0 2023-09-30T01:32:52.600 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.561+0000 7f10d0733700 1 -- 172.21.15.107:0/2046613534 --> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f10c8132e90 con 0x7f10b40042e0 2023-09-30T01:32:52.600 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.561+0000 7f10be7fc700 1 -- 172.21.15.107:0/2046613534 <== mon.5 v2:172.21.15.107:3302/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7f10c8132e90 con 0x7f10b40042e0 2023-09-30T01:32:52.600 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.561+0000 7f10be7fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:32:52.600 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.561+0000 7f10be7fc700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:32:52.643 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.637+0000 7f10d0733700 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "ca888e00-8f75-4dc4-b70c-6cd46880fe74", "id": 3}] 2023-09-30T01:32:52.644 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.637+0000 7f10d0733700 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "ca888e00-8f75-4dc4-b70c-6cd46880fe74", "id": 3}] 2023-09-30T01:32:52.644 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.637+0000 7f10d0733700 10 monclient: _send_mon_message to mon.h at v2:172.21.15.107:3302/0 2023-09-30T01:32:52.644 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.637+0000 7f10d0733700 1 -- 172.21.15.107:0/2046613534 --> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] -- mon_command({"prefix": "osd new", "uuid": "ca888e00-8f75-4dc4-b70c-6cd46880fe74", "id": 3} v 0) v1 -- 0x7f10c8071960 con 0x7f10b40042e0 2023-09-30T01:32:52.656 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.653+0000 7f10be7fc700 1 -- 172.21.15.107:0/2046613534 <== mon.5 v2:172.21.15.107:3302/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "ca888e00-8f75-4dc4-b70c-6cd46880fe74", "id": 3}]=0 v6) v1 ==== 112+0+2 (secure 0 0 0) 0x7f10c402b070 con 0x7f10b40042e0 2023-09-30T01:32:52.656 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.653+0000 7f10be7fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "ca888e00-8f75-4dc4-b70c-6cd46880fe74", "id": 3}] 2023-09-30T01:32:52.656 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.653+0000 7f10be7fc700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:32:52.656 INFO:teuthology.orchestra.run.smithi107.stdout:3 2023-09-30T01:32:52.658 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.653+0000 7f10d0733700 10 monclient: shutdown 2023-09-30T01:32:52.658 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.653+0000 7f10d0733700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:32:52.658 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.653+0000 7f10d0733700 1 -- 172.21.15.107:0/2046613534 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7f10b40042e0 msgr2=0x7f10c8142d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:32:52.658 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.653+0000 7f10d0733700 1 --2- 172.21.15.107:0/2046613534 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7f10b40042e0 0x7f10c8142d90 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f10c40091e0 tx=0x7f10c401a000 comp rx=0 tx=0).stop 2023-09-30T01:32:52.658 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.653+0000 7f10d0733700 1 -- 172.21.15.107:0/2046613534 shutdown_connections 2023-09-30T01:32:52.658 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.653+0000 7f10d0733700 1 --2- 172.21.15.107:0/2046613534 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f10c81436a0 0x7f10c81355d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:32:52.658 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.653+0000 7f10d0733700 1 --2- 172.21.15.107:0/2046613534 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f10c81432d0 0x7f10c8135090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:32:52.659 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.653+0000 7f10d0733700 1 --2- 172.21.15.107:0/2046613534 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7f10b40042e0 0x7f10c8142d90 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:32:52.659 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.653+0000 7f10d0733700 1 -- 172.21.15.107:0/2046613534 >> 172.21.15.107:0/2046613534 conn(0x7f10c80a67f0 msgr2=0x7f10c8131fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:32:52.659 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.653+0000 7f10d0733700 1 -- 172.21.15.107:0/2046613534 shutdown_connections 2023-09-30T01:32:52.659 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.653+0000 7f10d0733700 1 -- 172.21.15.107:0/2046613534 wait complete. 2023-09-30T01:32:52.669 DEBUG:teuthology.orchestra.run.smithi107:> sudo ceph --cluster ceph osd new c8c73bd1-7674-4529-9104-8690e25af2fc 4 2023-09-30T01:32:52.770 INFO:tasks.ceph.mgr.x.smithi107.stderr:2023-09-30T01:32:52.765+0000 7f46b28dd000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-09-30T01:32:52.771 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.765+0000 7fbde4959700 10 monclient: get_monmap_and_config 2023-09-30T01:32:52.771 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.765+0000 7fbde4959700 10 monclient: build_initial_monmap 2023-09-30T01:32:52.771 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.765+0000 7fbde4959700 10 monclient: monmap: 2023-09-30T01:32:52.771 INFO:teuthology.orchestra.run.smithi107.stderr:epoch 0 2023-09-30T01:32:52.771 INFO:teuthology.orchestra.run.smithi107.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:32:52.772 INFO:teuthology.orchestra.run.smithi107.stderr:last_changed 2023-09-30T01:32:52.771432+0000 2023-09-30T01:32:52.772 INFO:teuthology.orchestra.run.smithi107.stderr:created 2023-09-30T01:32:52.771432+0000 2023-09-30T01:32:52.772 INFO:teuthology.orchestra.run.smithi107.stderr:min_mon_release 0 (unknown) 2023-09-30T01:32:52.772 INFO:teuthology.orchestra.run.smithi107.stderr:election_strategy: 3 2023-09-30T01:32:52.772 INFO:teuthology.orchestra.run.smithi107.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a-legacy 2023-09-30T01:32:52.772 INFO:teuthology.orchestra.run.smithi107.stderr:1: v1:172.21.15.107:6789/0 mon.noname-f-legacy 2023-09-30T01:32:52.772 INFO:teuthology.orchestra.run.smithi107.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.noname-b 2023-09-30T01:32:52.772 INFO:teuthology.orchestra.run.smithi107.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.noname-g 2023-09-30T01:32:52.772 INFO:teuthology.orchestra.run.smithi107.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.noname-c 2023-09-30T01:32:52.773 INFO:teuthology.orchestra.run.smithi107.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.noname-h 2023-09-30T01:32:52.773 INFO:teuthology.orchestra.run.smithi107.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.noname-d 2023-09-30T01:32:52.773 INFO:teuthology.orchestra.run.smithi107.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.noname-i 2023-09-30T01:32:52.773 INFO:teuthology.orchestra.run.smithi107.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.noname-e 2023-09-30T01:32:52.773 INFO:teuthology.orchestra.run.smithi107.stderr:9: v2:172.21.15.88:3300/0 mon.noname-a 2023-09-30T01:32:52.773 INFO:teuthology.orchestra.run.smithi107.stderr:10: v2:172.21.15.107:3300/0 mon.noname-f 2023-09-30T01:32:52.773 INFO:teuthology.orchestra.run.smithi107.stderr: 2023-09-30T01:32:52.783 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.769+0000 7fbde4959700 1 Processor -- start 2023-09-30T01:32:52.784 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.769+0000 7fbde4959700 1 -- start start 2023-09-30T01:32:52.784 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.769+0000 7fbde4959700 10 monclient: init 2023-09-30T01:32:52.785 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.769+0000 7fbde4959700 10 monclient: _reopen_session rank -1 2023-09-30T01:32:52.785 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.769+0000 7fbde4959700 10 monclient: _add_conns ranks=[9,2,5,10,1,7,0,3,4,8,6] 2023-09-30T01:32:52.785 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.769+0000 7fbde4959700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fbde00a17d0 0x7fbde01315e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:32:52.785 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.769+0000 7fbde4959700 10 monclient(hunting): picked mon.noname-a con 0x7fbde00a17d0 addr v2:172.21.15.88:3300/0 2023-09-30T01:32:52.785 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.769+0000 7fbde4959700 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fbde0131b20 0x7fbde006ea20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:32:52.785 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.769+0000 7fbde4959700 10 monclient(hunting): picked mon.noname-b con 0x7fbde0131b20 addr [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] 2023-09-30T01:32:52.786 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.769+0000 7fbde4959700 1 --2- >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7fbde006ef60 0x7fbde009edf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:32:52.786 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.769+0000 7fbde4959700 10 monclient(hunting): picked mon.noname-h con 0x7fbde006ef60 addr [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] 2023-09-30T01:32:52.786 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.769+0000 7fbde4959700 10 monclient(hunting): start opening mon connection 2023-09-30T01:32:52.786 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.769+0000 7fbde4959700 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbde006c610 con 0x7fbde00a17d0 2023-09-30T01:32:52.786 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.769+0000 7fbde4959700 10 monclient(hunting): start opening mon connection 2023-09-30T01:32:52.786 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.769+0000 7fbde4959700 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbde006cc50 con 0x7fbde0131b20 2023-09-30T01:32:52.786 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.769+0000 7fbde4959700 10 monclient(hunting): start opening mon connection 2023-09-30T01:32:52.786 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.769+0000 7fbde4959700 1 -- --> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] -- mon_getmap magic: 0 v1 -- 0x7fbde006cf30 con 0x7fbde006ef60 2023-09-30T01:32:52.786 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.769+0000 7fbde4959700 10 monclient(hunting): _renew_subs 2023-09-30T01:32:52.786 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.769+0000 7fbde4959700 10 monclient(hunting): authenticate will time out at 856.193121s 2023-09-30T01:32:52.786 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.769+0000 7fbddffff700 1 --2- >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7fbde006ef60 0x7fbde009edf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:32:52.787 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.769+0000 7fbddffff700 1 --2- >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7fbde006ef60 0x7fbde009edf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3302/0 says I am v2:172.21.15.107:36266/0 (socket says 172.21.15.107:36266) 2023-09-30T01:32:52.787 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.769+0000 7fbddffff700 1 -- 172.21.15.107:0/4270609384 learned_addr learned my addr 172.21.15.107:0/4270609384 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-09-30T01:32:52.787 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.769+0000 7fbddffff700 10 monclient(hunting): get_auth_request con 0x7fbde006ef60 auth_method 0 2023-09-30T01:32:52.787 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.769+0000 7fbddffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:32:52.787 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.769+0000 7fbddffff700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:32:52.787 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.769+0000 7fbddffff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:32:52.787 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.769+0000 7fbddffff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:32:52.787 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.769+0000 7fbddffff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:32:52.787 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.769+0000 7fbddf7fe700 1 --2- 172.21.15.107:0/4270609384 >> v2:172.21.15.88:3300/0 conn(0x7fbde00a17d0 0x7fbde01315e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:32:52.787 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.769+0000 7fbdd6ffd700 1 --2- 172.21.15.107:0/4270609384 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fbde0131b20 0x7fbde006ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:32:52.787 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.769+0000 7fbddffff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:32:52.787 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.769+0000 7fbddf7fe700 10 monclient(hunting): get_auth_request con 0x7fbde00a17d0 auth_method 0 2023-09-30T01:32:52.788 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.769+0000 7fbddf7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:32:52.788 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.769+0000 7fbddf7fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:32:52.788 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.769+0000 7fbddf7fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:32:52.788 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.769+0000 7fbdd6ffd700 10 monclient(hunting): get_auth_request con 0x7fbde0131b20 auth_method 0 2023-09-30T01:32:52.788 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.769+0000 7fbdd6ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:32:52.788 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.769+0000 7fbdd6ffd700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:32:52.788 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.769+0000 7fbdd6ffd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:32:52.788 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.769+0000 7fbddffff700 10 monclient(hunting): handle_auth_done global_id 4172 payload 274 2023-09-30T01:32:52.788 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.769+0000 7fbddffff700 1 -- 172.21.15.107:0/4270609384 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fbde0131b20 msgr2=0x7fbde006ea20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:32:52.788 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.769+0000 7fbddffff700 1 --2- 172.21.15.107:0/4270609384 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fbde0131b20 0x7fbde006ea20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:32:52.788 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.769+0000 7fbddffff700 1 -- 172.21.15.107:0/4270609384 >> v2:172.21.15.88:3300/0 conn(0x7fbde00a17d0 msgr2=0x7fbde01315e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:32:52.789 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.769+0000 7fbddffff700 1 --2- 172.21.15.107:0/4270609384 >> v2:172.21.15.88:3300/0 conn(0x7fbde00a17d0 0x7fbde01315e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:32:52.789 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.769+0000 7fbddffff700 10 monclient: _finish_hunting 0 2023-09-30T01:32:52.789 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.769+0000 7fbddffff700 1 monclient: found mon.noname-h 2023-09-30T01:32:52.789 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.769+0000 7fbddffff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:32:52.789 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.769+0000 7fbddffff700 10 monclient: _send_mon_message to mon.noname-h at v2:172.21.15.107:3302/0 2023-09-30T01:32:52.789 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.769+0000 7fbddffff700 1 -- 172.21.15.107:0/4270609384 --> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbde004bb20 con 0x7fbde006ef60 2023-09-30T01:32:52.789 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.769+0000 7fbddf7fe700 1 --2- 172.21.15.107:0/4270609384 >> v2:172.21.15.88:3300/0 conn(0x7fbde00a17d0 0x7fbde01315e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-30T01:32:52.789 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.769+0000 7fbddffff700 1 --2- 172.21.15.107:0/4270609384 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7fbde006ef60 0x7fbde009edf0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fbdd8009f20 tx=0x7fbdd8009840 comp rx=0 tx=0).ready entity=mon.5 client_cookie=31af9acd38fa5c85 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:32:52.789 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.769+0000 7fbddeffd700 1 -- 172.21.15.107:0/4270609384 <== mon.5 v2:172.21.15.107:3302/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7fbdd800e070 con 0x7fbde006ef60 2023-09-30T01:32:52.789 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.769+0000 7fbddeffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:32:52.789 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.769+0000 7fbddeffd700 10 monclient: got monmap 1 from mon.noname-h (according to old e1) 2023-09-30T01:32:52.789 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.769+0000 7fbddeffd700 10 monclient: dump: 2023-09-30T01:32:52.790 INFO:teuthology.orchestra.run.smithi107.stderr:epoch 1 2023-09-30T01:32:52.790 INFO:teuthology.orchestra.run.smithi107.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:32:52.790 INFO:teuthology.orchestra.run.smithi107.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:32:52.790 INFO:teuthology.orchestra.run.smithi107.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:32:52.790 INFO:teuthology.orchestra.run.smithi107.stderr:min_mon_release 17 (quincy) 2023-09-30T01:32:52.790 INFO:teuthology.orchestra.run.smithi107.stderr:election_strategy: 3 2023-09-30T01:32:52.790 INFO:teuthology.orchestra.run.smithi107.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:32:52.790 INFO:teuthology.orchestra.run.smithi107.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:32:52.790 INFO:teuthology.orchestra.run.smithi107.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:32:52.790 INFO:teuthology.orchestra.run.smithi107.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:32:52.790 INFO:teuthology.orchestra.run.smithi107.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:32:52.791 INFO:teuthology.orchestra.run.smithi107.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:32:52.791 INFO:teuthology.orchestra.run.smithi107.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:32:52.791 INFO:teuthology.orchestra.run.smithi107.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:32:52.791 INFO:teuthology.orchestra.run.smithi107.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:32:52.791 INFO:teuthology.orchestra.run.smithi107.stderr: 2023-09-30T01:32:52.791 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.769+0000 7fbddeffd700 10 monclient: _finish_auth 0 2023-09-30T01:32:52.791 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.769+0000 7fbddeffd700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:32:52.791 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.769+0000 7fbddeffd700 1 -- 172.21.15.107:0/4270609384 <== mon.5 v2:172.21.15.107:3302/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbdd8007400 con 0x7fbde006ef60 2023-09-30T01:32:52.791 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.769+0000 7fbde4959700 5 monclient: authenticate success, global_id 4172 2023-09-30T01:32:52.791 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.769+0000 7fbde4959700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:32:52.791 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.769+0000 7fbddeffd700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:32:52.791 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.769+0000 7fbde4959700 10 monclient: get_monmap_and_config success 2023-09-30T01:32:52.791 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.769+0000 7fbddeffd700 1 -- 172.21.15.107:0/4270609384 <== mon.5 v2:172.21.15.107:3302/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7fbdd80092e0 con 0x7fbde006ef60 2023-09-30T01:32:52.792 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.769+0000 7fbddeffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:32:52.792 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.769+0000 7fbde4959700 4 set_mon_vals no callback set 2023-09-30T01:32:52.792 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.769+0000 7fbddeffd700 10 monclient: got monmap 1 from mon.h (according to old e1) 2023-09-30T01:32:52.792 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.769+0000 7fbddeffd700 10 monclient: dump: 2023-09-30T01:32:52.792 INFO:teuthology.orchestra.run.smithi107.stderr:epoch 1 2023-09-30T01:32:52.792 INFO:teuthology.orchestra.run.smithi107.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:32:52.792 INFO:teuthology.orchestra.run.smithi107.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:32:52.792 INFO:teuthology.orchestra.run.smithi107.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:32:52.792 INFO:teuthology.orchestra.run.smithi107.stderr:min_mon_release 17 (quincy) 2023-09-30T01:32:52.792 INFO:teuthology.orchestra.run.smithi107.stderr:election_strategy: 3 2023-09-30T01:32:52.792 INFO:teuthology.orchestra.run.smithi107.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:32:52.792 INFO:teuthology.orchestra.run.smithi107.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:32:52.793 INFO:teuthology.orchestra.run.smithi107.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:32:52.793 INFO:teuthology.orchestra.run.smithi107.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:32:52.793 INFO:teuthology.orchestra.run.smithi107.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:32:52.793 INFO:teuthology.orchestra.run.smithi107.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:32:52.793 INFO:teuthology.orchestra.run.smithi107.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:32:52.793 INFO:teuthology.orchestra.run.smithi107.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:32:52.793 INFO:teuthology.orchestra.run.smithi107.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:32:52.793 INFO:teuthology.orchestra.run.smithi107.stderr: 2023-09-30T01:32:52.793 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.769+0000 7fbde4959700 10 monclient: shutdown 2023-09-30T01:32:52.793 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.769+0000 7fbde4959700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:32:52.793 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.769+0000 7fbde4959700 1 -- 172.21.15.107:0/4270609384 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7fbde006ef60 msgr2=0x7fbde009edf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:32:52.793 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.769+0000 7fbde4959700 1 --2- 172.21.15.107:0/4270609384 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7fbde006ef60 0x7fbde009edf0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fbdd8009f20 tx=0x7fbdd8009840 comp rx=0 tx=0).stop 2023-09-30T01:32:52.793 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.769+0000 7fbde4959700 1 -- 172.21.15.107:0/4270609384 shutdown_connections 2023-09-30T01:32:52.794 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.769+0000 7fbde4959700 1 --2- 172.21.15.107:0/4270609384 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fbde0131b20 0x7fbde006ea20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:32:52.794 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.769+0000 7fbde4959700 1 --2- 172.21.15.107:0/4270609384 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7fbde006ef60 0x7fbde009edf0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:32:52.794 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.769+0000 7fbde4959700 1 --2- 172.21.15.107:0/4270609384 >> v2:172.21.15.88:3300/0 conn(0x7fbde00a17d0 0x7fbde01315e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:32:52.794 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.769+0000 7fbde4959700 1 -- 172.21.15.107:0/4270609384 >> 172.21.15.107:0/4270609384 conn(0x7fbde0097a40 msgr2=0x7fbde00a0ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:32:52.794 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.769+0000 7fbde4959700 1 -- 172.21.15.107:0/4270609384 shutdown_connections 2023-09-30T01:32:52.794 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.769+0000 7fbde4959700 1 -- 172.21.15.107:0/4270609384 wait complete. 2023-09-30T01:32:52.794 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.769+0000 7fbde4959700 10 monclient: build_initial_monmap 2023-09-30T01:32:52.794 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.769+0000 7fbde4959700 10 monclient: monmap: 2023-09-30T01:32:52.794 INFO:teuthology.orchestra.run.smithi107.stderr:epoch 0 2023-09-30T01:32:52.794 INFO:teuthology.orchestra.run.smithi107.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:32:52.794 INFO:teuthology.orchestra.run.smithi107.stderr:last_changed 0.000000 2023-09-30T01:32:52.794 INFO:teuthology.orchestra.run.smithi107.stderr:created 0.000000 2023-09-30T01:32:52.795 INFO:teuthology.orchestra.run.smithi107.stderr:min_mon_release 0 (unknown) 2023-09-30T01:32:52.795 INFO:teuthology.orchestra.run.smithi107.stderr:election_strategy: 1 2023-09-30T01:32:52.795 INFO:teuthology.orchestra.run.smithi107.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.noname-a 2023-09-30T01:32:52.795 INFO:teuthology.orchestra.run.smithi107.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.noname-f 2023-09-30T01:32:52.796 INFO:teuthology.orchestra.run.smithi107.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.noname-b 2023-09-30T01:32:52.796 INFO:teuthology.orchestra.run.smithi107.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.noname-g 2023-09-30T01:32:52.796 INFO:teuthology.orchestra.run.smithi107.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.noname-c 2023-09-30T01:32:52.796 INFO:teuthology.orchestra.run.smithi107.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.noname-h 2023-09-30T01:32:52.796 INFO:teuthology.orchestra.run.smithi107.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.noname-d 2023-09-30T01:32:52.796 INFO:teuthology.orchestra.run.smithi107.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.noname-i 2023-09-30T01:32:52.796 INFO:teuthology.orchestra.run.smithi107.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.noname-e 2023-09-30T01:32:52.796 INFO:teuthology.orchestra.run.smithi107.stderr: 2023-09-30T01:32:52.796 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.773+0000 7fbde4959700 1 Processor -- start 2023-09-30T01:32:52.797 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.773+0000 7fbde4959700 1 -- start start 2023-09-30T01:32:52.797 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.773+0000 7fbde4959700 10 monclient: init 2023-09-30T01:32:52.797 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.773+0000 7fbde4959700 10 monclient: _reopen_session rank -1 2023-09-30T01:32:52.797 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.773+0000 7fbde4959700 10 monclient: _add_conns ranks=[3,4,7,8,1,5,6,0,2] 2023-09-30T01:32:52.797 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.773+0000 7fbde4959700 1 --2- >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7fbde006ef60 0x7fbde012aac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:32:52.797 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.773+0000 7fbde4959700 10 monclient(hunting): picked mon.noname-g con 0x7fbde006ef60 addr [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] 2023-09-30T01:32:52.798 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.773+0000 7fbde4959700 1 --2- >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7fbde012b000 0x7fbde012c120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:32:52.798 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.773+0000 7fbde4959700 10 monclient(hunting): picked mon.noname-c con 0x7fbde012b000 addr [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] 2023-09-30T01:32:52.798 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.773+0000 7fbde4959700 1 --2- >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7fbde012c660 0x7fbde012cad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:32:52.798 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.773+0000 7fbde4959700 10 monclient(hunting): picked mon.noname-i con 0x7fbde012c660 addr [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] 2023-09-30T01:32:52.798 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.773+0000 7fbde4959700 10 monclient(hunting): start opening mon connection 2023-09-30T01:32:52.798 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.773+0000 7fbde4959700 1 -- --> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbde012b3d0 con 0x7fbde006ef60 2023-09-30T01:32:52.799 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.773+0000 7fbde4959700 10 monclient(hunting): start opening mon connection 2023-09-30T01:32:52.799 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.773+0000 7fbde4959700 1 -- --> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] -- mon_getmap magic: 0 v1 -- 0x7fbde011b260 con 0x7fbde012b000 2023-09-30T01:32:52.799 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.773+0000 7fbde4959700 10 monclient(hunting): start opening mon connection 2023-09-30T01:32:52.799 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.773+0000 7fbde4959700 1 -- --> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] -- mon_getmap magic: 0 v1 -- 0x7fbde012bc30 con 0x7fbde012c660 2023-09-30T01:32:52.799 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.773+0000 7fbde4959700 10 monclient(hunting): _renew_subs 2023-09-30T01:32:52.799 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.773+0000 7fbde4959700 10 monclient(hunting): authenticate will time out at 856.197343s 2023-09-30T01:32:52.799 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.773+0000 7fbddffff700 1 --2- >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7fbde012c660 0x7fbde012cad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:32:52.799 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.773+0000 7fbddf7fe700 1 --2- >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7fbde006ef60 0x7fbde012aac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:32:52.799 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.773+0000 7fbddffff700 1 --2- >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7fbde012c660 0x7fbde012cad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3303/0 says I am v2:172.21.15.107:60552/0 (socket says 172.21.15.107:60552) 2023-09-30T01:32:52.799 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.773+0000 7fbddffff700 1 -- 172.21.15.107:0/2742798598 learned_addr learned my addr 172.21.15.107:0/2742798598 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-09-30T01:32:52.799 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.773+0000 7fbddffff700 10 monclient(hunting): get_auth_request con 0x7fbde012c660 auth_method 0 2023-09-30T01:32:52.799 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.773+0000 7fbddffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:32:52.799 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.773+0000 7fbddffff700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:32:52.800 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.773+0000 7fbddffff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:32:52.800 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.773+0000 7fbddf7fe700 10 monclient(hunting): get_auth_request con 0x7fbde006ef60 auth_method 0 2023-09-30T01:32:52.800 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.773+0000 7fbddf7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:32:52.800 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.773+0000 7fbddf7fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:32:52.800 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.773+0000 7fbddf7fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:32:52.800 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.773+0000 7fbdd6ffd700 1 --2- 172.21.15.107:0/2742798598 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7fbde012b000 0x7fbde012c120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:32:52.800 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.773+0000 7fbddffff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:32:52.800 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.773+0000 7fbddffff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:32:52.800 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.773+0000 7fbddffff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:32:52.800 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.773+0000 7fbdd6ffd700 10 monclient(hunting): get_auth_request con 0x7fbde012b000 auth_method 0 2023-09-30T01:32:52.801 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.773+0000 7fbdd6ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:32:52.801 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.773+0000 7fbdd6ffd700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:32:52.801 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.773+0000 7fbdd6ffd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:32:52.801 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.773+0000 7fbddf7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:32:52.801 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.773+0000 7fbddf7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:32:52.801 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.773+0000 7fbddf7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:32:52.801 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.773+0000 7fbdd6ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:32:52.801 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.773+0000 7fbdd6ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:32:52.802 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.773+0000 7fbdd6ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:32:52.802 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.773+0000 7fbddffff700 10 monclient(hunting): handle_auth_done global_id 4120 payload 824 2023-09-30T01:32:52.802 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.773+0000 7fbddffff700 1 -- 172.21.15.107:0/2742798598 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7fbde012b000 msgr2=0x7fbde012c120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:32:52.802 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.773+0000 7fbddffff700 1 --2- 172.21.15.107:0/2742798598 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7fbde012b000 0x7fbde012c120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:32:52.802 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.773+0000 7fbddffff700 1 -- 172.21.15.107:0/2742798598 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7fbde006ef60 msgr2=0x7fbde012aac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:32:52.802 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.773+0000 7fbddffff700 1 --2- 172.21.15.107:0/2742798598 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7fbde006ef60 0x7fbde012aac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:32:52.802 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.773+0000 7fbddffff700 10 monclient: _finish_hunting 0 2023-09-30T01:32:52.802 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.773+0000 7fbddffff700 1 monclient: found mon.noname-i 2023-09-30T01:32:52.802 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.773+0000 7fbddffff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:32:52.802 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.773+0000 7fbddffff700 10 monclient: _send_mon_message to mon.noname-i at v2:172.21.15.107:3303/0 2023-09-30T01:32:52.802 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.773+0000 7fbddffff700 1 -- 172.21.15.107:0/2742798598 --> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbdd8009ae0 con 0x7fbde012c660 2023-09-30T01:32:52.803 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.773+0000 7fbddffff700 1 --2- 172.21.15.107:0/2742798598 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7fbde012c660 0x7fbde012cad0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fbdd800b6c0 tx=0x7fbdd8018000 comp rx=0 tx=0).ready entity=mon.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:32:52.803 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.773+0000 7fbdd7fff700 1 -- 172.21.15.107:0/2742798598 <== mon.7 v2:172.21.15.107:3303/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7fbdd8007040 con 0x7fbde012c660 2023-09-30T01:32:52.803 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.773+0000 7fbdd7fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:32:52.803 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.773+0000 7fbdd7fff700 10 monclient: got monmap 1 from mon.noname-i (according to old e1) 2023-09-30T01:32:52.803 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.773+0000 7fbdd7fff700 10 monclient: dump: 2023-09-30T01:32:52.803 INFO:teuthology.orchestra.run.smithi107.stderr:epoch 1 2023-09-30T01:32:52.803 INFO:teuthology.orchestra.run.smithi107.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:32:52.803 INFO:teuthology.orchestra.run.smithi107.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:32:52.803 INFO:teuthology.orchestra.run.smithi107.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:32:52.803 INFO:teuthology.orchestra.run.smithi107.stderr:min_mon_release 17 (quincy) 2023-09-30T01:32:52.803 INFO:teuthology.orchestra.run.smithi107.stderr:election_strategy: 3 2023-09-30T01:32:52.803 INFO:teuthology.orchestra.run.smithi107.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:32:52.804 INFO:teuthology.orchestra.run.smithi107.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:32:52.804 INFO:teuthology.orchestra.run.smithi107.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:32:52.804 INFO:teuthology.orchestra.run.smithi107.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:32:52.804 INFO:teuthology.orchestra.run.smithi107.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:32:52.804 INFO:teuthology.orchestra.run.smithi107.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:32:52.804 INFO:teuthology.orchestra.run.smithi107.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:32:52.804 INFO:teuthology.orchestra.run.smithi107.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:32:52.804 INFO:teuthology.orchestra.run.smithi107.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:32:52.804 INFO:teuthology.orchestra.run.smithi107.stderr: 2023-09-30T01:32:52.804 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.773+0000 7fbdd7fff700 10 monclient: _finish_auth 0 2023-09-30T01:32:52.804 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.773+0000 7fbdd7fff700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:32:52.804 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.773+0000 7fbdd7fff700 1 -- 172.21.15.107:0/2742798598 <== mon.7 v2:172.21.15.107:3303/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbdd8007c00 con 0x7fbde012c660 2023-09-30T01:32:52.805 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.773+0000 7fbde4959700 5 monclient: authenticate success, global_id 4120 2023-09-30T01:32:52.805 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.773+0000 7fbde4959700 10 monclient: _renew_subs 2023-09-30T01:32:52.805 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.773+0000 7fbde4959700 10 monclient: _send_mon_message to mon.i at v2:172.21.15.107:3303/0 2023-09-30T01:32:52.805 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.773+0000 7fbde4959700 1 -- 172.21.15.107:0/2742798598 --> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbde006c840 con 0x7fbde012c660 2023-09-30T01:32:52.805 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.773+0000 7fbdd7fff700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:32:52.805 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.773+0000 7fbdd7fff700 1 -- 172.21.15.107:0/2742798598 <== mon.7 v2:172.21.15.107:3303/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7fbdd8007d70 con 0x7fbde012c660 2023-09-30T01:32:52.805 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.773+0000 7fbdd7fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:32:52.805 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.773+0000 7fbdd7fff700 10 monclient: got monmap 1 from mon.i (according to old e1) 2023-09-30T01:32:52.805 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.773+0000 7fbdd7fff700 10 monclient: dump: 2023-09-30T01:32:52.805 INFO:teuthology.orchestra.run.smithi107.stderr:epoch 1 2023-09-30T01:32:52.805 INFO:teuthology.orchestra.run.smithi107.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:32:52.805 INFO:teuthology.orchestra.run.smithi107.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:32:52.805 INFO:teuthology.orchestra.run.smithi107.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:32:52.806 INFO:teuthology.orchestra.run.smithi107.stderr:min_mon_release 17 (quincy) 2023-09-30T01:32:52.806 INFO:teuthology.orchestra.run.smithi107.stderr:election_strategy: 3 2023-09-30T01:32:52.806 INFO:teuthology.orchestra.run.smithi107.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:32:52.806 INFO:teuthology.orchestra.run.smithi107.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:32:52.806 INFO:teuthology.orchestra.run.smithi107.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:32:52.806 INFO:teuthology.orchestra.run.smithi107.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:32:52.806 INFO:teuthology.orchestra.run.smithi107.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:32:52.806 INFO:teuthology.orchestra.run.smithi107.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:32:52.806 INFO:teuthology.orchestra.run.smithi107.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:32:52.806 INFO:teuthology.orchestra.run.smithi107.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:32:52.806 INFO:teuthology.orchestra.run.smithi107.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:32:52.806 INFO:teuthology.orchestra.run.smithi107.stderr: 2023-09-30T01:32:52.807 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.773+0000 7fbdddffb700 4 set_mon_vals no callback set 2023-09-30T01:32:52.807 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.773+0000 7fbde4959700 10 monclient: _renew_subs 2023-09-30T01:32:52.807 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.773+0000 7fbde4959700 10 monclient: _send_mon_message to mon.i at v2:172.21.15.107:3303/0 2023-09-30T01:32:52.807 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.773+0000 7fbde4959700 1 -- 172.21.15.107:0/2742798598 --> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbde0132710 con 0x7fbde012c660 2023-09-30T01:32:52.807 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.773+0000 7fbdd7fff700 1 -- 172.21.15.107:0/2742798598 <== mon.7 v2:172.21.15.107:3303/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7fbdd8007780 con 0x7fbde012c660 2023-09-30T01:32:52.807 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.773+0000 7fbdd7fff700 1 -- 172.21.15.107:0/2742798598 <== mon.7 v2:172.21.15.107:3303/0 5 ==== osd_map(6..6 src has 1..6) v4 ==== 1181+0+0 (secure 0 0 0) 0x7fbdd8026ce0 con 0x7fbde012c660 2023-09-30T01:32:52.807 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.773+0000 7fbde4959700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:32:52.807 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.773+0000 7fbde4959700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:32:52.807 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.773+0000 7fbde4959700 10 monclient: _send_mon_message to mon.i at v2:172.21.15.107:3303/0 2023-09-30T01:32:52.807 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.773+0000 7fbde4959700 1 -- 172.21.15.107:0/2742798598 --> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbde0071960 con 0x7fbde012c660 2023-09-30T01:32:52.807 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.777+0000 7fbdd7fff700 1 -- 172.21.15.107:0/2742798598 <== mon.7 v2:172.21.15.107:3303/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7fbdd8021070 con 0x7fbde012c660 2023-09-30T01:32:52.807 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.777+0000 7fbdd7fff700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:32:52.807 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.777+0000 7fbdd7fff700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:32:52.821 INFO:tasks.ceph.mgr.x.smithi107.stderr:2023-09-30T01:32:52.817+0000 7f46b28dd000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-09-30T01:32:52.858 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.853+0000 7fbde4959700 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "c8c73bd1-7674-4529-9104-8690e25af2fc", "id": 4}] 2023-09-30T01:32:52.858 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.853+0000 7fbde4959700 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "c8c73bd1-7674-4529-9104-8690e25af2fc", "id": 4}] 2023-09-30T01:32:52.858 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.853+0000 7fbde4959700 10 monclient: _send_mon_message to mon.i at v2:172.21.15.107:3303/0 2023-09-30T01:32:52.858 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.853+0000 7fbde4959700 1 -- 172.21.15.107:0/2742798598 --> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] -- mon_command({"prefix": "osd new", "uuid": "c8c73bd1-7674-4529-9104-8690e25af2fc", "id": 4} v 0) v1 -- 0x7fbde012e800 con 0x7fbde012c660 2023-09-30T01:32:52.868 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.865+0000 7fbdd7fff700 1 -- 172.21.15.107:0/2742798598 <== mon.7 v2:172.21.15.107:3303/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "c8c73bd1-7674-4529-9104-8690e25af2fc", "id": 4}]=0 v7) v1 ==== 112+0+2 (secure 0 0 0) 0x7fbdd80267d0 con 0x7fbde012c660 2023-09-30T01:32:52.868 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.865+0000 7fbdd7fff700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "c8c73bd1-7674-4529-9104-8690e25af2fc", "id": 4}] 2023-09-30T01:32:52.868 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.865+0000 7fbdd7fff700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:32:52.868 INFO:teuthology.orchestra.run.smithi107.stdout:4 2023-09-30T01:32:52.870 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.865+0000 7fbde4959700 10 monclient: shutdown 2023-09-30T01:32:52.870 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.865+0000 7fbde4959700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:32:52.870 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.865+0000 7fbde4959700 1 -- 172.21.15.107:0/2742798598 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7fbde012c660 msgr2=0x7fbde012cad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:32:52.871 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.865+0000 7fbde4959700 1 --2- 172.21.15.107:0/2742798598 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7fbde012c660 0x7fbde012cad0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fbdd800b6c0 tx=0x7fbdd8018000 comp rx=0 tx=0).stop 2023-09-30T01:32:52.871 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.865+0000 7fbde4959700 1 -- 172.21.15.107:0/2742798598 shutdown_connections 2023-09-30T01:32:52.871 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.865+0000 7fbde4959700 1 --2- 172.21.15.107:0/2742798598 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7fbde012c660 0x7fbde012cad0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:32:52.871 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.865+0000 7fbde4959700 1 --2- 172.21.15.107:0/2742798598 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7fbde012b000 0x7fbde012c120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:32:52.871 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.865+0000 7fbde4959700 1 --2- 172.21.15.107:0/2742798598 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7fbde006ef60 0x7fbde012aac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:32:52.871 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.865+0000 7fbde4959700 1 -- 172.21.15.107:0/2742798598 >> 172.21.15.107:0/2742798598 conn(0x7fbde0097a40 msgr2=0x7fbde009a2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:32:52.871 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.865+0000 7fbde4959700 1 -- 172.21.15.107:0/2742798598 shutdown_connections 2023-09-30T01:32:52.871 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.865+0000 7fbde4959700 1 -- 172.21.15.107:0/2742798598 wait complete. 2023-09-30T01:32:52.881 DEBUG:teuthology.orchestra.run.smithi107:> sudo ceph --cluster ceph osd new a3994780-9d6a-4bcd-b718-09f9d8774b3d 5 2023-09-30T01:32:52.982 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.977+0000 7f6c5404f700 10 monclient: get_monmap_and_config 2023-09-30T01:32:52.983 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.977+0000 7f6c5404f700 10 monclient: build_initial_monmap 2023-09-30T01:32:52.983 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.977+0000 7f6c5404f700 10 monclient: monmap: 2023-09-30T01:32:52.983 INFO:teuthology.orchestra.run.smithi107.stderr:epoch 0 2023-09-30T01:32:52.983 INFO:teuthology.orchestra.run.smithi107.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:32:52.983 INFO:teuthology.orchestra.run.smithi107.stderr:last_changed 2023-09-30T01:32:52.982780+0000 2023-09-30T01:32:52.983 INFO:teuthology.orchestra.run.smithi107.stderr:created 2023-09-30T01:32:52.982780+0000 2023-09-30T01:32:52.983 INFO:teuthology.orchestra.run.smithi107.stderr:min_mon_release 0 (unknown) 2023-09-30T01:32:52.983 INFO:teuthology.orchestra.run.smithi107.stderr:election_strategy: 3 2023-09-30T01:32:52.983 INFO:teuthology.orchestra.run.smithi107.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a-legacy 2023-09-30T01:32:52.983 INFO:teuthology.orchestra.run.smithi107.stderr:1: v1:172.21.15.107:6789/0 mon.noname-f-legacy 2023-09-30T01:32:52.983 INFO:teuthology.orchestra.run.smithi107.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.noname-b 2023-09-30T01:32:52.984 INFO:teuthology.orchestra.run.smithi107.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.noname-g 2023-09-30T01:32:52.984 INFO:teuthology.orchestra.run.smithi107.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.noname-c 2023-09-30T01:32:52.984 INFO:teuthology.orchestra.run.smithi107.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.noname-h 2023-09-30T01:32:52.984 INFO:teuthology.orchestra.run.smithi107.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.noname-d 2023-09-30T01:32:52.984 INFO:teuthology.orchestra.run.smithi107.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.noname-i 2023-09-30T01:32:52.984 INFO:teuthology.orchestra.run.smithi107.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.noname-e 2023-09-30T01:32:52.984 INFO:teuthology.orchestra.run.smithi107.stderr:9: v2:172.21.15.88:3300/0 mon.noname-a 2023-09-30T01:32:52.984 INFO:teuthology.orchestra.run.smithi107.stderr:10: v2:172.21.15.107:3300/0 mon.noname-f 2023-09-30T01:32:52.984 INFO:teuthology.orchestra.run.smithi107.stderr: 2023-09-30T01:32:52.994 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.977+0000 7f6c5404f700 1 Processor -- start 2023-09-30T01:32:52.994 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.977+0000 7f6c5404f700 1 -- start start 2023-09-30T01:32:52.995 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.977+0000 7f6c5404f700 10 monclient: init 2023-09-30T01:32:52.995 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.981+0000 7f6c5404f700 10 monclient: _reopen_session rank -1 2023-09-30T01:32:52.995 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.981+0000 7f6c5404f700 10 monclient: _add_conns ranks=[2,4,1,0,8,9,7,6,3,5,10] 2023-09-30T01:32:52.995 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.981+0000 7f6c5404f700 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6c4c0a1480 0x7f6c4c069670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:32:52.995 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.981+0000 7f6c5404f700 10 monclient(hunting): picked mon.noname-b con 0x7f6c4c0a1480 addr [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] 2023-09-30T01:32:52.995 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.981+0000 7f6c5404f700 1 --2- >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f6c4c133890 0x7f6c4c133ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:32:52.995 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.981+0000 7f6c5404f700 10 monclient(hunting): picked mon.noname-c con 0x7f6c4c133890 addr [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] 2023-09-30T01:32:52.995 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.981+0000 7f6c5404f700 10 monclient(hunting): picked mon.noname-f-legacy con 0x7f6c4c1341e0 addr v1:172.21.15.107:6789/0 2023-09-30T01:32:52.995 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.981+0000 7f6c5404f700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c4c05ecb0 con 0x7f6c4c1341e0 2023-09-30T01:32:52.995 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.981+0000 7f6c5404f700 10 monclient(hunting): start opening mon connection 2023-09-30T01:32:52.996 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.981+0000 7f6c5404f700 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c4c09e990 con 0x7f6c4c0a1480 2023-09-30T01:32:52.996 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.981+0000 7f6c5404f700 10 monclient(hunting): start opening mon connection 2023-09-30T01:32:52.996 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.981+0000 7f6c5404f700 1 -- --> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] -- mon_getmap magic: 0 v1 -- 0x7f6c4c0a0240 con 0x7f6c4c133890 2023-09-30T01:32:52.996 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.981+0000 7f6c5404f700 10 monclient(hunting): _renew_subs 2023-09-30T01:32:52.996 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.981+0000 7f6c5404f700 10 monclient(hunting): authenticate will time out at 856.404223s 2023-09-30T01:32:52.996 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.981+0000 7f6c525ec700 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f6c4c1341e0 0x7f6c4c09eef0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:59732/0 (socket says 172.21.15.107:59732) 2023-09-30T01:32:52.996 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.981+0000 7f6c525ec700 1 -- 172.21.15.107:0/4005582091 learned_addr learned my addr 172.21.15.107:0/4005582091 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-09-30T01:32:52.996 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.981+0000 7f6c51deb700 1 --2- 172.21.15.107:0/4005582091 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6c4c0a1480 0x7f6c4c069670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:32:52.996 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.981+0000 7f6c515ea700 1 --2- 172.21.15.107:0/4005582091 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f6c4c133890 0x7f6c4c133ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:32:52.997 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.981+0000 7f6c50de9700 1 -- 172.21.15.107:0/4005582091 <== mon.1 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3936539290 0 0) 0x7f6c4c05ecb0 con 0x7f6c4c1341e0 2023-09-30T01:32:52.997 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.981+0000 7f6c50de9700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:32:52.997 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.981+0000 7f6c50de9700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:32:52.997 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.981+0000 7f6c50de9700 10 monclient(hunting): my global_id is 4159 2023-09-30T01:32:52.997 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.981+0000 7f6c50de9700 1 -- 172.21.15.107:0/4005582091 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6c38003570 con 0x7f6c4c1341e0 2023-09-30T01:32:52.997 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.981+0000 7f6c515ea700 10 monclient(hunting): get_auth_request con 0x7f6c4c133890 auth_method 0 2023-09-30T01:32:52.997 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.981+0000 7f6c515ea700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:32:52.997 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.981+0000 7f6c515ea700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:32:52.997 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.981+0000 7f6c515ea700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:32:52.997 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.981+0000 7f6c51deb700 10 monclient(hunting): get_auth_request con 0x7f6c4c0a1480 auth_method 0 2023-09-30T01:32:52.997 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.981+0000 7f6c51deb700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:32:52.997 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.981+0000 7f6c51deb700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:32:52.997 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.981+0000 7f6c51deb700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:32:52.998 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.981+0000 7f6c515ea700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:32:52.998 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.981+0000 7f6c515ea700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:32:52.998 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.981+0000 7f6c515ea700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:32:52.998 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.981+0000 7f6c51deb700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:32:52.998 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.981+0000 7f6c51deb700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:32:52.998 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.981+0000 7f6c50de9700 1 -- 172.21.15.107:0/4005582091 <== mon.1 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2403615512 0 0) 0x7f6c38003570 con 0x7f6c4c1341e0 2023-09-30T01:32:52.998 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.981+0000 7f6c51deb700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:32:52.998 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.981+0000 7f6c50de9700 1 -- 172.21.15.107:0/4005582091 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f6c4c133890 msgr2=0x7f6c4c133ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:32:52.998 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.981+0000 7f6c50de9700 1 --2- 172.21.15.107:0/4005582091 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f6c4c133890 0x7f6c4c133ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:32:52.998 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.981+0000 7f6c50de9700 1 -- 172.21.15.107:0/4005582091 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6c4c0a1480 msgr2=0x7f6c4c069670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:32:52.998 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.981+0000 7f6c50de9700 1 --2- 172.21.15.107:0/4005582091 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6c4c0a1480 0x7f6c4c069670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:32:52.998 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.981+0000 7f6c50de9700 10 monclient: _finish_hunting 0 2023-09-30T01:32:52.999 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.981+0000 7f6c50de9700 1 monclient: found mon.noname-f-legacy 2023-09-30T01:32:52.999 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.981+0000 7f6c50de9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:32:52.999 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.981+0000 7f6c50de9700 10 monclient: _send_mon_message to mon.noname-f-legacy at v1:172.21.15.107:6789/0 2023-09-30T01:32:52.999 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.981+0000 7f6c50de9700 1 -- 172.21.15.107:0/4005582091 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c4c068600 con 0x7f6c4c1341e0 2023-09-30T01:32:52.999 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.981+0000 7f6c50de9700 10 monclient: _finish_auth 0 2023-09-30T01:32:52.999 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.981+0000 7f6c50de9700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:32:52.999 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.981+0000 7f6c50de9700 1 -- 172.21.15.107:0/4005582091 <== mon.1 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (unknown 176869241 0 0) 0x7f6c44001c70 con 0x7f6c4c1341e0 2023-09-30T01:32:52.999 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.981+0000 7f6c50de9700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:32:52.999 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.981+0000 7f6c50de9700 10 monclient: got monmap 1 from mon.noname-f-legacy (according to old e1) 2023-09-30T01:32:52.999 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.981+0000 7f6c50de9700 10 monclient: dump: 2023-09-30T01:32:52.999 INFO:teuthology.orchestra.run.smithi107.stderr:epoch 1 2023-09-30T01:32:52.999 INFO:teuthology.orchestra.run.smithi107.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:32:53.000 INFO:teuthology.orchestra.run.smithi107.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:32:53.000 INFO:teuthology.orchestra.run.smithi107.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:32:53.000 INFO:teuthology.orchestra.run.smithi107.stderr:min_mon_release 17 (quincy) 2023-09-30T01:32:53.000 INFO:teuthology.orchestra.run.smithi107.stderr:election_strategy: 3 2023-09-30T01:32:53.000 INFO:teuthology.orchestra.run.smithi107.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:32:53.000 INFO:teuthology.orchestra.run.smithi107.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:32:53.000 INFO:teuthology.orchestra.run.smithi107.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:32:53.000 INFO:teuthology.orchestra.run.smithi107.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:32:53.000 INFO:teuthology.orchestra.run.smithi107.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:32:53.000 INFO:teuthology.orchestra.run.smithi107.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:32:53.000 INFO:teuthology.orchestra.run.smithi107.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:32:53.000 INFO:teuthology.orchestra.run.smithi107.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:32:53.001 INFO:teuthology.orchestra.run.smithi107.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:32:53.001 INFO:teuthology.orchestra.run.smithi107.stderr: 2023-09-30T01:32:53.001 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.981+0000 7f6c50de9700 1 monclient: mon.f has (v2) addrs [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] but i'm connected to v1:172.21.15.107:6789/0, reconnecting 2023-09-30T01:32:53.001 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.981+0000 7f6c50de9700 10 monclient: _reopen_session rank -1 2023-09-30T01:32:53.001 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.981+0000 7f6c50de9700 1 -- 172.21.15.107:0/4005582091 >> v1:172.21.15.107:6789/0 conn(0x7f6c4c1341e0 legacy=0x7f6c4c09eef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:32:53.001 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.981+0000 7f6c50de9700 10 monclient: _add_conns ranks=[4,7,0,5,8,6,3,1,2] 2023-09-30T01:32:53.001 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.981+0000 7f6c50de9700 1 --2- 172.21.15.107:0/4005582091 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f6c4c133890 0x7f6c38004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:32:53.001 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.981+0000 7f6c50de9700 10 monclient(hunting): picked mon.c con 0x7f6c4c133890 addr [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] 2023-09-30T01:32:53.001 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.981+0000 7f6c50de9700 1 --2- 172.21.15.107:0/4005582091 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7f6c38004370 0x7f6c38004d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:32:53.001 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.981+0000 7f6c50de9700 10 monclient(hunting): picked mon.i con 0x7f6c38004370 addr [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] 2023-09-30T01:32:53.001 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.981+0000 7f6c50de9700 1 --2- 172.21.15.107:0/4005582091 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6c380053e0 0x7f6c3800baa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:32:53.001 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.981+0000 7f6c50de9700 10 monclient(hunting): picked mon.a con 0x7f6c380053e0 addr [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] 2023-09-30T01:32:53.001 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.981+0000 7f6c50de9700 10 monclient(hunting): start opening mon connection 2023-09-30T01:32:53.002 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.981+0000 7f6c50de9700 1 -- 172.21.15.107:0/4005582091 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c4c09e990 con 0x7f6c380053e0 2023-09-30T01:32:53.002 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.981+0000 7f6c50de9700 10 monclient(hunting): start opening mon connection 2023-09-30T01:32:53.002 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.981+0000 7f6c50de9700 1 -- 172.21.15.107:0/4005582091 --> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] -- mon_getmap magic: 0 v1 -- 0x7f6c4c0a0240 con 0x7f6c4c133890 2023-09-30T01:32:53.002 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.981+0000 7f6c50de9700 10 monclient(hunting): start opening mon connection 2023-09-30T01:32:53.002 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.981+0000 7f6c50de9700 1 -- 172.21.15.107:0/4005582091 --> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] -- mon_getmap magic: 0 v1 -- 0x7f6c3800c3b0 con 0x7f6c38004370 2023-09-30T01:32:53.002 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.981+0000 7f6c50de9700 10 monclient(hunting): _renew_subs 2023-09-30T01:32:53.002 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.981+0000 7f6c50de9700 10 monclient(hunting): _finish_auth 0 2023-09-30T01:32:53.002 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.981+0000 7f6c515ea700 1 --2- 172.21.15.107:0/4005582091 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7f6c38004370 0x7f6c38004d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:32:53.002 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.981+0000 7f6c515ea700 10 monclient(hunting): get_auth_request con 0x7f6c38004370 auth_method 0 2023-09-30T01:32:53.002 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.981+0000 7f6c515ea700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:32:53.002 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.981+0000 7f6c515ea700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:32:53.002 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.981+0000 7f6c515ea700 10 monclient(hunting): _init_auth already have auth, reseting 2023-09-30T01:32:53.003 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.981+0000 7f6c51deb700 1 --2- 172.21.15.107:0/4005582091 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f6c4c133890 0x7f6c38004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:32:53.003 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.981+0000 7f6c515ea700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:32:53.003 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.981+0000 7f6c515ea700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:32:53.003 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.981+0000 7f6c515ea700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-09-30T01:32:53.003 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.981+0000 7f6c51deb700 10 monclient(hunting): get_auth_request con 0x7f6c4c133890 auth_method 0 2023-09-30T01:32:53.003 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.981+0000 7f6c51deb700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:32:53.003 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.981+0000 7f6c51deb700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:32:53.003 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.981+0000 7f6c51deb700 10 monclient(hunting): _init_auth already have auth, reseting 2023-09-30T01:32:53.003 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.981+0000 7f6c525ec700 1 --2- 172.21.15.107:0/4005582091 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6c380053e0 0x7f6c3800baa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:32:53.003 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.981+0000 7f6c525ec700 10 monclient(hunting): get_auth_request con 0x7f6c380053e0 auth_method 0 2023-09-30T01:32:53.003 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.981+0000 7f6c525ec700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:32:53.004 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.981+0000 7f6c525ec700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:32:53.004 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.981+0000 7f6c525ec700 10 monclient(hunting): _init_auth already have auth, reseting 2023-09-30T01:32:53.004 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.981+0000 7f6c51deb700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:32:53.004 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.981+0000 7f6c51deb700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:32:53.004 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.981+0000 7f6c51deb700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-09-30T01:32:53.004 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.981+0000 7f6c515ea700 10 monclient(hunting): handle_auth_done global_id 4159 payload 293 2023-09-30T01:32:53.004 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.981+0000 7f6c515ea700 1 -- 172.21.15.107:0/4005582091 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f6c4c133890 msgr2=0x7f6c38004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:32:53.004 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.981+0000 7f6c515ea700 1 --2- 172.21.15.107:0/4005582091 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f6c4c133890 0x7f6c38004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:32:53.004 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.981+0000 7f6c515ea700 1 -- 172.21.15.107:0/4005582091 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6c380053e0 msgr2=0x7f6c3800baa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:32:53.004 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.981+0000 7f6c515ea700 1 --2- 172.21.15.107:0/4005582091 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6c380053e0 0x7f6c3800baa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:32:53.004 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.981+0000 7f6c515ea700 10 monclient: _finish_hunting 0 2023-09-30T01:32:53.004 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.981+0000 7f6c515ea700 1 monclient: found mon.i 2023-09-30T01:32:53.005 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.981+0000 7f6c515ea700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:32:53.005 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.981+0000 7f6c515ea700 10 monclient: _send_mon_message to mon.i at v2:172.21.15.107:3303/0 2023-09-30T01:32:53.005 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.981+0000 7f6c515ea700 1 -- 172.21.15.107:0/4005582091 --> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c38003570 con 0x7f6c38004370 2023-09-30T01:32:53.005 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.981+0000 7f6c515ea700 10 monclient: _finish_auth 0 2023-09-30T01:32:53.005 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.981+0000 7f6c515ea700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:32:53.005 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.981+0000 7f6c5404f700 5 monclient: authenticate success, global_id 4159 2023-09-30T01:32:53.005 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.981+0000 7f6c5404f700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:32:53.005 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.981+0000 7f6c525ec700 1 --2- 172.21.15.107:0/4005582091 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6c380053e0 0x7f6c3800baa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-30T01:32:53.005 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.981+0000 7f6c515ea700 1 --2- 172.21.15.107:0/4005582091 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7f6c38004370 0x7f6c38004d70 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f6c4c06bf50 tx=0x7f6c3c0091a0 comp rx=0 tx=0).ready entity=mon.7 client_cookie=f5427591e7637fea server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:32:53.005 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.981+0000 7f6c50de9700 1 -- 172.21.15.107:0/4005582091 <== mon.7 v2:172.21.15.107:3303/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f6c3c00eb90 con 0x7f6c38004370 2023-09-30T01:32:53.005 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.981+0000 7f6c50de9700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:32:53.005 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.981+0000 7f6c50de9700 10 monclient: got monmap 1 from mon.i (according to old e1) 2023-09-30T01:32:53.005 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.981+0000 7f6c50de9700 10 monclient: dump: 2023-09-30T01:32:53.006 INFO:teuthology.orchestra.run.smithi107.stderr:epoch 1 2023-09-30T01:32:53.006 INFO:teuthology.orchestra.run.smithi107.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:32:53.006 INFO:teuthology.orchestra.run.smithi107.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:32:53.006 INFO:teuthology.orchestra.run.smithi107.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:32:53.006 INFO:teuthology.orchestra.run.smithi107.stderr:min_mon_release 17 (quincy) 2023-09-30T01:32:53.006 INFO:teuthology.orchestra.run.smithi107.stderr:election_strategy: 3 2023-09-30T01:32:53.006 INFO:teuthology.orchestra.run.smithi107.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:32:53.006 INFO:teuthology.orchestra.run.smithi107.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:32:53.006 INFO:teuthology.orchestra.run.smithi107.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:32:53.006 INFO:teuthology.orchestra.run.smithi107.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:32:53.006 INFO:teuthology.orchestra.run.smithi107.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:32:53.006 INFO:teuthology.orchestra.run.smithi107.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:32:53.006 INFO:teuthology.orchestra.run.smithi107.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:32:53.007 INFO:teuthology.orchestra.run.smithi107.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:32:53.007 INFO:teuthology.orchestra.run.smithi107.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:32:53.007 INFO:teuthology.orchestra.run.smithi107.stderr: 2023-09-30T01:32:53.007 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.981+0000 7f6c5404f700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:32:53.007 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.981+0000 7f6c50de9700 1 -- 172.21.15.107:0/4005582091 <== mon.7 v2:172.21.15.107:3303/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c3c00ed10 con 0x7f6c38004370 2023-09-30T01:32:53.007 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.981+0000 7f6c50de9700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:32:53.007 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.981+0000 7f6c5404f700 10 monclient: get_monmap_and_config success 2023-09-30T01:32:53.007 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.981+0000 7f6c5404f700 4 set_mon_vals no callback set 2023-09-30T01:32:53.007 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.981+0000 7f6c50de9700 1 -- 172.21.15.107:0/4005582091 <== mon.7 v2:172.21.15.107:3303/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f6c3c00ee80 con 0x7f6c38004370 2023-09-30T01:32:53.007 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.981+0000 7f6c50de9700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:32:53.007 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.981+0000 7f6c50de9700 10 monclient: got monmap 1 from mon.i (according to old e1) 2023-09-30T01:32:53.007 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.981+0000 7f6c50de9700 10 monclient: dump: 2023-09-30T01:32:53.007 INFO:teuthology.orchestra.run.smithi107.stderr:epoch 1 2023-09-30T01:32:53.007 INFO:teuthology.orchestra.run.smithi107.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:32:53.008 INFO:teuthology.orchestra.run.smithi107.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:32:53.008 INFO:teuthology.orchestra.run.smithi107.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:32:53.008 INFO:teuthology.orchestra.run.smithi107.stderr:min_mon_release 17 (quincy) 2023-09-30T01:32:53.008 INFO:teuthology.orchestra.run.smithi107.stderr:election_strategy: 3 2023-09-30T01:32:53.008 INFO:teuthology.orchestra.run.smithi107.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:32:53.008 INFO:teuthology.orchestra.run.smithi107.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:32:53.008 INFO:teuthology.orchestra.run.smithi107.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:32:53.009 INFO:teuthology.orchestra.run.smithi107.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:32:53.009 INFO:teuthology.orchestra.run.smithi107.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:32:53.009 INFO:teuthology.orchestra.run.smithi107.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:32:53.009 INFO:teuthology.orchestra.run.smithi107.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:32:53.009 INFO:teuthology.orchestra.run.smithi107.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:32:53.009 INFO:teuthology.orchestra.run.smithi107.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:32:53.009 INFO:teuthology.orchestra.run.smithi107.stderr: 2023-09-30T01:32:53.009 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.981+0000 7f6c5404f700 10 monclient: shutdown 2023-09-30T01:32:53.009 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.981+0000 7f6c5404f700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:32:53.009 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.981+0000 7f6c5404f700 1 -- 172.21.15.107:0/4005582091 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7f6c38004370 msgr2=0x7f6c38004d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:32:53.010 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.981+0000 7f6c5404f700 1 --2- 172.21.15.107:0/4005582091 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7f6c38004370 0x7f6c38004d70 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f6c4c06bf50 tx=0x7f6c3c0091a0 comp rx=0 tx=0).stop 2023-09-30T01:32:53.010 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.981+0000 7f6c525ec700 1 -- 172.21.15.107:0/4005582091 reap_dead start 2023-09-30T01:32:53.010 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.981+0000 7f6c5404f700 1 -- 172.21.15.107:0/4005582091 shutdown_connections 2023-09-30T01:32:53.010 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.981+0000 7f6c5404f700 1 -- 172.21.15.107:0/4005582091 >> 172.21.15.107:0/4005582091 conn(0x7f6c4c097a40 msgr2=0x7f6c4c063450 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:32:53.010 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.981+0000 7f6c5404f700 1 -- 172.21.15.107:0/4005582091 shutdown_connections 2023-09-30T01:32:53.010 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.981+0000 7f6c5404f700 1 -- 172.21.15.107:0/4005582091 wait complete. 2023-09-30T01:32:53.010 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.981+0000 7f6c5404f700 10 monclient: build_initial_monmap 2023-09-30T01:32:53.010 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.981+0000 7f6c5404f700 10 monclient: monmap: 2023-09-30T01:32:53.010 INFO:teuthology.orchestra.run.smithi107.stderr:epoch 0 2023-09-30T01:32:53.010 INFO:teuthology.orchestra.run.smithi107.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:32:53.010 INFO:teuthology.orchestra.run.smithi107.stderr:last_changed 0.000000 2023-09-30T01:32:53.010 INFO:teuthology.orchestra.run.smithi107.stderr:created 0.000000 2023-09-30T01:32:53.011 INFO:teuthology.orchestra.run.smithi107.stderr:min_mon_release 0 (unknown) 2023-09-30T01:32:53.011 INFO:teuthology.orchestra.run.smithi107.stderr:election_strategy: 1 2023-09-30T01:32:53.011 INFO:teuthology.orchestra.run.smithi107.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.noname-a 2023-09-30T01:32:53.011 INFO:teuthology.orchestra.run.smithi107.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.noname-f 2023-09-30T01:32:53.011 INFO:teuthology.orchestra.run.smithi107.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.noname-b 2023-09-30T01:32:53.011 INFO:teuthology.orchestra.run.smithi107.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.noname-g 2023-09-30T01:32:53.011 INFO:teuthology.orchestra.run.smithi107.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.noname-c 2023-09-30T01:32:53.011 INFO:teuthology.orchestra.run.smithi107.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.noname-h 2023-09-30T01:32:53.011 INFO:teuthology.orchestra.run.smithi107.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.noname-d 2023-09-30T01:32:53.011 INFO:teuthology.orchestra.run.smithi107.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.noname-i 2023-09-30T01:32:53.011 INFO:teuthology.orchestra.run.smithi107.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.noname-e 2023-09-30T01:32:53.011 INFO:teuthology.orchestra.run.smithi107.stderr: 2023-09-30T01:32:53.011 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.985+0000 7f6c5404f700 1 Processor -- start 2023-09-30T01:32:53.012 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.985+0000 7f6c5404f700 1 -- start start 2023-09-30T01:32:53.012 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.985+0000 7f6c5404f700 10 monclient: init 2023-09-30T01:32:53.012 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.985+0000 7f6c5404f700 10 monclient: _reopen_session rank -1 2023-09-30T01:32:53.012 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.985+0000 7f6c5404f700 10 monclient: _add_conns ranks=[4,3,2,8,6,0,7,1,5] 2023-09-30T01:32:53.012 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.985+0000 7f6c5404f700 1 --2- >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f6c38004370 0x7f6c4c09d400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:32:53.012 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.985+0000 7f6c5404f700 10 monclient(hunting): picked mon.noname-c con 0x7f6c38004370 addr [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] 2023-09-30T01:32:53.012 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.985+0000 7f6c5404f700 1 --2- >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7f6c4c1426e0 0x7f6c4c142ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:32:53.012 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.985+0000 7f6c5404f700 10 monclient(hunting): picked mon.noname-g con 0x7f6c4c1426e0 addr [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] 2023-09-30T01:32:53.012 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.985+0000 7f6c5404f700 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6c4c142ff0 0x7f6c4c11f050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:32:53.012 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.985+0000 7f6c5404f700 10 monclient(hunting): picked mon.noname-b con 0x7f6c4c142ff0 addr [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] 2023-09-30T01:32:53.012 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.985+0000 7f6c5404f700 10 monclient(hunting): start opening mon connection 2023-09-30T01:32:53.012 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.985+0000 7f6c5404f700 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c4c1217e0 con 0x7f6c4c142ff0 2023-09-30T01:32:53.013 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.985+0000 7f6c5404f700 10 monclient(hunting): start opening mon connection 2023-09-30T01:32:53.013 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.985+0000 7f6c5404f700 1 -- --> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c4c0a0950 con 0x7f6c4c1426e0 2023-09-30T01:32:53.013 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.985+0000 7f6c5404f700 10 monclient(hunting): start opening mon connection 2023-09-30T01:32:53.013 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.985+0000 7f6c5404f700 1 -- --> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] -- mon_getmap magic: 0 v1 -- 0x7f6c4c069ed0 con 0x7f6c38004370 2023-09-30T01:32:53.013 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.985+0000 7f6c5404f700 10 monclient(hunting): _renew_subs 2023-09-30T01:32:53.013 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.985+0000 7f6c5404f700 10 monclient(hunting): authenticate will time out at 856.408684s 2023-09-30T01:32:53.013 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.985+0000 7f6c515ea700 1 --2- >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7f6c4c1426e0 0x7f6c4c142ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:32:53.013 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.985+0000 7f6c515ea700 1 --2- >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7f6c4c1426e0 0x7f6c4c142ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3301/0 says I am v2:172.21.15.107:39496/0 (socket says 172.21.15.107:39496) 2023-09-30T01:32:53.013 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.985+0000 7f6c515ea700 1 -- 172.21.15.107:0/819149134 learned_addr learned my addr 172.21.15.107:0/819149134 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-09-30T01:32:53.013 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.985+0000 7f6c515ea700 10 monclient(hunting): get_auth_request con 0x7f6c4c1426e0 auth_method 0 2023-09-30T01:32:53.013 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.985+0000 7f6c515ea700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:32:53.014 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.985+0000 7f6c515ea700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:32:53.014 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.985+0000 7f6c515ea700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:32:53.014 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.985+0000 7f6c515ea700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:32:53.014 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.985+0000 7f6c515ea700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:32:53.014 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.985+0000 7f6c515ea700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:32:53.014 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.985+0000 7f6c51deb700 1 --2- 172.21.15.107:0/819149134 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f6c38004370 0x7f6c4c09d400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:32:53.014 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.985+0000 7f6c51deb700 10 monclient(hunting): get_auth_request con 0x7f6c38004370 auth_method 0 2023-09-30T01:32:53.014 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.985+0000 7f6c51deb700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:32:53.014 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.985+0000 7f6c51deb700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:32:53.014 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.985+0000 7f6c51deb700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:32:53.014 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.985+0000 7f6c515ea700 10 monclient(hunting): handle_auth_done global_id 4134 payload 824 2023-09-30T01:32:53.014 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.985+0000 7f6c525ec700 1 --2- 172.21.15.107:0/819149134 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6c4c142ff0 0x7f6c4c11f050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:32:53.015 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.985+0000 7f6c515ea700 1 -- 172.21.15.107:0/819149134 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f6c38004370 msgr2=0x7f6c4c09d400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:32:53.015 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.985+0000 7f6c515ea700 1 --2- 172.21.15.107:0/819149134 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f6c38004370 0x7f6c4c09d400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:32:53.015 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.985+0000 7f6c515ea700 1 -- 172.21.15.107:0/819149134 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6c4c142ff0 msgr2=0x7f6c4c11f050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:32:53.015 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.985+0000 7f6c515ea700 1 --2- 172.21.15.107:0/819149134 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6c4c142ff0 0x7f6c4c11f050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:32:53.015 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.985+0000 7f6c515ea700 10 monclient: _finish_hunting 0 2023-09-30T01:32:53.015 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.985+0000 7f6c515ea700 1 monclient: found mon.noname-g 2023-09-30T01:32:53.015 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.985+0000 7f6c515ea700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:32:53.015 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.985+0000 7f6c515ea700 10 monclient: _send_mon_message to mon.noname-g at v2:172.21.15.107:3301/0 2023-09-30T01:32:53.015 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.985+0000 7f6c515ea700 1 -- 172.21.15.107:0/819149134 --> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c4c122420 con 0x7f6c4c1426e0 2023-09-30T01:32:53.015 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.985+0000 7f6c525ec700 10 monclient: get_auth_request con 0x7f6c4c142ff0 auth_method 0 2023-09-30T01:32:53.015 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.985+0000 7f6c525ec700 1 --2- 172.21.15.107:0/819149134 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6c4c142ff0 0x7f6c4c11f050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-09-30T01:32:53.015 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.985+0000 7f6c515ea700 1 --2- 172.21.15.107:0/819149134 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7f6c4c1426e0 0x7f6c4c142ab0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f6c3c013a70 tx=0x7f6c3c01a000 comp rx=0 tx=0).ready entity=mon.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:32:53.015 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.985+0000 7f6c41ffb700 1 -- 172.21.15.107:0/819149134 <== mon.3 v2:172.21.15.107:3301/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f6c3c022070 con 0x7f6c4c1426e0 2023-09-30T01:32:53.016 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.985+0000 7f6c41ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:32:53.016 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.985+0000 7f6c41ffb700 10 monclient: got monmap 1 from mon.noname-g (according to old e1) 2023-09-30T01:32:53.016 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.985+0000 7f6c41ffb700 10 monclient: dump: 2023-09-30T01:32:53.016 INFO:teuthology.orchestra.run.smithi107.stderr:epoch 1 2023-09-30T01:32:53.016 INFO:teuthology.orchestra.run.smithi107.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:32:53.016 INFO:teuthology.orchestra.run.smithi107.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:32:53.016 INFO:teuthology.orchestra.run.smithi107.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:32:53.016 INFO:teuthology.orchestra.run.smithi107.stderr:min_mon_release 17 (quincy) 2023-09-30T01:32:53.016 INFO:teuthology.orchestra.run.smithi107.stderr:election_strategy: 3 2023-09-30T01:32:53.016 INFO:teuthology.orchestra.run.smithi107.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:32:53.016 INFO:teuthology.orchestra.run.smithi107.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:32:53.016 INFO:teuthology.orchestra.run.smithi107.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:32:53.016 INFO:teuthology.orchestra.run.smithi107.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:32:53.017 INFO:teuthology.orchestra.run.smithi107.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:32:53.017 INFO:teuthology.orchestra.run.smithi107.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:32:53.017 INFO:teuthology.orchestra.run.smithi107.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:32:53.017 INFO:teuthology.orchestra.run.smithi107.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:32:53.017 INFO:teuthology.orchestra.run.smithi107.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:32:53.017 INFO:teuthology.orchestra.run.smithi107.stderr: 2023-09-30T01:32:53.017 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.985+0000 7f6c41ffb700 10 monclient: _finish_auth 0 2023-09-30T01:32:53.017 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.985+0000 7f6c41ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:32:53.017 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.985+0000 7f6c5404f700 5 monclient: authenticate success, global_id 4134 2023-09-30T01:32:53.017 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.985+0000 7f6c41ffb700 1 -- 172.21.15.107:0/819149134 <== mon.3 v2:172.21.15.107:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c3c00ee80 con 0x7f6c4c1426e0 2023-09-30T01:32:53.017 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.985+0000 7f6c41ffb700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:32:53.017 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.985+0000 7f6c41ffb700 1 -- 172.21.15.107:0/819149134 <== mon.3 v2:172.21.15.107:3301/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f6c3c00eb10 con 0x7f6c4c1426e0 2023-09-30T01:32:53.018 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.985+0000 7f6c5404f700 10 monclient: _renew_subs 2023-09-30T01:32:53.018 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.985+0000 7f6c5404f700 10 monclient: _send_mon_message to mon.g at v2:172.21.15.107:3301/0 2023-09-30T01:32:53.018 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.985+0000 7f6c5404f700 1 -- 172.21.15.107:0/819149134 --> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c4c1202a0 con 0x7f6c4c1426e0 2023-09-30T01:32:53.018 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.985+0000 7f6c41ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:32:53.018 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.985+0000 7f6c43fff700 4 set_mon_vals no callback set 2023-09-30T01:32:53.018 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.985+0000 7f6c41ffb700 10 monclient: got monmap 1 from mon.g (according to old e1) 2023-09-30T01:32:53.018 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.985+0000 7f6c41ffb700 10 monclient: dump: 2023-09-30T01:32:53.018 INFO:teuthology.orchestra.run.smithi107.stderr:epoch 1 2023-09-30T01:32:53.018 INFO:teuthology.orchestra.run.smithi107.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:32:53.018 INFO:teuthology.orchestra.run.smithi107.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:32:53.018 INFO:teuthology.orchestra.run.smithi107.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:32:53.018 INFO:teuthology.orchestra.run.smithi107.stderr:min_mon_release 17 (quincy) 2023-09-30T01:32:53.018 INFO:teuthology.orchestra.run.smithi107.stderr:election_strategy: 3 2023-09-30T01:32:53.019 INFO:teuthology.orchestra.run.smithi107.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:32:53.019 INFO:teuthology.orchestra.run.smithi107.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:32:53.019 INFO:teuthology.orchestra.run.smithi107.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:32:53.019 INFO:teuthology.orchestra.run.smithi107.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:32:53.019 INFO:teuthology.orchestra.run.smithi107.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:32:53.019 INFO:teuthology.orchestra.run.smithi107.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:32:53.019 INFO:teuthology.orchestra.run.smithi107.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:32:53.019 INFO:teuthology.orchestra.run.smithi107.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:32:53.019 INFO:teuthology.orchestra.run.smithi107.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:32:53.019 INFO:teuthology.orchestra.run.smithi107.stderr: 2023-09-30T01:32:53.019 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.985+0000 7f6c5404f700 10 monclient: _renew_subs 2023-09-30T01:32:53.019 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.985+0000 7f6c5404f700 10 monclient: _send_mon_message to mon.g at v2:172.21.15.107:3301/0 2023-09-30T01:32:53.020 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.985+0000 7f6c5404f700 1 -- 172.21.15.107:0/819149134 --> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c4c143400 con 0x7f6c4c1426e0 2023-09-30T01:32:53.020 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.985+0000 7f6c41ffb700 1 -- 172.21.15.107:0/819149134 <== mon.3 v2:172.21.15.107:3301/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7f6c3c026070 con 0x7f6c4c1426e0 2023-09-30T01:32:53.020 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.985+0000 7f6c41ffb700 1 -- 172.21.15.107:0/819149134 <== mon.3 v2:172.21.15.107:3301/0 5 ==== osd_map(7..7 src has 1..7) v4 ==== 1296+0+0 (secure 0 0 0) 0x7f6c3c013db0 con 0x7f6c4c1426e0 2023-09-30T01:32:53.020 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.985+0000 7f6c5404f700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:32:53.020 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.985+0000 7f6c5404f700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:32:53.020 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.985+0000 7f6c5404f700 10 monclient: _send_mon_message to mon.g at v2:172.21.15.107:3301/0 2023-09-30T01:32:53.020 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.985+0000 7f6c5404f700 1 -- 172.21.15.107:0/819149134 --> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6c4c0a0950 con 0x7f6c4c1426e0 2023-09-30T01:32:53.020 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.989+0000 7f6c41ffb700 1 -- 172.21.15.107:0/819149134 <== mon.3 v2:172.21.15.107:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7f6c3c02d070 con 0x7f6c4c1426e0 2023-09-30T01:32:53.020 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.989+0000 7f6c41ffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:32:53.020 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:52.989+0000 7f6c41ffb700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:32:53.021 INFO:tasks.ceph.mgr.x.smithi107.stderr:2023-09-30T01:32:53.009+0000 7f46b28dd000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-09-30T01:32:53.065 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:53.061+0000 7f6c5404f700 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "a3994780-9d6a-4bcd-b718-09f9d8774b3d", "id": 5}] 2023-09-30T01:32:53.065 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:53.061+0000 7f6c5404f700 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "a3994780-9d6a-4bcd-b718-09f9d8774b3d", "id": 5}] 2023-09-30T01:32:53.065 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:53.061+0000 7f6c5404f700 10 monclient: _send_mon_message to mon.g at v2:172.21.15.107:3301/0 2023-09-30T01:32:53.065 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:53.061+0000 7f6c5404f700 1 -- 172.21.15.107:0/819149134 --> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] -- mon_command({"prefix": "osd new", "uuid": "a3994780-9d6a-4bcd-b718-09f9d8774b3d", "id": 5} v 0) v1 -- 0x7f6c4c06e8d0 con 0x7f6c4c1426e0 2023-09-30T01:32:53.077 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:53.073+0000 7f6c41ffb700 1 -- 172.21.15.107:0/819149134 <== mon.3 v2:172.21.15.107:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "a3994780-9d6a-4bcd-b718-09f9d8774b3d", "id": 5}]=0 v8) v1 ==== 112+0+2 (secure 0 0 0) 0x7f6c4c06e8d0 con 0x7f6c4c1426e0 2023-09-30T01:32:53.077 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:53.073+0000 7f6c41ffb700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "a3994780-9d6a-4bcd-b718-09f9d8774b3d", "id": 5}] 2023-09-30T01:32:53.077 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:53.073+0000 7f6c41ffb700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:32:53.077 INFO:teuthology.orchestra.run.smithi107.stdout:5 2023-09-30T01:32:53.078 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:53.073+0000 7f6c5404f700 10 monclient: shutdown 2023-09-30T01:32:53.078 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:53.073+0000 7f6c5404f700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:32:53.078 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:53.073+0000 7f6c5404f700 1 -- 172.21.15.107:0/819149134 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7f6c4c1426e0 msgr2=0x7f6c4c142ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:32:53.078 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:53.073+0000 7f6c5404f700 1 --2- 172.21.15.107:0/819149134 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7f6c4c1426e0 0x7f6c4c142ab0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f6c3c013a70 tx=0x7f6c3c01a000 comp rx=0 tx=0).stop 2023-09-30T01:32:53.078 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:53.073+0000 7f6c5404f700 1 -- 172.21.15.107:0/819149134 shutdown_connections 2023-09-30T01:32:53.079 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:53.073+0000 7f6c5404f700 1 --2- 172.21.15.107:0/819149134 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6c4c142ff0 0x7f6c4c11f050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:32:53.079 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:53.073+0000 7f6c5404f700 1 --2- 172.21.15.107:0/819149134 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7f6c4c1426e0 0x7f6c4c142ab0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:32:53.079 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:53.073+0000 7f6c5404f700 1 --2- 172.21.15.107:0/819149134 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f6c38004370 0x7f6c4c09d400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:32:53.079 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:53.073+0000 7f6c5404f700 1 -- 172.21.15.107:0/819149134 >> 172.21.15.107:0/819149134 conn(0x7f6c4c097a40 msgr2=0x7f6c4c06d4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:32:53.079 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:53.073+0000 7f6c5404f700 1 -- 172.21.15.107:0/819149134 shutdown_connections 2023-09-30T01:32:53.079 INFO:teuthology.orchestra.run.smithi107.stderr:2023-09-30T01:32:53.073+0000 7f6c5404f700 1 -- 172.21.15.107:0/819149134 wait complete. 2023-09-30T01:32:53.079 INFO:tasks.ceph.mgr.x.smithi107.stderr:2023-09-30T01:32:53.077+0000 7f46b28dd000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-09-30T01:32:53.090 INFO:tasks.ceph.osd.0:Restarting daemon 2023-09-30T01:32:53.090 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2023-09-30T01:32:53.092 INFO:tasks.ceph.osd.0:Started 2023-09-30T01:32:53.092 INFO:tasks.ceph.osd.1:Restarting daemon 2023-09-30T01:32:53.093 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2023-09-30T01:32:53.094 INFO:tasks.ceph.osd.1:Started 2023-09-30T01:32:53.094 INFO:tasks.ceph.osd.2:Restarting daemon 2023-09-30T01:32:53.095 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2023-09-30T01:32:53.096 INFO:tasks.ceph.osd.2:Started 2023-09-30T01:32:53.097 INFO:tasks.ceph.osd.3:Restarting daemon 2023-09-30T01:32:53.097 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2023-09-30T01:32:53.098 INFO:tasks.ceph.osd.3:Started 2023-09-30T01:32:53.099 INFO:tasks.ceph.osd.4:Restarting daemon 2023-09-30T01:32:53.099 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2023-09-30T01:32:53.101 INFO:tasks.ceph.osd.4:Started 2023-09-30T01:32:53.101 INFO:tasks.ceph.osd.5:Restarting daemon 2023-09-30T01:32:53.101 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2023-09-30T01:32:53.103 INFO:tasks.ceph.osd.5:Started 2023-09-30T01:32:53.103 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-09-30T01:32:53.161 INFO:tasks.ceph.mgr.x.smithi107.stderr:2023-09-30T01:32:53.157+0000 7f46b28dd000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-09-30T01:32:53.215 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:53.214+0000 7f57a56c2700 10 monclient: get_monmap_and_config 2023-09-30T01:32:53.215 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:53.214+0000 7f57a56c2700 10 monclient: build_initial_monmap 2023-09-30T01:32:53.215 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:53.214+0000 7f57a56c2700 10 monclient: monmap: 2023-09-30T01:32:53.215 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2023-09-30T01:32:53.215 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:32:53.215 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:53.216013+0000 2023-09-30T01:32:53.215 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:53.216013+0000 2023-09-30T01:32:53.216 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2023-09-30T01:32:53.216 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:32:53.216 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a-legacy 2023-09-30T01:32:53.216 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.107:6789/0 mon.noname-f-legacy 2023-09-30T01:32:53.216 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.noname-b 2023-09-30T01:32:53.216 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.noname-g 2023-09-30T01:32:53.216 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.noname-c 2023-09-30T01:32:53.216 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.noname-h 2023-09-30T01:32:53.216 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.noname-d 2023-09-30T01:32:53.216 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.noname-i 2023-09-30T01:32:53.216 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.noname-e 2023-09-30T01:32:53.216 INFO:teuthology.orchestra.run.smithi088.stderr:9: v2:172.21.15.88:3300/0 mon.noname-a 2023-09-30T01:32:53.217 INFO:teuthology.orchestra.run.smithi088.stderr:10: v2:172.21.15.107:3300/0 mon.noname-f 2023-09-30T01:32:53.217 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:32:53.220 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:53.214+0000 7f57a56c2700 1 Processor -- start 2023-09-30T01:32:53.220 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:53.214+0000 7f57a56c2700 1 -- start start 2023-09-30T01:32:53.221 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:53.214+0000 7f57a56c2700 10 monclient: init 2023-09-30T01:32:53.221 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:53.214+0000 7f57a56c2700 10 monclient: _reopen_session rank -1 2023-09-30T01:32:53.221 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:53.214+0000 7f57a56c2700 10 monclient: _add_conns ranks=[6,2,3,1,4,5,8,0,10,9,7] 2023-09-30T01:32:53.221 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:53.214+0000 7f57a56c2700 1 --2- >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7f57a012f7c0 0x7f57a012fe70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:32:53.221 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:53.214+0000 7f57a56c2700 10 monclient(hunting): picked mon.noname-d con 0x7f57a012f7c0 addr [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] 2023-09-30T01:32:53.221 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:53.214+0000 7f57a56c2700 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f57a006e610 0x7f57a006ea20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:32:53.221 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:53.214+0000 7f57a56c2700 10 monclient(hunting): picked mon.noname-b con 0x7f57a006e610 addr [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] 2023-09-30T01:32:53.221 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:53.214+0000 7f57a56c2700 1 --2- >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7f57a009d580 0x7f57a009d9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:32:53.221 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:53.214+0000 7f57a56c2700 10 monclient(hunting): picked mon.noname-g con 0x7f57a009d580 addr [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] 2023-09-30T01:32:53.221 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:53.214+0000 7f57a56c2700 10 monclient(hunting): start opening mon connection 2023-09-30T01:32:53.221 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:53.214+0000 7f57a56c2700 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57a006bdf0 con 0x7f57a006e610 2023-09-30T01:32:53.222 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:53.214+0000 7f57a56c2700 10 monclient(hunting): start opening mon connection 2023-09-30T01:32:53.222 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:53.214+0000 7f57a56c2700 1 -- --> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57a006a4c0 con 0x7f57a009d580 2023-09-30T01:32:53.222 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:53.214+0000 7f57a56c2700 10 monclient(hunting): start opening mon connection 2023-09-30T01:32:53.222 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:53.214+0000 7f57a56c2700 1 -- --> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] -- mon_getmap magic: 0 v1 -- 0x7f57a012fb90 con 0x7f57a012f7c0 2023-09-30T01:32:53.222 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:53.214+0000 7f57a56c2700 10 monclient(hunting): _renew_subs 2023-09-30T01:32:53.222 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:53.214+0000 7f57a56c2700 10 monclient(hunting): authenticate will time out at 845.254900s 2023-09-30T01:32:53.222 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:53.214+0000 7f579ffff700 1 --2- >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7f57a012f7c0 0x7f57a012fe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:32:53.222 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:53.214+0000 7f579f7fe700 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f57a006e610 0x7f57a006ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:32:53.222 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:53.214+0000 7f579ffff700 1 --2- >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7f57a012f7c0 0x7f57a012fe70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3303/0 says I am v2:172.21.15.88:38150/0 (socket says 172.21.15.88:38150) 2023-09-30T01:32:53.222 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:53.214+0000 7f579ffff700 1 -- 172.21.15.88:0/2045590057 learned_addr learned my addr 172.21.15.88:0/2045590057 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-09-30T01:32:53.222 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:53.214+0000 7f579ffff700 10 monclient(hunting): get_auth_request con 0x7f57a012f7c0 auth_method 0 2023-09-30T01:32:53.223 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:53.214+0000 7f579ffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:32:53.223 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:53.214+0000 7f579ffff700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:32:53.223 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:53.214+0000 7f579ffff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:32:53.223 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:53.214+0000 7f579f7fe700 10 monclient(hunting): get_auth_request con 0x7f57a006e610 auth_method 0 2023-09-30T01:32:53.223 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:53.214+0000 7f579f7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:32:53.223 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:53.214+0000 7f579f7fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:32:53.223 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:53.214+0000 7f579f7fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:32:53.223 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:53.214+0000 7f579ffff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:32:53.223 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:53.214+0000 7f579ffff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:32:53.223 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:53.214+0000 7f579ffff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:32:53.223 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:53.214+0000 7f57a4ec1700 1 --2- 172.21.15.88:0/2045590057 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7f57a009d580 0x7f57a009d9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:32:53.223 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:53.214+0000 7f57a4ec1700 10 monclient(hunting): get_auth_request con 0x7f57a009d580 auth_method 0 2023-09-30T01:32:53.224 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:53.214+0000 7f57a4ec1700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:32:53.224 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:53.214+0000 7f57a4ec1700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:32:53.224 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:53.214+0000 7f57a4ec1700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:32:53.224 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:53.214+0000 7f579f7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:32:53.224 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:53.214+0000 7f579f7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:32:53.224 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:53.214+0000 7f579f7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:32:53.224 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:53.214+0000 7f579ffff700 10 monclient(hunting): handle_auth_done global_id 4128 payload 274 2023-09-30T01:32:53.224 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:53.214+0000 7f579ffff700 1 -- 172.21.15.88:0/2045590057 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7f57a009d580 msgr2=0x7f57a009d9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:32:53.224 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:53.214+0000 7f579ffff700 1 --2- 172.21.15.88:0/2045590057 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7f57a009d580 0x7f57a009d9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:32:53.224 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:53.214+0000 7f579ffff700 1 -- 172.21.15.88:0/2045590057 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f57a006e610 msgr2=0x7f57a006ea20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:32:53.224 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:53.214+0000 7f579ffff700 1 --2- 172.21.15.88:0/2045590057 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f57a006e610 0x7f57a006ea20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:32:53.224 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:53.214+0000 7f579ffff700 10 monclient: _finish_hunting 0 2023-09-30T01:32:53.224 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:53.214+0000 7f579ffff700 1 monclient: found mon.noname-d 2023-09-30T01:32:53.225 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:53.214+0000 7f579ffff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:32:53.225 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:53.214+0000 7f579ffff700 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.88:3303/0 2023-09-30T01:32:53.225 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:53.214+0000 7f579ffff700 1 -- 172.21.15.88:0/2045590057 --> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57a004bb20 con 0x7f57a012f7c0 2023-09-30T01:32:53.225 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:53.214+0000 7f579ffff700 1 --2- 172.21.15.88:0/2045590057 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7f57a012f7c0 0x7f57a012fe70 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f57900099e0 tx=0x7f5790009250 comp rx=0 tx=0).ready entity=mon.6 client_cookie=8b8d941bc9b93d23 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:32:53.225 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:53.218+0000 7f579effd700 1 -- 172.21.15.88:0/2045590057 <== mon.6 v2:172.21.15.88:3303/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f579000bce0 con 0x7f57a012f7c0 2023-09-30T01:32:53.225 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:53.218+0000 7f579effd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:32:53.225 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:53.218+0000 7f579effd700 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2023-09-30T01:32:53.225 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:53.218+0000 7f579effd700 10 monclient: dump: 2023-09-30T01:32:53.225 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:32:53.225 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:32:53.225 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:32:53.225 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:32:53.226 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:32:53.226 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:32:53.226 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:32:53.226 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:32:53.226 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:32:53.226 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:32:53.226 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:32:53.226 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:32:53.226 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:32:53.226 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:32:53.226 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:32:53.226 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:32:53.226 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:53.218+0000 7f579effd700 10 monclient: _finish_auth 0 2023-09-30T01:32:53.227 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:53.218+0000 7f579effd700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:32:53.227 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:53.218+0000 7f579effd700 1 -- 172.21.15.88:0/2045590057 <== mon.6 v2:172.21.15.88:3303/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f579000be60 con 0x7f57a012f7c0 2023-09-30T01:32:53.227 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:53.218+0000 7f579effd700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:32:53.227 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:53.218+0000 7f57a56c2700 5 monclient: authenticate success, global_id 4128 2023-09-30T01:32:53.227 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:53.218+0000 7f579effd700 1 -- 172.21.15.88:0/2045590057 <== mon.6 v2:172.21.15.88:3303/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f579000b860 con 0x7f57a012f7c0 2023-09-30T01:32:53.227 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:53.218+0000 7f57a56c2700 10 monclient: get_monmap_and_config success 2023-09-30T01:32:53.227 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:53.218+0000 7f579effd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:32:53.227 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:53.218+0000 7f57a56c2700 4 set_mon_vals no callback set 2023-09-30T01:32:53.227 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:53.218+0000 7f579effd700 10 monclient: got monmap 1 from mon.d (according to old e1) 2023-09-30T01:32:53.227 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:53.218+0000 7f579effd700 10 monclient: dump: 2023-09-30T01:32:53.227 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:32:53.227 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:32:53.228 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:32:53.228 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:32:53.228 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:32:53.228 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:32:53.228 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:32:53.228 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:32:53.228 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:32:53.228 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:32:53.228 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:32:53.228 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:32:53.228 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:32:53.228 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:32:53.229 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:32:53.229 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:32:53.229 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:53.218+0000 7f57a56c2700 10 monclient: shutdown 2023-09-30T01:32:53.229 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:53.218+0000 7f57a56c2700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:32:53.229 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:53.218+0000 7f57a56c2700 1 -- 172.21.15.88:0/2045590057 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7f57a012f7c0 msgr2=0x7f57a012fe70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:32:53.229 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:53.218+0000 7f57a56c2700 1 --2- 172.21.15.88:0/2045590057 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7f57a012f7c0 0x7f57a012fe70 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f57900099e0 tx=0x7f5790009250 comp rx=0 tx=0).stop 2023-09-30T01:32:53.229 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:53.218+0000 7f57a56c2700 1 -- 172.21.15.88:0/2045590057 shutdown_connections 2023-09-30T01:32:53.229 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:53.218+0000 7f57a56c2700 1 --2- 172.21.15.88:0/2045590057 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7f57a009d580 0x7f57a009d9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:32:53.229 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:53.218+0000 7f57a56c2700 1 --2- 172.21.15.88:0/2045590057 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f57a006e610 0x7f57a006ea20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:32:53.229 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:53.218+0000 7f57a56c2700 1 --2- 172.21.15.88:0/2045590057 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7f57a012f7c0 0x7f57a012fe70 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:32:53.230 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:53.218+0000 7f57a56c2700 1 -- 172.21.15.88:0/2045590057 >> 172.21.15.88:0/2045590057 conn(0x7f57a0063610 msgr2=0x7f57a00a2b90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:32:53.230 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:53.218+0000 7f57a56c2700 1 -- 172.21.15.88:0/2045590057 shutdown_connections 2023-09-30T01:32:53.230 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:53.218+0000 7f57a56c2700 1 -- 172.21.15.88:0/2045590057 wait complete. 2023-09-30T01:32:53.233 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:53.218+0000 7f57a56c2700 10 monclient: build_initial_monmap 2023-09-30T01:32:53.233 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:53.218+0000 7f57a56c2700 10 monclient: monmap: 2023-09-30T01:32:53.234 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2023-09-30T01:32:53.234 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:32:53.234 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 0.000000 2023-09-30T01:32:53.234 INFO:teuthology.orchestra.run.smithi088.stderr:created 0.000000 2023-09-30T01:32:53.234 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2023-09-30T01:32:53.234 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 1 2023-09-30T01:32:53.234 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.noname-a 2023-09-30T01:32:53.234 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.noname-f 2023-09-30T01:32:53.234 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.noname-b 2023-09-30T01:32:53.234 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.noname-g 2023-09-30T01:32:53.234 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.noname-c 2023-09-30T01:32:53.234 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.noname-h 2023-09-30T01:32:53.235 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.noname-d 2023-09-30T01:32:53.235 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.noname-i 2023-09-30T01:32:53.235 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.noname-e 2023-09-30T01:32:53.235 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:32:53.235 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:53.218+0000 7f57a56c2700 1 Processor -- start 2023-09-30T01:32:53.235 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:53.218+0000 7f57a56c2700 1 -- start start 2023-09-30T01:32:53.235 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:53.218+0000 7f57a56c2700 10 monclient: init 2023-09-30T01:32:53.235 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:53.222+0000 7f57a56c2700 10 monclient: _reopen_session rank -1 2023-09-30T01:32:53.235 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:53.222+0000 7f57a56c2700 10 monclient: _add_conns ranks=[1,8,3,0,7,5,2,6,4] 2023-09-30T01:32:53.235 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:53.222+0000 7f57a56c2700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f57a006e610 0x7f57a013ff80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:32:53.235 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:53.222+0000 7f57a56c2700 10 monclient(hunting): picked mon.noname-f con 0x7f57a006e610 addr [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] 2023-09-30T01:32:53.235 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:53.222+0000 7f57a56c2700 1 --2- >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f57a009d580 0x7f57a01355d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:32:53.236 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:53.222+0000 7f57a56c2700 10 monclient(hunting): picked mon.noname-e con 0x7f57a009d580 addr [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] 2023-09-30T01:32:53.236 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:53.222+0000 7f57a56c2700 1 --2- >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7f57a012f7c0 0x7f57a0099e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:32:53.236 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:53.222+0000 7f57a56c2700 10 monclient(hunting): picked mon.noname-g con 0x7f57a012f7c0 addr [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] 2023-09-30T01:32:53.236 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:53.222+0000 7f57a56c2700 10 monclient(hunting): start opening mon connection 2023-09-30T01:32:53.236 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:53.222+0000 7f57a56c2700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57a006b200 con 0x7f57a006e610 2023-09-30T01:32:53.236 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:53.222+0000 7f57a56c2700 10 monclient(hunting): start opening mon connection 2023-09-30T01:32:53.236 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:53.222+0000 7f57a56c2700 1 -- --> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57a0140dc0 con 0x7f57a012f7c0 2023-09-30T01:32:53.236 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:53.222+0000 7f57a56c2700 10 monclient(hunting): start opening mon connection 2023-09-30T01:32:53.236 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:53.222+0000 7f57a56c2700 1 -- --> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] -- mon_getmap magic: 0 v1 -- 0x7f57a01307d0 con 0x7f57a009d580 2023-09-30T01:32:53.236 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:53.222+0000 7f57a56c2700 10 monclient(hunting): _renew_subs 2023-09-30T01:32:53.236 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:53.222+0000 7f57a56c2700 10 monclient(hunting): authenticate will time out at 845.261047s 2023-09-30T01:32:53.236 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:53.222+0000 7f579f7fe700 1 --2- >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f57a009d580 0x7f57a01355d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:32:53.236 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:53.222+0000 7f579f7fe700 1 --2- >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f57a009d580 0x7f57a01355d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3304/0 says I am v2:172.21.15.88:52314/0 (socket says 172.21.15.88:52314) 2023-09-30T01:32:53.237 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:53.222+0000 7f579f7fe700 1 -- 172.21.15.88:0/1116620363 learned_addr learned my addr 172.21.15.88:0/1116620363 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-09-30T01:32:53.237 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:53.222+0000 7f579f7fe700 10 monclient(hunting): get_auth_request con 0x7f57a009d580 auth_method 0 2023-09-30T01:32:53.237 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:53.222+0000 7f579f7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:32:53.237 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:53.222+0000 7f579f7fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:32:53.237 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:53.222+0000 7f579f7fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:32:53.237 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:53.222+0000 7f57a4ec1700 1 --2- 172.21.15.88:0/1116620363 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7f57a012f7c0 0x7f57a0099e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:32:53.237 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:53.222+0000 7f579ffff700 1 --2- 172.21.15.88:0/1116620363 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f57a006e610 0x7f57a013ff80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:32:53.237 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:53.222+0000 7f57a4ec1700 10 monclient(hunting): get_auth_request con 0x7f57a012f7c0 auth_method 0 2023-09-30T01:32:53.237 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:53.222+0000 7f57a4ec1700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:32:53.238 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:53.222+0000 7f57a4ec1700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:32:53.238 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:53.222+0000 7f57a4ec1700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:32:53.238 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:53.222+0000 7f579f7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:32:53.238 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:53.222+0000 7f579f7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:32:53.238 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:53.222+0000 7f579f7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:32:53.238 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:53.222+0000 7f579ffff700 10 monclient(hunting): get_auth_request con 0x7f57a006e610 auth_method 0 2023-09-30T01:32:53.238 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:53.222+0000 7f579ffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:32:53.238 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:53.222+0000 7f579ffff700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:32:53.238 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:53.222+0000 7f579ffff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:32:53.239 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:53.222+0000 7f57a4ec1700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:32:53.239 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:53.222+0000 7f57a4ec1700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:32:53.239 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:53.222+0000 7f57a4ec1700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:32:53.239 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:53.222+0000 7f579ffff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:32:53.239 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:53.222+0000 7f579ffff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:32:53.239 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:53.222+0000 7f579ffff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:32:53.239 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:53.222+0000 7f579f7fe700 10 monclient(hunting): handle_auth_done global_id 4166 payload 824 2023-09-30T01:32:53.239 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:53.222+0000 7f579f7fe700 1 -- 172.21.15.88:0/1116620363 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7f57a012f7c0 msgr2=0x7f57a0099e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:32:53.239 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:53.222+0000 7f579f7fe700 1 --2- 172.21.15.88:0/1116620363 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7f57a012f7c0 0x7f57a0099e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:32:53.239 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:53.222+0000 7f579f7fe700 1 -- 172.21.15.88:0/1116620363 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f57a006e610 msgr2=0x7f57a013ff80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:32:53.239 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:53.222+0000 7f579f7fe700 1 --2- 172.21.15.88:0/1116620363 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f57a006e610 0x7f57a013ff80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:32:53.239 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:53.222+0000 7f579f7fe700 10 monclient: _finish_hunting 0 2023-09-30T01:32:53.240 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:53.222+0000 7f579f7fe700 1 monclient: found mon.noname-e 2023-09-30T01:32:53.240 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:53.222+0000 7f579f7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:32:53.240 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:53.222+0000 7f579f7fe700 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.88:3304/0 2023-09-30T01:32:53.240 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:53.222+0000 7f579f7fe700 1 -- 172.21.15.88:0/1116620363 --> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57900094f0 con 0x7f57a009d580 2023-09-30T01:32:53.240 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:53.222+0000 7f579f7fe700 1 --2- 172.21.15.88:0/1116620363 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f57a009d580 0x7f57a01355d0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f579400e900 tx=0x7f57940091a0 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:32:53.240 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:53.222+0000 7f5787fff700 1 -- 172.21.15.88:0/1116620363 <== mon.8 v2:172.21.15.88:3304/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f5794013b00 con 0x7f57a009d580 2023-09-30T01:32:53.240 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:53.222+0000 7f5787fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:32:53.240 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:53.222+0000 7f5787fff700 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2023-09-30T01:32:53.240 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:53.222+0000 7f5787fff700 10 monclient: dump: 2023-09-30T01:32:53.241 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:32:53.241 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:32:53.241 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:32:53.241 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:32:53.241 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:32:53.241 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:32:53.241 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:32:53.241 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:32:53.241 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:32:53.241 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:32:53.242 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:32:53.242 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:32:53.242 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:32:53.242 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:32:53.242 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:32:53.242 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:32:53.242 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:53.222+0000 7f5787fff700 10 monclient: _finish_auth 0 2023-09-30T01:32:53.242 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:53.222+0000 7f5787fff700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:32:53.242 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:53.222+0000 7f57a56c2700 5 monclient: authenticate success, global_id 4166 2023-09-30T01:32:53.242 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:53.222+0000 7f5787fff700 1 -- 172.21.15.88:0/1116620363 <== mon.8 v2:172.21.15.88:3304/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5794013c60 con 0x7f57a009d580 2023-09-30T01:32:53.242 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:53.222+0000 7f57a56c2700 10 monclient: _renew_subs 2023-09-30T01:32:53.242 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:53.222+0000 7f57a56c2700 10 monclient: _send_mon_message to mon.e at v2:172.21.15.88:3304/0 2023-09-30T01:32:53.243 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:53.222+0000 7f57a56c2700 1 -- 172.21.15.88:0/1116620363 --> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f57a0130ab0 con 0x7f57a009d580 2023-09-30T01:32:53.243 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:53.222+0000 7f5787fff700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:32:53.243 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:53.222+0000 7f5787fff700 1 -- 172.21.15.88:0/1116620363 <== mon.8 v2:172.21.15.88:3304/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f5794013dd0 con 0x7f57a009d580 2023-09-30T01:32:53.243 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:53.222+0000 7f5787fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:32:53.243 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:53.222+0000 7f5787fff700 10 monclient: got monmap 1 from mon.e (according to old e1) 2023-09-30T01:32:53.243 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:53.222+0000 7f5787fff700 10 monclient: dump: 2023-09-30T01:32:53.243 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:32:53.243 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:32:53.243 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:32:53.243 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:32:53.243 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:32:53.243 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:32:53.243 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:32:53.244 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:32:53.244 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:32:53.244 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:32:53.244 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:32:53.244 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:32:53.244 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:32:53.244 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:32:53.244 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:32:53.244 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:32:53.244 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:53.222+0000 7f579e7fc700 4 set_mon_vals no callback set 2023-09-30T01:32:53.244 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:53.222+0000 7f57a56c2700 10 monclient: _renew_subs 2023-09-30T01:32:53.244 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:53.222+0000 7f57a56c2700 10 monclient: _send_mon_message to mon.e at v2:172.21.15.88:3304/0 2023-09-30T01:32:53.245 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:53.222+0000 7f57a56c2700 1 -- 172.21.15.88:0/1116620363 --> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f57a009c550 con 0x7f57a009d580 2023-09-30T01:32:53.245 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:53.222+0000 7f5787fff700 1 -- 172.21.15.88:0/1116620363 <== mon.8 v2:172.21.15.88:3304/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7f579401c450 con 0x7f57a009d580 2023-09-30T01:32:53.245 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:53.222+0000 7f5787fff700 1 -- 172.21.15.88:0/1116620363 <== mon.8 v2:172.21.15.88:3304/0 5 ==== osd_map(8..8 src has 1..8) v4 ==== 1411+0+0 (secure 0 0 0) 0x7f5794013900 con 0x7f57a009d580 2023-09-30T01:32:53.245 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:53.222+0000 7f57a56c2700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:32:53.245 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:53.222+0000 7f57a56c2700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:32:53.245 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:53.222+0000 7f57a56c2700 10 monclient: _send_mon_message to mon.e at v2:172.21.15.88:3304/0 2023-09-30T01:32:53.245 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:53.222+0000 7f57a56c2700 1 -- 172.21.15.88:0/1116620363 --> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f57a0130eb0 con 0x7f57a009d580 2023-09-30T01:32:53.245 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:53.226+0000 7f5787fff700 1 -- 172.21.15.88:0/1116620363 <== mon.8 v2:172.21.15.88:3304/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7f5794021060 con 0x7f57a009d580 2023-09-30T01:32:53.245 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:53.226+0000 7f5787fff700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:32:53.245 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:53.226+0000 7f5787fff700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:32:53.296 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:53.294+0000 7f57a56c2700 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2023-09-30T01:32:53.296 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:53.294+0000 7f57a56c2700 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2023-09-30T01:32:53.296 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:53.294+0000 7f57a56c2700 10 monclient: _send_mon_message to mon.e at v2:172.21.15.88:3304/0 2023-09-30T01:32:53.296 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:53.294+0000 7f57a56c2700 1 -- 172.21.15.88:0/1116620363 --> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f57a0071960 con 0x7f57a009d580 2023-09-30T01:32:53.297 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:53.294+0000 7f5787fff700 1 -- 172.21.15.88:0/1116620363 <== mon.8 v2:172.21.15.88:3304/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v8) v1 ==== 74+0+5560 (secure 0 0 0) 0x7f5794017070 con 0x7f57a009d580 2023-09-30T01:32:53.297 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:53.294+0000 7f5787fff700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2023-09-30T01:32:53.297 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:53.294+0000 7f5787fff700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:32:53.297 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-09-30T01:32:53.297 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":8,"fsid":"650badb5-899b-4472-88a6-cfdccee24d52","created":"2023-09-30T01:32:51.511154+0000","modified":"2023-09-30T01:32:53.067458+0000","last_up_change":"0.000000","last_in_change":"2023-09-30T01:32:53.067458+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"1243f825-b99c-4546-a77d-90b8637698b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e9831b75-399b-4d6d-bf97-9ef104adfdb5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"fbc7be74-cd43-4c72-89c1-469c17055b85","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ca888e00-8f75-4dc4-b70c-6cd46880fe74","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"c8c73bd1-7674-4529-9104-8690e25af2fc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"a3994780-9d6a-4bcd-b718-09f9d8774b3d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-09-30T01:32:53.298 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:53.294+0000 7f57a56c2700 10 monclient: shutdown 2023-09-30T01:32:53.298 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:53.294+0000 7f57a56c2700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:32:53.298 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:53.294+0000 7f57a56c2700 1 -- 172.21.15.88:0/1116620363 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f57a009d580 msgr2=0x7f57a01355d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:32:53.298 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:53.294+0000 7f57a56c2700 1 --2- 172.21.15.88:0/1116620363 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f57a009d580 0x7f57a01355d0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f579400e900 tx=0x7f57940091a0 comp rx=0 tx=0).stop 2023-09-30T01:32:53.299 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:53.294+0000 7f57a56c2700 1 -- 172.21.15.88:0/1116620363 shutdown_connections 2023-09-30T01:32:53.299 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:53.294+0000 7f57a56c2700 1 --2- 172.21.15.88:0/1116620363 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7f57a012f7c0 0x7f57a0099e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:32:53.299 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:53.294+0000 7f57a56c2700 1 --2- 172.21.15.88:0/1116620363 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f57a009d580 0x7f57a01355d0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:32:53.299 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:53.294+0000 7f57a56c2700 1 --2- 172.21.15.88:0/1116620363 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f57a006e610 0x7f57a013ff80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:32:53.299 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:53.294+0000 7f57a56c2700 1 -- 172.21.15.88:0/1116620363 >> 172.21.15.88:0/1116620363 conn(0x7f57a0063610 msgr2=0x7f57a009f680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:32:53.299 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:53.294+0000 7f57a56c2700 1 -- 172.21.15.88:0/1116620363 shutdown_connections 2023-09-30T01:32:53.299 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:32:53.294+0000 7f57a56c2700 1 -- 172.21.15.88:0/1116620363 wait complete. 2023-09-30T01:32:53.309 INFO:tasks.ceph.ceph_manager.ceph:[] 2023-09-30T01:32:53.309 INFO:tasks.ceph:Waiting for OSDs to come up 2023-09-30T01:32:53.453 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-09-30T01:32:53.450+0000 7f7ba463d3c0 -1 Falling back to public interface 2023-09-30T01:32:53.469 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-09-30T01:32:53.465+0000 7f2b191cd3c0 -1 Falling back to public interface 2023-09-30T01:32:53.568 INFO:tasks.ceph.mgr.x.smithi107.stderr:2023-09-30T01:32:53.565+0000 7f46b28dd000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-09-30T01:32:53.686 INFO:tasks.ceph.mgr.x.smithi107.stderr:2023-09-30T01:32:53.681+0000 7f46b28dd000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-09-30T01:32:53.693 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-09-30T01:32:53.690+0000 7fd3d2cdd3c0 -1 Falling back to public interface 2023-09-30T01:32:53.728 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-09-30T01:32:53.726+0000 7f59b8c603c0 -1 Falling back to public interface 2023-09-30T01:32:53.729 INFO:tasks.ceph.osd.3.smithi107.stderr:2023-09-30T01:32:53.725+0000 7fb9d02a03c0 -1 Falling back to public interface 2023-09-30T01:32:53.745 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-09-30T01:32:53.741+0000 7f6907df03c0 -1 Falling back to public interface 2023-09-30T01:32:53.914 DEBUG:teuthology.orchestra.run.smithi088:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-09-30T01:32:53.973 INFO:tasks.ceph.mgr.x.smithi107.stderr:2023-09-30T01:32:53.969+0000 7f46b28dd000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-09-30T01:32:54.011 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:32:54.010+0000 7f23ade97700 10 monclient: get_monmap_and_config 2023-09-30T01:32:54.011 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:32:54.010+0000 7f23ade97700 10 monclient: build_initial_monmap 2023-09-30T01:32:54.011 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:32:54.010+0000 7f23ade97700 10 monclient: monmap: 2023-09-30T01:32:54.011 INFO:teuthology.misc.health.smithi088.stderr:epoch 0 2023-09-30T01:32:54.011 INFO:teuthology.misc.health.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:32:54.012 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2023-09-30T01:32:54.011953+0000 2023-09-30T01:32:54.012 INFO:teuthology.misc.health.smithi088.stderr:created 2023-09-30T01:32:54.011953+0000 2023-09-30T01:32:54.012 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 0 (unknown) 2023-09-30T01:32:54.012 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 3 2023-09-30T01:32:54.012 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a-legacy 2023-09-30T01:32:54.012 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.107:6789/0 mon.noname-f-legacy 2023-09-30T01:32:54.012 INFO:teuthology.misc.health.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.noname-b 2023-09-30T01:32:54.012 INFO:teuthology.misc.health.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.noname-g 2023-09-30T01:32:54.012 INFO:teuthology.misc.health.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.noname-c 2023-09-30T01:32:54.012 INFO:teuthology.misc.health.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.noname-h 2023-09-30T01:32:54.012 INFO:teuthology.misc.health.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.noname-d 2023-09-30T01:32:54.012 INFO:teuthology.misc.health.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.noname-i 2023-09-30T01:32:54.013 INFO:teuthology.misc.health.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.noname-e 2023-09-30T01:32:54.013 INFO:teuthology.misc.health.smithi088.stderr:9: v2:172.21.15.88:3300/0 mon.noname-a 2023-09-30T01:32:54.013 INFO:teuthology.misc.health.smithi088.stderr:10: v2:172.21.15.107:3300/0 mon.noname-f 2023-09-30T01:32:54.013 INFO:teuthology.misc.health.smithi088.stderr: 2023-09-30T01:32:54.016 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:32:54.010+0000 7f23ade97700 1 Processor -- start 2023-09-30T01:32:54.016 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:32:54.010+0000 7f23ade97700 1 -- start start 2023-09-30T01:32:54.016 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:32:54.010+0000 7f23ade97700 10 monclient: init 2023-09-30T01:32:54.016 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:32:54.010+0000 7f23ade97700 10 monclient: _reopen_session rank -1 2023-09-30T01:32:54.016 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:32:54.010+0000 7f23ade97700 10 monclient: _add_conns ranks=[1,4,6,10,2,7,9,8,5,0,3] 2023-09-30T01:32:54.017 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:32:54.010+0000 7f23ade97700 10 monclient(hunting): picked mon.noname-f-legacy con 0x7f23a809d680 addr v1:172.21.15.107:6789/0 2023-09-30T01:32:54.017 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:32:54.010+0000 7f23ade97700 1 --2- >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f23a809a250 0x7f23a81366b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:32:54.017 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:32:54.010+0000 7f23ade97700 10 monclient(hunting): picked mon.noname-c con 0x7f23a809a250 addr [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] 2023-09-30T01:32:54.017 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:32:54.010+0000 7f23ade97700 1 --2- >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7f23a809b9a0 0x7f23a8138c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:32:54.017 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:32:54.010+0000 7f23ade97700 10 monclient(hunting): picked mon.noname-d con 0x7f23a809b9a0 addr [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] 2023-09-30T01:32:54.017 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:32:54.010+0000 7f23ade97700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23a805ecb0 con 0x7f23a809d680 2023-09-30T01:32:54.017 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:32:54.010+0000 7f23ade97700 10 monclient(hunting): start opening mon connection 2023-09-30T01:32:54.017 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:32:54.010+0000 7f23ade97700 1 -- --> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] -- mon_getmap magic: 0 v1 -- 0x7f23a809b020 con 0x7f23a809a250 2023-09-30T01:32:54.017 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:32:54.010+0000 7f23ade97700 10 monclient(hunting): start opening mon connection 2023-09-30T01:32:54.017 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:32:54.010+0000 7f23ade97700 1 -- --> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] -- mon_getmap magic: 0 v1 -- 0x7f23a809a9f0 con 0x7f23a809b9a0 2023-09-30T01:32:54.017 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:32:54.010+0000 7f23ade97700 10 monclient(hunting): _renew_subs 2023-09-30T01:32:54.017 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:32:54.010+0000 7f23ade97700 10 monclient(hunting): authenticate will time out at 846.049504s 2023-09-30T01:32:54.018 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:32:54.010+0000 7f23a6ffd700 1 --2- >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f23a809a250 0x7f23a81366b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:32:54.018 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:32:54.010+0000 7f23a7fff700 1 --2- >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7f23a809b9a0 0x7f23a8138c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:32:54.018 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:32:54.010+0000 7f23a6ffd700 1 --2- >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f23a809a250 0x7f23a81366b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3302/0 says I am v2:172.21.15.88:45308/0 (socket says 172.21.15.88:45308) 2023-09-30T01:32:54.018 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:32:54.010+0000 7f23a6ffd700 1 -- 172.21.15.88:0/621026321 learned_addr learned my addr 172.21.15.88:0/621026321 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-09-30T01:32:54.018 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:32:54.010+0000 7f23a7fff700 1 --2- >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7f23a809b9a0 0x7f23a8138c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3303/0 says I am v2:172.21.15.88:38152/0 (socket says 172.21.15.88:38152) 2023-09-30T01:32:54.018 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:32:54.010+0000 7f23a6ffd700 10 monclient(hunting): get_auth_request con 0x7f23a809a250 auth_method 0 2023-09-30T01:32:54.018 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:32:54.010+0000 7f23a6ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:32:54.018 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:32:54.010+0000 7f23a6ffd700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:32:54.018 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:32:54.010+0000 7f23a6ffd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:32:54.018 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:32:54.010+0000 7f23a7fff700 10 monclient(hunting): get_auth_request con 0x7f23a809b9a0 auth_method 0 2023-09-30T01:32:54.018 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:32:54.010+0000 7f23a7fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:32:54.019 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:32:54.010+0000 7f23a7fff700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:32:54.019 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:32:54.010+0000 7f23a7fff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:32:54.019 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:32:54.010+0000 7f23a6ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:32:54.019 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:32:54.010+0000 7f23a6ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:32:54.019 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:32:54.010+0000 7f23a6ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:32:54.019 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:32:54.010+0000 7f23a7fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:32:54.019 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:32:54.010+0000 7f23a7fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:32:54.020 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:32:54.010+0000 7f23a7fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:32:54.020 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:32:54.010+0000 7f23a6ffd700 10 monclient(hunting): handle_auth_done global_id 4162 payload 274 2023-09-30T01:32:54.020 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:32:54.010+0000 7f23a6ffd700 1 -- 172.21.15.88:0/621026321 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7f23a809b9a0 msgr2=0x7f23a8138c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:32:54.020 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:32:54.010+0000 7f23a67fc700 1 -- 172.21.15.88:0/621026321 <== mon.1 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3578484308 0 0) 0x7f23a805ecb0 con 0x7f23a809d680 2023-09-30T01:32:54.021 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:32:54.010+0000 7f23a6ffd700 1 --2- 172.21.15.88:0/621026321 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7f23a809b9a0 0x7f23a8138c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:32:54.021 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:32:54.010+0000 7f23a6ffd700 1 -- 172.21.15.88:0/621026321 >> v1:172.21.15.107:6789/0 conn(0x7f23a809d680 legacy=0x7f23a809da50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:32:54.021 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:32:54.010+0000 7f23a6ffd700 10 monclient: _finish_hunting 0 2023-09-30T01:32:54.021 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:32:54.010+0000 7f23a6ffd700 1 monclient: found mon.noname-c 2023-09-30T01:32:54.021 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:32:54.010+0000 7f23a6ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:32:54.022 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:32:54.010+0000 7f23a6ffd700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.88:3302/0 2023-09-30T01:32:54.022 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:32:54.010+0000 7f23a6ffd700 1 -- 172.21.15.88:0/621026321 --> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23a809e850 con 0x7f23a809a250 2023-09-30T01:32:54.022 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:32:54.010+0000 7f23a67fc700 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2023-09-30T01:32:54.022 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:32:54.010+0000 7f23a7fff700 1 --2- 172.21.15.88:0/621026321 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7f23a809b9a0 0x7f23a8138c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:32:54.022 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:32:54.010+0000 7f23a6ffd700 1 --2- 172.21.15.88:0/621026321 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f23a809a250 0x7f23a81366b0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f2398009f50 tx=0x7f2398009840 comp rx=0 tx=0).ready entity=mon.4 client_cookie=b86618f90c4d6d50 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:32:54.022 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:32:54.010+0000 7f23a67fc700 1 -- 172.21.15.88:0/621026321 <== mon.4 v2:172.21.15.88:3302/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f239800e070 con 0x7f23a809a250 2023-09-30T01:32:54.022 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:32:54.010+0000 7f23a67fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:32:54.022 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:32:54.010+0000 7f23a67fc700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T01:32:54.022 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:32:54.010+0000 7f23a67fc700 10 monclient: dump: 2023-09-30T01:32:54.022 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2023-09-30T01:32:54.022 INFO:teuthology.misc.health.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:32:54.022 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:32:54.023 INFO:teuthology.misc.health.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:32:54.023 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:32:54.023 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 3 2023-09-30T01:32:54.023 INFO:teuthology.misc.health.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:32:54.025 INFO:teuthology.misc.health.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:32:54.025 INFO:teuthology.misc.health.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:32:54.025 INFO:teuthology.misc.health.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:32:54.025 INFO:teuthology.misc.health.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:32:54.025 INFO:teuthology.misc.health.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:32:54.025 INFO:teuthology.misc.health.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:32:54.025 INFO:teuthology.misc.health.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:32:54.025 INFO:teuthology.misc.health.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:32:54.025 INFO:teuthology.misc.health.smithi088.stderr: 2023-09-30T01:32:54.026 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:32:54.010+0000 7f23a67fc700 10 monclient: _finish_auth 0 2023-09-30T01:32:54.026 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:32:54.010+0000 7f23a67fc700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:32:54.026 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:32:54.010+0000 7f23ade97700 5 monclient: authenticate success, global_id 4162 2023-09-30T01:32:54.026 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:32:54.010+0000 7f23ade97700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:32:54.026 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:32:54.010+0000 7f23a67fc700 1 -- 172.21.15.88:0/621026321 <== mon.4 v2:172.21.15.88:3302/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f239800be80 con 0x7f23a809a250 2023-09-30T01:32:54.026 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:32:54.010+0000 7f23a67fc700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:32:54.026 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:32:54.010+0000 7f23ade97700 10 monclient: get_monmap_and_config success 2023-09-30T01:32:54.026 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:32:54.010+0000 7f23ade97700 4 set_mon_vals no callback set 2023-09-30T01:32:54.026 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:32:54.010+0000 7f23a67fc700 1 -- 172.21.15.88:0/621026321 <== mon.4 v2:172.21.15.88:3302/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f23980092e0 con 0x7f23a809a250 2023-09-30T01:32:54.026 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:32:54.010+0000 7f23a67fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:32:54.026 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:32:54.010+0000 7f23a67fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T01:32:54.027 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:32:54.010+0000 7f23a67fc700 10 monclient: dump: 2023-09-30T01:32:54.027 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2023-09-30T01:32:54.027 INFO:teuthology.misc.health.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:32:54.027 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:32:54.027 INFO:teuthology.misc.health.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:32:54.027 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:32:54.027 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 3 2023-09-30T01:32:54.027 INFO:teuthology.misc.health.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:32:54.027 INFO:teuthology.misc.health.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:32:54.027 INFO:teuthology.misc.health.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:32:54.027 INFO:teuthology.misc.health.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:32:54.027 INFO:teuthology.misc.health.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:32:54.027 INFO:teuthology.misc.health.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:32:54.028 INFO:teuthology.misc.health.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:32:54.028 INFO:teuthology.misc.health.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:32:54.028 INFO:teuthology.misc.health.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:32:54.028 INFO:teuthology.misc.health.smithi088.stderr: 2023-09-30T01:32:54.028 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:32:54.010+0000 7f23ade97700 10 monclient: shutdown 2023-09-30T01:32:54.028 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:32:54.010+0000 7f23ade97700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:32:54.028 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:32:54.010+0000 7f23ade97700 1 -- 172.21.15.88:0/621026321 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f23a809a250 msgr2=0x7f23a81366b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:32:54.028 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:32:54.010+0000 7f23ade97700 1 --2- 172.21.15.88:0/621026321 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f23a809a250 0x7f23a81366b0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f2398009f50 tx=0x7f2398009840 comp rx=0 tx=0).stop 2023-09-30T01:32:54.028 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:32:54.010+0000 7f23ade97700 1 -- 172.21.15.88:0/621026321 shutdown_connections 2023-09-30T01:32:54.028 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:32:54.010+0000 7f23ade97700 1 --2- 172.21.15.88:0/621026321 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7f23a809b9a0 0x7f23a8138c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:32:54.028 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:32:54.010+0000 7f23ade97700 1 --2- 172.21.15.88:0/621026321 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f23a809a250 0x7f23a81366b0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:32:54.029 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:32:54.010+0000 7f23ade97700 1 -- 172.21.15.88:0/621026321 >> 172.21.15.88:0/621026321 conn(0x7f23a80637a0 msgr2=0x7f23a80a2c90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:32:54.029 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:32:54.010+0000 7f23ade97700 1 -- 172.21.15.88:0/621026321 shutdown_connections 2023-09-30T01:32:54.029 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:32:54.010+0000 7f23ade97700 1 -- 172.21.15.88:0/621026321 wait complete. 2023-09-30T01:32:54.029 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:32:54.014+0000 7f23ade97700 10 monclient: build_initial_monmap 2023-09-30T01:32:54.029 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:32:54.014+0000 7f23ade97700 10 monclient: monmap: 2023-09-30T01:32:54.029 INFO:teuthology.misc.health.smithi088.stderr:epoch 0 2023-09-30T01:32:54.029 INFO:teuthology.misc.health.smithi088.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:32:54.029 INFO:teuthology.misc.health.smithi088.stderr:last_changed 0.000000 2023-09-30T01:32:54.029 INFO:teuthology.misc.health.smithi088.stderr:created 0.000000 2023-09-30T01:32:54.029 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 0 (unknown) 2023-09-30T01:32:54.029 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2023-09-30T01:32:54.029 INFO:teuthology.misc.health.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.noname-a 2023-09-30T01:32:54.030 INFO:teuthology.misc.health.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.noname-f 2023-09-30T01:32:54.030 INFO:teuthology.misc.health.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.noname-b 2023-09-30T01:32:54.030 INFO:teuthology.misc.health.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.noname-g 2023-09-30T01:32:54.030 INFO:teuthology.misc.health.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.noname-c 2023-09-30T01:32:54.030 INFO:teuthology.misc.health.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.noname-h 2023-09-30T01:32:54.030 INFO:teuthology.misc.health.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.noname-d 2023-09-30T01:32:54.030 INFO:teuthology.misc.health.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.noname-i 2023-09-30T01:32:54.030 INFO:teuthology.misc.health.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.noname-e 2023-09-30T01:32:54.030 INFO:teuthology.misc.health.smithi088.stderr: 2023-09-30T01:32:54.034 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:32:54.014+0000 7f23ade97700 1 Processor -- start 2023-09-30T01:32:54.034 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:32:54.014+0000 7f23ade97700 1 -- start start 2023-09-30T01:32:54.034 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:32:54.014+0000 7f23ade97700 10 monclient: init 2023-09-30T01:32:54.034 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:32:54.014+0000 7f23ade97700 10 monclient: _reopen_session rank -1 2023-09-30T01:32:54.034 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:32:54.014+0000 7f23ade97700 10 monclient: _add_conns ranks=[0,7,4,5,8,3,6,2,1] 2023-09-30T01:32:54.035 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:32:54.014+0000 7f23ade97700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f23a809a250 0x7f23a811ea70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:32:54.035 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:32:54.014+0000 7f23ade97700 10 monclient(hunting): picked mon.noname-a con 0x7f23a809a250 addr [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] 2023-09-30T01:32:54.035 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:32:54.014+0000 7f23ade97700 1 --2- >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7f23a809b9a0 0x7f23a813c0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:32:54.035 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:32:54.014+0000 7f23ade97700 10 monclient(hunting): picked mon.noname-i con 0x7f23a809b9a0 addr [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] 2023-09-30T01:32:54.035 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:32:54.014+0000 7f23ade97700 1 --2- >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f23a809d680 0x7f23a813ca20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:32:54.035 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:32:54.014+0000 7f23ade97700 10 monclient(hunting): picked mon.noname-c con 0x7f23a809d680 addr [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] 2023-09-30T01:32:54.035 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:32:54.014+0000 7f23ade97700 10 monclient(hunting): start opening mon connection 2023-09-30T01:32:54.035 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:32:54.014+0000 7f23ade97700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23a8121450 con 0x7f23a809a250 2023-09-30T01:32:54.035 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:32:54.014+0000 7f23ade97700 10 monclient(hunting): start opening mon connection 2023-09-30T01:32:54.035 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:32:54.014+0000 7f23ade97700 1 -- --> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] -- mon_getmap magic: 0 v1 -- 0x7f23a8121170 con 0x7f23a809d680 2023-09-30T01:32:54.035 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:32:54.014+0000 7f23ade97700 10 monclient(hunting): start opening mon connection 2023-09-30T01:32:54.035 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:32:54.014+0000 7f23ade97700 1 -- --> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] -- mon_getmap magic: 0 v1 -- 0x7f23a8121730 con 0x7f23a809b9a0 2023-09-30T01:32:54.036 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:32:54.014+0000 7f23ade97700 10 monclient(hunting): _renew_subs 2023-09-30T01:32:54.036 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:32:54.014+0000 7f23ade97700 10 monclient(hunting): authenticate will time out at 846.055408s 2023-09-30T01:32:54.036 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:32:54.014+0000 7f23a7fff700 1 --2- >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f23a809d680 0x7f23a813ca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:32:54.036 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:32:54.014+0000 7f23a7fff700 1 --2- >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f23a809d680 0x7f23a813ca20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3302/0 says I am v2:172.21.15.88:45316/0 (socket says 172.21.15.88:45316) 2023-09-30T01:32:54.036 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:32:54.014+0000 7f23a7fff700 1 -- 172.21.15.88:0/482904701 learned_addr learned my addr 172.21.15.88:0/482904701 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-09-30T01:32:54.036 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:32:54.014+0000 7f23a7fff700 10 monclient(hunting): get_auth_request con 0x7f23a809d680 auth_method 0 2023-09-30T01:32:54.036 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:32:54.014+0000 7f23a7fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:32:54.036 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:32:54.014+0000 7f23a7fff700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:32:54.036 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:32:54.014+0000 7f23a7fff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:32:54.036 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:32:54.014+0000 7f23a77fe700 1 --2- 172.21.15.88:0/482904701 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f23a809a250 0x7f23a811ea70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:32:54.036 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:32:54.014+0000 7f23a7fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:32:54.036 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:32:54.014+0000 7f23a7fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:32:54.037 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:32:54.014+0000 7f23a7fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:32:54.037 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:32:54.014+0000 7f23a77fe700 10 monclient(hunting): get_auth_request con 0x7f23a809a250 auth_method 0 2023-09-30T01:32:54.037 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:32:54.014+0000 7f23a6ffd700 1 --2- 172.21.15.88:0/482904701 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7f23a809b9a0 0x7f23a813c0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:32:54.037 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:32:54.014+0000 7f23a77fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:32:54.037 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:32:54.014+0000 7f23a77fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:32:54.037 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:32:54.014+0000 7f23a77fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:32:54.037 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:32:54.014+0000 7f23a6ffd700 10 monclient(hunting): get_auth_request con 0x7f23a809b9a0 auth_method 0 2023-09-30T01:32:54.037 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:32:54.014+0000 7f23a6ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:32:54.037 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:32:54.014+0000 7f23a6ffd700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:32:54.037 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:32:54.014+0000 7f23a6ffd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:32:54.037 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:32:54.014+0000 7f23a77fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:32:54.037 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:32:54.014+0000 7f23a77fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:32:54.037 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:32:54.014+0000 7f23a77fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:32:54.038 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:32:54.018+0000 7f23a6ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:32:54.038 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:32:54.018+0000 7f23a6ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:32:54.038 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:32:54.018+0000 7f23a6ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:32:54.038 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:32:54.018+0000 7f23a7fff700 10 monclient(hunting): handle_auth_done global_id 4171 payload 824 2023-09-30T01:32:54.038 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:32:54.018+0000 7f23a7fff700 1 -- 172.21.15.88:0/482904701 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7f23a809b9a0 msgr2=0x7f23a813c0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:32:54.038 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:32:54.018+0000 7f23a7fff700 1 --2- 172.21.15.88:0/482904701 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7f23a809b9a0 0x7f23a813c0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:32:54.038 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:32:54.018+0000 7f23a7fff700 1 -- 172.21.15.88:0/482904701 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f23a809a250 msgr2=0x7f23a811ea70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:32:54.038 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:32:54.018+0000 7f23a7fff700 1 --2- 172.21.15.88:0/482904701 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f23a809a250 0x7f23a811ea70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:32:54.038 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:32:54.018+0000 7f23a7fff700 10 monclient: _finish_hunting 0 2023-09-30T01:32:54.038 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:32:54.018+0000 7f23a7fff700 1 monclient: found mon.noname-c 2023-09-30T01:32:54.038 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:32:54.018+0000 7f23a7fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:32:54.039 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:32:54.018+0000 7f23a7fff700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.88:3302/0 2023-09-30T01:32:54.039 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:32:54.018+0000 7f23a7fff700 1 -- 172.21.15.88:0/482904701 --> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23a0009420 con 0x7f23a809d680 2023-09-30T01:32:54.039 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:32:54.018+0000 7f23a77fe700 1 --2- 172.21.15.88:0/482904701 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f23a809a250 0x7f23a811ea70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:32:54.039 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:32:54.018+0000 7f23a7fff700 1 --2- 172.21.15.88:0/482904701 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f23a809d680 0x7f23a813ca20 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f23a000ec80 tx=0x7f23a00091a0 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:32:54.039 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:32:54.018+0000 7f238f7fe700 1 -- 172.21.15.88:0/482904701 <== mon.4 v2:172.21.15.88:3302/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f23a0009720 con 0x7f23a809d680 2023-09-30T01:32:54.039 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:32:54.018+0000 7f238f7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:32:54.039 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:32:54.018+0000 7f238f7fe700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T01:32:54.039 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:32:54.018+0000 7f238f7fe700 10 monclient: dump: 2023-09-30T01:32:54.039 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2023-09-30T01:32:54.039 INFO:teuthology.misc.health.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:32:54.039 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:32:54.039 INFO:teuthology.misc.health.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:32:54.040 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:32:54.040 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 3 2023-09-30T01:32:54.040 INFO:teuthology.misc.health.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:32:54.040 INFO:teuthology.misc.health.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:32:54.040 INFO:teuthology.misc.health.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:32:54.040 INFO:teuthology.misc.health.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:32:54.040 INFO:teuthology.misc.health.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:32:54.040 INFO:teuthology.misc.health.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:32:54.040 INFO:teuthology.misc.health.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:32:54.040 INFO:teuthology.misc.health.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:32:54.040 INFO:teuthology.misc.health.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:32:54.040 INFO:teuthology.misc.health.smithi088.stderr: 2023-09-30T01:32:54.040 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:32:54.018+0000 7f238f7fe700 10 monclient: _finish_auth 0 2023-09-30T01:32:54.041 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:32:54.018+0000 7f238f7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:32:54.041 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:32:54.018+0000 7f238f7fe700 1 -- 172.21.15.88:0/482904701 <== mon.4 v2:172.21.15.88:3302/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f23a000ccd0 con 0x7f23a809d680 2023-09-30T01:32:54.041 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:32:54.018+0000 7f23ade97700 5 monclient: authenticate success, global_id 4171 2023-09-30T01:32:54.041 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:32:54.018+0000 7f23ade97700 10 monclient: _renew_subs 2023-09-30T01:32:54.041 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:32:54.018+0000 7f23ade97700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.88:3302/0 2023-09-30T01:32:54.041 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:32:54.018+0000 7f23ade97700 1 -- 172.21.15.88:0/482904701 --> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2398009ae0 con 0x7f23a809d680 2023-09-30T01:32:54.041 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:32:54.018+0000 7f238f7fe700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:32:54.041 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:32:54.018+0000 7f238f7fe700 1 -- 172.21.15.88:0/482904701 <== mon.4 v2:172.21.15.88:3302/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f23a000ce40 con 0x7f23a809d680 2023-09-30T01:32:54.041 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:32:54.018+0000 7f238f7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:32:54.041 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:32:54.018+0000 7f238f7fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T01:32:54.041 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:32:54.018+0000 7f238f7fe700 10 monclient: dump: 2023-09-30T01:32:54.041 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2023-09-30T01:32:54.041 INFO:teuthology.misc.health.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:32:54.042 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:32:54.042 INFO:teuthology.misc.health.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:32:54.042 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:32:54.042 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 3 2023-09-30T01:32:54.042 INFO:teuthology.misc.health.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:32:54.042 INFO:teuthology.misc.health.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:32:54.042 INFO:teuthology.misc.health.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:32:54.042 INFO:teuthology.misc.health.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:32:54.042 INFO:teuthology.misc.health.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:32:54.042 INFO:teuthology.misc.health.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:32:54.042 INFO:teuthology.misc.health.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:32:54.042 INFO:teuthology.misc.health.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:32:54.043 INFO:teuthology.misc.health.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:32:54.043 INFO:teuthology.misc.health.smithi088.stderr: 2023-09-30T01:32:54.043 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:32:54.018+0000 7f23a57fa700 4 set_mon_vals no callback set 2023-09-30T01:32:54.043 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:32:54.018+0000 7f23ade97700 10 monclient: _renew_subs 2023-09-30T01:32:54.043 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:32:54.018+0000 7f23ade97700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.88:3302/0 2023-09-30T01:32:54.043 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:32:54.018+0000 7f23ade97700 1 -- 172.21.15.88:0/482904701 --> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f23a8140250 con 0x7f23a809d680 2023-09-30T01:32:54.043 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:32:54.018+0000 7f238f7fe700 1 -- 172.21.15.88:0/482904701 <== mon.4 v2:172.21.15.88:3302/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7f23a0005a00 con 0x7f23a809d680 2023-09-30T01:32:54.043 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:32:54.018+0000 7f238f7fe700 1 -- 172.21.15.88:0/482904701 <== mon.4 v2:172.21.15.88:3302/0 5 ==== osd_map(8..8 src has 1..8) v4 ==== 1411+0+0 (secure 0 0 0) 0x7f23a0013470 con 0x7f23a809d680 2023-09-30T01:32:54.043 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:32:54.018+0000 7f23ade97700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:32:54.043 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:32:54.018+0000 7f23ade97700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:32:54.043 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:32:54.018+0000 7f23ade97700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.88:3302/0 2023-09-30T01:32:54.043 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:32:54.018+0000 7f23ade97700 1 -- 172.21.15.88:0/482904701 --> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f23a8072c90 con 0x7f23a809d680 2023-09-30T01:32:54.043 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:32:54.022+0000 7f238f7fe700 1 -- 172.21.15.88:0/482904701 <== mon.4 v2:172.21.15.88:3302/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7f23a0020060 con 0x7f23a809d680 2023-09-30T01:32:54.044 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:32:54.022+0000 7f238f7fe700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:32:54.044 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:32:54.022+0000 7f238f7fe700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:32:54.090 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:32:54.086+0000 7f23ade97700 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2023-09-30T01:32:54.091 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:32:54.086+0000 7f23ade97700 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2023-09-30T01:32:54.091 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:32:54.086+0000 7f23ade97700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.88:3302/0 2023-09-30T01:32:54.091 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:32:54.086+0000 7f23ade97700 1 -- 172.21.15.88:0/482904701 --> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f23a80719f0 con 0x7f23a809d680 2023-09-30T01:32:54.091 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:32:54.090+0000 7f238f7fe700 1 -- 172.21.15.88:0/482904701 <== mon.4 v2:172.21.15.88:3302/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v8) v1 ==== 74+0+5560 (secure 0 0 0) 0x7f23a0017070 con 0x7f23a809d680 2023-09-30T01:32:54.091 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:32:54.090+0000 7f238f7fe700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2023-09-30T01:32:54.091 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:32:54.090+0000 7f238f7fe700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:32:54.092 INFO:teuthology.misc.health.smithi088.stdout: 2023-09-30T01:32:54.092 INFO:teuthology.misc.health.smithi088.stdout:{"epoch":8,"fsid":"650badb5-899b-4472-88a6-cfdccee24d52","created":"2023-09-30T01:32:51.511154+0000","modified":"2023-09-30T01:32:53.067458+0000","last_up_change":"0.000000","last_in_change":"2023-09-30T01:32:53.067458+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"1243f825-b99c-4546-a77d-90b8637698b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e9831b75-399b-4d6d-bf97-9ef104adfdb5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"fbc7be74-cd43-4c72-89c1-469c17055b85","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ca888e00-8f75-4dc4-b70c-6cd46880fe74","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"c8c73bd1-7674-4529-9104-8690e25af2fc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"a3994780-9d6a-4bcd-b718-09f9d8774b3d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-09-30T01:32:54.093 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:32:54.090+0000 7f23ade97700 10 monclient: shutdown 2023-09-30T01:32:54.093 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:32:54.090+0000 7f23ade97700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:32:54.093 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:32:54.090+0000 7f23ade97700 1 -- 172.21.15.88:0/482904701 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f23a809d680 msgr2=0x7f23a813ca20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:32:54.093 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:32:54.090+0000 7f23ade97700 1 --2- 172.21.15.88:0/482904701 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f23a809d680 0x7f23a813ca20 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f23a000ec80 tx=0x7f23a00091a0 comp rx=0 tx=0).stop 2023-09-30T01:32:54.093 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:32:54.090+0000 7f23ade97700 1 -- 172.21.15.88:0/482904701 shutdown_connections 2023-09-30T01:32:54.093 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:32:54.090+0000 7f23ade97700 1 --2- 172.21.15.88:0/482904701 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f23a809d680 0x7f23a813ca20 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:32:54.093 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:32:54.090+0000 7f23ade97700 1 --2- 172.21.15.88:0/482904701 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7f23a809b9a0 0x7f23a813c0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:32:54.093 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:32:54.090+0000 7f23ade97700 1 --2- 172.21.15.88:0/482904701 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f23a809a250 0x7f23a811ea70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:32:54.093 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:32:54.090+0000 7f23ade97700 1 -- 172.21.15.88:0/482904701 >> 172.21.15.88:0/482904701 conn(0x7f23a80637a0 msgr2=0x7f23a806d090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:32:54.094 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:32:54.090+0000 7f23ade97700 1 -- 172.21.15.88:0/482904701 shutdown_connections 2023-09-30T01:32:54.094 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:32:54.090+0000 7f23ade97700 1 -- 172.21.15.88:0/482904701 wait complete. 2023-09-30T01:32:54.103 DEBUG:teuthology.misc:0 of 6 OSDs are up 2023-09-30T01:32:54.136 INFO:tasks.ceph.mgr.x.smithi107.stderr:2023-09-30T01:32:54.133+0000 7f46b28dd000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-09-30T01:32:54.201 INFO:tasks.ceph.mgr.x.smithi107.stderr:2023-09-30T01:32:54.197+0000 7f46b28dd000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-09-30T01:32:54.253 INFO:tasks.ceph.mgr.x.smithi107.stderr:2023-09-30T01:32:54.249+0000 7f46b28dd000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-09-30T01:32:54.307 INFO:tasks.ceph.mgr.x.smithi107.stderr:2023-09-30T01:32:54.305+0000 7f46b28dd000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-09-30T01:32:54.582 INFO:tasks.ceph.mgr.x.smithi107.stderr:2023-09-30T01:32:54.577+0000 7f46b28dd000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-09-30T01:32:54.639 INFO:tasks.ceph.mgr.x.smithi107.stderr:2023-09-30T01:32:54.633+0000 7f46b28dd000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-09-30T01:32:54.703 INFO:tasks.ceph.mgr.x.smithi107.stderr:2023-09-30T01:32:54.697+0000 7f46b28dd000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-09-30T01:32:54.860 INFO:tasks.ceph.mgr.x.smithi107.stderr:2023-09-30T01:32:54.857+0000 7f46b28dd000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-09-30T01:32:54.925 INFO:tasks.ceph.mgr.x.smithi107.stderr:2023-09-30T01:32:54.921+0000 7f46b28dd000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-09-30T01:32:55.618 INFO:tasks.ceph.mgr.x.smithi107.stderr:2023-09-30T01:32:55.613+0000 7f46b28dd000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-09-30T01:32:55.672 INFO:tasks.ceph.mgr.x.smithi107.stderr:2023-09-30T01:32:55.669+0000 7f46901c2700 -1 log_channel(cluster) log [ERR] : Unhandled exception from module 'pg_autoscaler' while running on mgr.x: 'PgAutoscaler' object has no attribute 'noautoscale' 2023-09-30T01:32:55.672 INFO:tasks.ceph.mgr.x.smithi107.stderr:2023-09-30T01:32:55.669+0000 7f46901c2700 -1 pg_autoscaler.serve: 2023-09-30T01:32:55.672 INFO:tasks.ceph.mgr.x.smithi107.stderr:2023-09-30T01:32:55.669+0000 7f46901c2700 -1 Traceback (most recent call last): 2023-09-30T01:32:55.672 INFO:tasks.ceph.mgr.x.smithi107.stderr: File "/usr/share/ceph/mgr/pg_autoscaler/module.py", line 325, in serve 2023-09-30T01:32:55.673 INFO:tasks.ceph.mgr.x.smithi107.stderr: self._maybe_adjust() 2023-09-30T01:32:55.673 INFO:tasks.ceph.mgr.x.smithi107.stderr: File "/usr/share/ceph/mgr/pg_autoscaler/module.py", line 682, in _maybe_adjust 2023-09-30T01:32:55.673 INFO:tasks.ceph.mgr.x.smithi107.stderr: if self.noautoscale: 2023-09-30T01:32:55.673 INFO:tasks.ceph.mgr.x.smithi107.stderr:AttributeError: 'PgAutoscaler' object has no attribute 'noautoscale' 2023-09-30T01:32:55.673 INFO:tasks.ceph.mgr.x.smithi107.stderr: 2023-09-30T01:32:56.659 INFO:tasks.ceph.mgr.x.smithi107.stderr:2023-09-30T01:32:56.653+0000 7f469b9d9700 -1 mgr.server handle_report got status from non-daemon mon.d 2023-09-30T01:32:56.660 INFO:tasks.ceph.mgr.x.smithi107.stderr:2023-09-30T01:32:56.657+0000 7f469b9d9700 -1 mgr.server handle_report got status from non-daemon mon.b 2023-09-30T01:32:56.660 INFO:tasks.ceph.mgr.x.smithi107.stderr:2023-09-30T01:32:56.657+0000 7f469b9d9700 -1 mgr.server handle_report got status from non-daemon mon.e 2023-09-30T01:32:56.661 INFO:tasks.ceph.mgr.x.smithi107.stderr:2023-09-30T01:32:56.657+0000 7f469b9d9700 -1 mgr.server handle_report got status from non-daemon mon.g 2023-09-30T01:32:56.661 INFO:tasks.ceph.mgr.x.smithi107.stderr:2023-09-30T01:32:56.657+0000 7f469b9d9700 -1 mgr.server handle_report got status from non-daemon mon.c 2023-09-30T01:32:56.662 INFO:tasks.ceph.mgr.x.smithi107.stderr:2023-09-30T01:32:56.657+0000 7f469b9d9700 -1 mgr.server handle_report got status from non-daemon mon.i 2023-09-30T01:32:56.662 INFO:tasks.ceph.mgr.x.smithi107.stderr:2023-09-30T01:32:56.657+0000 7f469b9d9700 -1 mgr.server handle_report got status from non-daemon mon.h 2023-09-30T01:32:56.664 INFO:tasks.ceph.mgr.x.smithi107.stderr:2023-09-30T01:32:56.661+0000 7f469b9d9700 -1 mgr.server handle_report got status from non-daemon mon.f 2023-09-30T01:32:57.281 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-09-30T01:32:57.277+0000 7f2b191cd3c0 -1 osd.5 0 log_to_monitors true 2023-09-30T01:32:57.300 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-09-30T01:32:57.298+0000 7f7ba463d3c0 -1 osd.2 0 log_to_monitors true 2023-09-30T01:32:57.470 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-09-30T01:32:57.466+0000 7fd3d2cdd3c0 -1 osd.1 0 log_to_monitors true 2023-09-30T01:32:57.490 INFO:tasks.ceph.osd.3.smithi107.stderr:2023-09-30T01:32:57.485+0000 7fb9d02a03c0 -1 osd.3 0 log_to_monitors true 2023-09-30T01:32:57.569 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-09-30T01:32:57.566+0000 7f59b8c603c0 -1 osd.0 0 log_to_monitors true 2023-09-30T01:32:57.598 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-09-30T01:32:57.593+0000 7f6907df03c0 -1 osd.4 0 log_to_monitors true 2023-09-30T01:32:58.733 INFO:tasks.ceph.osd.3.smithi107.stderr:2023-09-30T01:32:58.729+0000 7fb9cd265700 -1 osd.3 0 waiting for initial osdmap 2023-09-30T01:32:58.737 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-09-30T01:32:58.733+0000 7f2b16192700 -1 osd.5 0 waiting for initial osdmap 2023-09-30T01:32:58.737 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-09-30T01:32:58.733+0000 7f6903590700 -1 osd.4 0 waiting for initial osdmap 2023-09-30T01:32:58.738 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-09-30T01:32:58.734+0000 7fd3cfca2700 -1 osd.1 0 waiting for initial osdmap 2023-09-30T01:32:58.738 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-09-30T01:32:58.734+0000 7f7b9fddd700 -1 osd.2 0 waiting for initial osdmap 2023-09-30T01:32:58.739 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-09-30T01:32:58.738+0000 7f59b4400700 -1 osd.0 0 waiting for initial osdmap 2023-09-30T01:32:58.741 INFO:tasks.ceph.osd.5.smithi107.stderr:2023-09-30T01:32:58.737+0000 7f2b0ef14700 -1 osd.5 10 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-09-30T01:32:58.742 INFO:tasks.ceph.osd.3.smithi107.stderr:2023-09-30T01:32:58.737+0000 7fb9c5fe7700 -1 osd.3 10 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-09-30T01:32:58.742 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-09-30T01:32:58.742+0000 7fd3c8a24700 -1 osd.1 10 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-09-30T01:32:58.743 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-09-30T01:32:58.742+0000 7f7b9a384700 -1 osd.2 10 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-09-30T01:32:58.743 INFO:tasks.ceph.osd.4.smithi107.stderr:2023-09-30T01:32:58.741+0000 7f68fdb37700 -1 osd.4 10 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-09-30T01:32:58.745 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-09-30T01:32:58.742+0000 7f59ae9a7700 -1 osd.0 10 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-09-30T01:33:00.707 DEBUG:teuthology.orchestra.run.smithi088:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-09-30T01:33:00.740 INFO:tasks.ceph.mon.d.smithi088.stderr:2023-09-30T01:33:00.738+0000 7fabfe1e4700 -1 log_channel(cluster) log [ERR] : Health check failed: Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale' (MGR_MODULE_ERROR) 2023-09-30T01:33:00.814 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:33:00.810+0000 7f9c36073700 10 monclient: get_monmap_and_config 2023-09-30T01:33:00.814 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:33:00.810+0000 7f9c36073700 10 monclient: build_initial_monmap 2023-09-30T01:33:00.814 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:33:00.810+0000 7f9c36073700 10 monclient: monmap: 2023-09-30T01:33:00.814 INFO:teuthology.misc.health.smithi088.stderr:epoch 0 2023-09-30T01:33:00.814 INFO:teuthology.misc.health.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:33:00.814 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2023-09-30T01:33:00.814539+0000 2023-09-30T01:33:00.814 INFO:teuthology.misc.health.smithi088.stderr:created 2023-09-30T01:33:00.814539+0000 2023-09-30T01:33:00.814 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 0 (unknown) 2023-09-30T01:33:00.814 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 3 2023-09-30T01:33:00.814 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a-legacy 2023-09-30T01:33:00.815 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.107:6789/0 mon.noname-f-legacy 2023-09-30T01:33:00.815 INFO:teuthology.misc.health.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.noname-b 2023-09-30T01:33:00.815 INFO:teuthology.misc.health.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.noname-g 2023-09-30T01:33:00.815 INFO:teuthology.misc.health.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.noname-c 2023-09-30T01:33:00.815 INFO:teuthology.misc.health.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.noname-h 2023-09-30T01:33:00.815 INFO:teuthology.misc.health.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.noname-d 2023-09-30T01:33:00.815 INFO:teuthology.misc.health.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.noname-i 2023-09-30T01:33:00.815 INFO:teuthology.misc.health.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.noname-e 2023-09-30T01:33:00.815 INFO:teuthology.misc.health.smithi088.stderr:9: v2:172.21.15.88:3300/0 mon.noname-a 2023-09-30T01:33:00.815 INFO:teuthology.misc.health.smithi088.stderr:10: v2:172.21.15.107:3300/0 mon.noname-f 2023-09-30T01:33:00.815 INFO:teuthology.misc.health.smithi088.stderr: 2023-09-30T01:33:00.819 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:33:00.810+0000 7f9c36073700 1 Processor -- start 2023-09-30T01:33:00.819 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:33:00.810+0000 7f9c36073700 1 -- start start 2023-09-30T01:33:00.819 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:33:00.810+0000 7f9c36073700 10 monclient: init 2023-09-30T01:33:00.819 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:33:00.810+0000 7f9c36073700 10 monclient: _reopen_session rank -1 2023-09-30T01:33:00.819 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:33:00.810+0000 7f9c36073700 10 monclient: _add_conns ranks=[1,10,0,2,6,4,8,9,7,3,5] 2023-09-30T01:33:00.819 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:33:00.810+0000 7f9c36073700 10 monclient(hunting): picked mon.noname-f-legacy con 0x7f9c300994e0 addr v1:172.21.15.107:6789/0 2023-09-30T01:33:00.819 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:33:00.810+0000 7f9c36073700 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f9c300998b0 0x7f9c30099d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:33:00.820 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:33:00.810+0000 7f9c36073700 10 monclient(hunting): picked mon.noname-f con 0x7f9c300998b0 addr v2:172.21.15.107:3300/0 2023-09-30T01:33:00.820 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:33:00.810+0000 7f9c36073700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f9c30138670 addr v1:172.21.15.88:6789/0 2023-09-30T01:33:00.820 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:33:00.814+0000 7f9c36073700 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c3004bb20 con 0x7f9c30138670 2023-09-30T01:33:00.820 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:33:00.814+0000 7f9c36073700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c30098060 con 0x7f9c300994e0 2023-09-30T01:33:00.820 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:33:00.814+0000 7f9c36073700 10 monclient(hunting): start opening mon connection 2023-09-30T01:33:00.820 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:33:00.814+0000 7f9c36073700 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9c30068370 con 0x7f9c300998b0 2023-09-30T01:33:00.820 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:33:00.814+0000 7f9c36073700 10 monclient(hunting): _renew_subs 2023-09-30T01:33:00.820 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:33:00.814+0000 7f9c36073700 10 monclient(hunting): authenticate will time out at 852.852056s 2023-09-30T01:33:00.820 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:33:00.814+0000 7f9c35872700 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f9c30138670 0x7f9c3013aa50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:54080/0 (socket says 172.21.15.88:54080) 2023-09-30T01:33:00.820 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:33:00.814+0000 7f9c35872700 1 -- 172.21.15.88:0/37131300 learned_addr learned my addr 172.21.15.88:0/37131300 (peer_addr_for_me v1:172.21.15.88:0/0) 2023-09-30T01:33:00.820 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:33:00.814+0000 7f9c34870700 1 --2- 172.21.15.88:0/37131300 >> v2:172.21.15.107:3300/0 conn(0x7f9c300998b0 0x7f9c30099d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:33:00.821 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:33:00.814+0000 7f9c34870700 10 monclient(hunting): get_auth_request con 0x7f9c300998b0 auth_method 0 2023-09-30T01:33:00.821 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:33:00.814+0000 7f9c34870700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:33:00.821 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:33:00.814+0000 7f9c34870700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:33:00.821 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:33:00.814+0000 7f9c34870700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:33:00.821 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:33:00.814+0000 7f9c34870700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:33:00.821 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:33:00.814+0000 7f9c34870700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:33:00.821 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:33:00.814+0000 7f9c34870700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:33:00.821 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:33:00.814+0000 7f9c27fff700 1 -- 172.21.15.88:0/37131300 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 385815153 0 0) 0x7f9c3004bb20 con 0x7f9c30138670 2023-09-30T01:33:00.821 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:33:00.814+0000 7f9c27fff700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:33:00.821 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:33:00.814+0000 7f9c27fff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:33:00.821 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:33:00.814+0000 7f9c27fff700 10 monclient(hunting): my global_id is 4221 2023-09-30T01:33:00.822 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:33:00.814+0000 7f9c27fff700 1 -- 172.21.15.88:0/37131300 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9c1c003570 con 0x7f9c30138670 2023-09-30T01:33:00.822 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:33:00.814+0000 7f9c34870700 10 monclient(hunting): handle_auth_done global_id 4294 payload 274 2023-09-30T01:33:00.822 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:33:00.814+0000 7f9c34870700 1 -- 172.21.15.88:0/37131300 >> v1:172.21.15.107:6789/0 conn(0x7f9c300994e0 legacy=0x7f9c301337a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:33:00.822 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:33:00.814+0000 7f9c34870700 1 -- 172.21.15.88:0/37131300 >> v1:172.21.15.88:6789/0 conn(0x7f9c30138670 legacy=0x7f9c3013aa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:33:00.822 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:33:00.814+0000 7f9c34870700 10 monclient: _finish_hunting 0 2023-09-30T01:33:00.822 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:33:00.814+0000 7f9c34870700 1 monclient: found mon.noname-f 2023-09-30T01:33:00.822 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:33:00.814+0000 7f9c34870700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:33:00.822 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:33:00.814+0000 7f9c34870700 10 monclient: _send_mon_message to mon.noname-f at v2:172.21.15.107:3300/0 2023-09-30T01:33:00.822 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:33:00.814+0000 7f9c34870700 1 -- 172.21.15.88:0/37131300 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c30133eb0 con 0x7f9c300998b0 2023-09-30T01:33:00.822 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:33:00.814+0000 7f9c34870700 1 --2- 172.21.15.88:0/37131300 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9c300998b0 0x7f9c30099d00 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f9c20009910 tx=0x7f9c200091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=eb91565ff739db85 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:33:00.822 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:33:00.814+0000 7f9c27fff700 1 -- 172.21.15.88:0/37131300 <== mon.1 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f9c2000bac0 con 0x7f9c300998b0 2023-09-30T01:33:00.822 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:33:00.814+0000 7f9c27fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:33:00.823 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:33:00.814+0000 7f9c27fff700 10 monclient: got monmap 1 from mon.noname-f (according to old e1) 2023-09-30T01:33:00.823 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:33:00.814+0000 7f9c27fff700 10 monclient: dump: 2023-09-30T01:33:00.823 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2023-09-30T01:33:00.823 INFO:teuthology.misc.health.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:33:00.823 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:00.823 INFO:teuthology.misc.health.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:00.823 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:33:00.823 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 3 2023-09-30T01:33:00.823 INFO:teuthology.misc.health.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:33:00.823 INFO:teuthology.misc.health.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:33:00.823 INFO:teuthology.misc.health.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:33:00.823 INFO:teuthology.misc.health.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:33:00.824 INFO:teuthology.misc.health.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:33:00.824 INFO:teuthology.misc.health.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:33:00.824 INFO:teuthology.misc.health.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:33:00.824 INFO:teuthology.misc.health.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:33:00.824 INFO:teuthology.misc.health.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:33:00.824 INFO:teuthology.misc.health.smithi088.stderr: 2023-09-30T01:33:00.824 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:33:00.814+0000 7f9c27fff700 10 monclient: _finish_auth 0 2023-09-30T01:33:00.824 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:33:00.814+0000 7f9c27fff700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:33:00.824 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:33:00.814+0000 7f9c36073700 5 monclient: authenticate success, global_id 4294 2023-09-30T01:33:00.824 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:33:00.814+0000 7f9c36073700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:33:00.824 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:33:00.814+0000 7f9c27fff700 1 -- 172.21.15.88:0/37131300 <== mon.1 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c2000bc40 con 0x7f9c300998b0 2023-09-30T01:33:00.824 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:33:00.814+0000 7f9c27fff700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:33:00.825 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:33:00.814+0000 7f9c36073700 10 monclient: get_monmap_and_config success 2023-09-30T01:33:00.825 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:33:00.814+0000 7f9c27fff700 1 -- 172.21.15.88:0/37131300 <== mon.1 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f9c2000bdb0 con 0x7f9c300998b0 2023-09-30T01:33:00.825 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:33:00.814+0000 7f9c36073700 4 set_mon_vals no callback set 2023-09-30T01:33:00.825 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:33:00.814+0000 7f9c27fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:33:00.825 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:33:00.814+0000 7f9c27fff700 10 monclient: got monmap 1 from mon.f (according to old e1) 2023-09-30T01:33:00.825 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:33:00.814+0000 7f9c27fff700 10 monclient: dump: 2023-09-30T01:33:00.825 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2023-09-30T01:33:00.825 INFO:teuthology.misc.health.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:33:00.825 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:00.825 INFO:teuthology.misc.health.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:00.825 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:33:00.825 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 3 2023-09-30T01:33:00.825 INFO:teuthology.misc.health.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:33:00.826 INFO:teuthology.misc.health.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:33:00.826 INFO:teuthology.misc.health.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:33:00.826 INFO:teuthology.misc.health.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:33:00.826 INFO:teuthology.misc.health.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:33:00.826 INFO:teuthology.misc.health.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:33:00.826 INFO:teuthology.misc.health.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:33:00.826 INFO:teuthology.misc.health.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:33:00.826 INFO:teuthology.misc.health.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:33:00.826 INFO:teuthology.misc.health.smithi088.stderr: 2023-09-30T01:33:00.826 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:33:00.814+0000 7f9c36073700 10 monclient: shutdown 2023-09-30T01:33:00.826 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:33:00.814+0000 7f9c36073700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:33:00.826 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:33:00.814+0000 7f9c36073700 1 -- 172.21.15.88:0/37131300 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9c300998b0 msgr2=0x7f9c30099d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:33:00.827 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:33:00.814+0000 7f9c36073700 1 --2- 172.21.15.88:0/37131300 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9c300998b0 0x7f9c30099d00 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f9c20009910 tx=0x7f9c200091a0 comp rx=0 tx=0).stop 2023-09-30T01:33:00.827 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:33:00.814+0000 7f9c36073700 1 -- 172.21.15.88:0/37131300 shutdown_connections 2023-09-30T01:33:00.827 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:33:00.814+0000 7f9c36073700 1 --2- 172.21.15.88:0/37131300 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9c300998b0 0x7f9c30099d00 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:00.827 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:33:00.814+0000 7f9c36073700 1 -- 172.21.15.88:0/37131300 >> 172.21.15.88:0/37131300 conn(0x7f9c300a7150 msgr2=0x7f9c300a3120 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:33:00.827 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:33:00.814+0000 7f9c36073700 1 -- 172.21.15.88:0/37131300 shutdown_connections 2023-09-30T01:33:00.827 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:33:00.814+0000 7f9c36073700 1 -- 172.21.15.88:0/37131300 wait complete. 2023-09-30T01:33:00.827 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:33:00.814+0000 7f9c36073700 10 monclient: build_initial_monmap 2023-09-30T01:33:00.827 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:33:00.814+0000 7f9c36073700 10 monclient: monmap: 2023-09-30T01:33:00.827 INFO:teuthology.misc.health.smithi088.stderr:epoch 0 2023-09-30T01:33:00.827 INFO:teuthology.misc.health.smithi088.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:33:00.827 INFO:teuthology.misc.health.smithi088.stderr:last_changed 0.000000 2023-09-30T01:33:00.827 INFO:teuthology.misc.health.smithi088.stderr:created 0.000000 2023-09-30T01:33:00.828 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 0 (unknown) 2023-09-30T01:33:00.828 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2023-09-30T01:33:00.828 INFO:teuthology.misc.health.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.noname-a 2023-09-30T01:33:00.828 INFO:teuthology.misc.health.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.noname-f 2023-09-30T01:33:00.828 INFO:teuthology.misc.health.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.noname-b 2023-09-30T01:33:00.828 INFO:teuthology.misc.health.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.noname-g 2023-09-30T01:33:00.828 INFO:teuthology.misc.health.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.noname-c 2023-09-30T01:33:00.828 INFO:teuthology.misc.health.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.noname-h 2023-09-30T01:33:00.828 INFO:teuthology.misc.health.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.noname-d 2023-09-30T01:33:00.828 INFO:teuthology.misc.health.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.noname-i 2023-09-30T01:33:00.828 INFO:teuthology.misc.health.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.noname-e 2023-09-30T01:33:00.828 INFO:teuthology.misc.health.smithi088.stderr: 2023-09-30T01:33:00.828 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:33:00.818+0000 7f9c36073700 1 Processor -- start 2023-09-30T01:33:00.829 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:33:00.818+0000 7f9c36073700 1 -- start start 2023-09-30T01:33:00.829 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:33:00.818+0000 7f9c36073700 10 monclient: init 2023-09-30T01:33:00.833 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:33:00.818+0000 7f9c36073700 10 monclient: _reopen_session rank -1 2023-09-30T01:33:00.834 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:33:00.818+0000 7f9c36073700 10 monclient: _add_conns ranks=[2,1,0,3,8,6,4,7,5] 2023-09-30T01:33:00.834 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:33:00.818+0000 7f9c36073700 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f9c300994e0 0x7f9c301343c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:33:00.834 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:33:00.818+0000 7f9c36073700 10 monclient(hunting): picked mon.noname-b con 0x7f9c300994e0 addr [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] 2023-09-30T01:33:00.834 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:33:00.818+0000 7f9c36073700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9c300998b0 0x7f9c30134900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:33:00.834 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:33:00.818+0000 7f9c36073700 10 monclient(hunting): picked mon.noname-f con 0x7f9c300998b0 addr [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] 2023-09-30T01:33:00.834 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:33:00.818+0000 7f9c36073700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9c30138670 0x7f9c30134e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:33:00.834 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:33:00.818+0000 7f9c36073700 10 monclient(hunting): picked mon.noname-a con 0x7f9c30138670 addr [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] 2023-09-30T01:33:00.834 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:33:00.818+0000 7f9c36073700 10 monclient(hunting): start opening mon connection 2023-09-30T01:33:00.834 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:33:00.818+0000 7f9c36073700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c301362a0 con 0x7f9c30138670 2023-09-30T01:33:00.834 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:33:00.818+0000 7f9c36073700 10 monclient(hunting): start opening mon connection 2023-09-30T01:33:00.834 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:33:00.818+0000 7f9c36073700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c301306e0 con 0x7f9c300998b0 2023-09-30T01:33:00.835 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:33:00.818+0000 7f9c36073700 10 monclient(hunting): start opening mon connection 2023-09-30T01:33:00.835 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:33:00.818+0000 7f9c36073700 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c30131170 con 0x7f9c300994e0 2023-09-30T01:33:00.835 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:33:00.818+0000 7f9c36073700 10 monclient(hunting): _renew_subs 2023-09-30T01:33:00.835 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:33:00.818+0000 7f9c36073700 10 monclient(hunting): authenticate will time out at 852.856945s 2023-09-30T01:33:00.835 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:33:00.818+0000 7f9c35872700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9c30138670 0x7f9c30134e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:33:00.835 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:33:00.818+0000 7f9c35071700 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f9c300994e0 0x7f9c301343c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:33:00.835 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:33:00.818+0000 7f9c35872700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9c30138670 0x7f9c30134e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:57736/0 (socket says 172.21.15.88:57736) 2023-09-30T01:33:00.835 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:33:00.818+0000 7f9c35872700 1 -- 172.21.15.88:0/2361314053 learned_addr learned my addr 172.21.15.88:0/2361314053 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-09-30T01:33:00.835 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:33:00.818+0000 7f9c35071700 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f9c300994e0 0x7f9c301343c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3301/0 says I am v2:172.21.15.88:52380/0 (socket says 172.21.15.88:52380) 2023-09-30T01:33:00.835 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:33:00.818+0000 7f9c35872700 10 monclient(hunting): get_auth_request con 0x7f9c30138670 auth_method 0 2023-09-30T01:33:00.835 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:33:00.818+0000 7f9c35872700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:33:00.835 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:33:00.818+0000 7f9c35872700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:33:00.836 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:33:00.818+0000 7f9c35872700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:33:00.836 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:33:00.818+0000 7f9c35071700 10 monclient(hunting): get_auth_request con 0x7f9c300994e0 auth_method 0 2023-09-30T01:33:00.836 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:33:00.818+0000 7f9c35071700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:33:00.836 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:33:00.818+0000 7f9c35071700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:33:00.836 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:33:00.818+0000 7f9c35071700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:33:00.836 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:33:00.818+0000 7f9c34870700 1 --2- 172.21.15.88:0/2361314053 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9c300998b0 0x7f9c30134900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:33:00.836 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:33:00.818+0000 7f9c34870700 10 monclient(hunting): get_auth_request con 0x7f9c300998b0 auth_method 0 2023-09-30T01:33:00.836 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:33:00.818+0000 7f9c34870700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:33:00.836 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:33:00.818+0000 7f9c34870700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:33:00.836 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:33:00.818+0000 7f9c34870700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:33:00.836 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:33:00.818+0000 7f9c35071700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:33:00.836 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:33:00.818+0000 7f9c35071700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:33:00.837 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:33:00.818+0000 7f9c35071700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:33:00.837 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:33:00.818+0000 7f9c35872700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:33:00.837 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:33:00.818+0000 7f9c35872700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:33:00.837 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:33:00.818+0000 7f9c35872700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:33:00.837 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:33:00.818+0000 7f9c34870700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:33:00.837 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:33:00.818+0000 7f9c34870700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:33:00.837 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:33:00.818+0000 7f9c34870700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:33:00.837 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:33:00.818+0000 7f9c35071700 10 monclient(hunting): handle_auth_done global_id 4196 payload 824 2023-09-30T01:33:00.837 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:33:00.818+0000 7f9c35071700 1 -- 172.21.15.88:0/2361314053 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9c300998b0 msgr2=0x7f9c30134900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:33:00.837 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:33:00.818+0000 7f9c35071700 1 --2- 172.21.15.88:0/2361314053 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9c300998b0 0x7f9c30134900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:00.837 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:33:00.818+0000 7f9c35071700 1 -- 172.21.15.88:0/2361314053 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9c30138670 msgr2=0x7f9c30134e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:33:00.837 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:33:00.818+0000 7f9c35071700 1 --2- 172.21.15.88:0/2361314053 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9c30138670 0x7f9c30134e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:00.838 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:33:00.818+0000 7f9c35071700 10 monclient: _finish_hunting 0 2023-09-30T01:33:00.838 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:33:00.818+0000 7f9c35071700 1 monclient: found mon.noname-b 2023-09-30T01:33:00.838 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:33:00.818+0000 7f9c35071700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:33:00.838 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:33:00.818+0000 7f9c35071700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.88:3301/0 2023-09-30T01:33:00.838 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:33:00.818+0000 7f9c35071700 1 -- 172.21.15.88:0/2361314053 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c20009420 con 0x7f9c300994e0 2023-09-30T01:33:00.838 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:33:00.818+0000 7f9c35071700 1 --2- 172.21.15.88:0/2361314053 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f9c300994e0 0x7f9c301343c0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f9c2800ccb0 tx=0x7f9c28001b00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:33:00.838 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:33:00.818+0000 7f9c257fa700 1 -- 172.21.15.88:0/2361314053 <== mon.2 v2:172.21.15.88:3301/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f9c2800ad10 con 0x7f9c300994e0 2023-09-30T01:33:00.838 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:33:00.818+0000 7f9c257fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:33:00.838 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:33:00.818+0000 7f9c257fa700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T01:33:00.838 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:33:00.818+0000 7f9c257fa700 10 monclient: dump: 2023-09-30T01:33:00.838 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2023-09-30T01:33:00.838 INFO:teuthology.misc.health.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:33:00.839 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:00.839 INFO:teuthology.misc.health.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:00.839 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:33:00.839 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 3 2023-09-30T01:33:00.839 INFO:teuthology.misc.health.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:33:00.839 INFO:teuthology.misc.health.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:33:00.839 INFO:teuthology.misc.health.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:33:00.839 INFO:teuthology.misc.health.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:33:00.839 INFO:teuthology.misc.health.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:33:00.840 INFO:teuthology.misc.health.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:33:00.840 INFO:teuthology.misc.health.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:33:00.840 INFO:teuthology.misc.health.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:33:00.840 INFO:teuthology.misc.health.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:33:00.840 INFO:teuthology.misc.health.smithi088.stderr: 2023-09-30T01:33:00.840 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:33:00.818+0000 7f9c257fa700 10 monclient: _finish_auth 0 2023-09-30T01:33:00.840 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:33:00.818+0000 7f9c257fa700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:33:00.840 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:33:00.818+0000 7f9c257fa700 1 -- 172.21.15.88:0/2361314053 <== mon.2 v2:172.21.15.88:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c2800ae90 con 0x7f9c300994e0 2023-09-30T01:33:00.840 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:33:00.818+0000 7f9c36073700 5 monclient: authenticate success, global_id 4196 2023-09-30T01:33:00.840 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:33:00.818+0000 7f9c36073700 10 monclient: _renew_subs 2023-09-30T01:33:00.840 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:33:00.818+0000 7f9c36073700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.88:3301/0 2023-09-30T01:33:00.840 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:33:00.818+0000 7f9c36073700 1 -- 172.21.15.88:0/2361314053 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c30097c70 con 0x7f9c300994e0 2023-09-30T01:33:00.841 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:33:00.818+0000 7f9c257fa700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:33:00.841 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:33:00.818+0000 7f9c257fa700 1 -- 172.21.15.88:0/2361314053 <== mon.2 v2:172.21.15.88:3301/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f9c280128a0 con 0x7f9c300994e0 2023-09-30T01:33:00.841 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:33:00.818+0000 7f9c257fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:33:00.841 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:33:00.818+0000 7f9c277fe700 4 set_mon_vals no callback set 2023-09-30T01:33:00.841 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:33:00.818+0000 7f9c257fa700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T01:33:00.841 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:33:00.818+0000 7f9c257fa700 10 monclient: dump: 2023-09-30T01:33:00.841 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2023-09-30T01:33:00.841 INFO:teuthology.misc.health.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:33:00.841 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:00.841 INFO:teuthology.misc.health.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:00.842 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:33:00.842 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 3 2023-09-30T01:33:00.842 INFO:teuthology.misc.health.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:33:00.842 INFO:teuthology.misc.health.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:33:00.842 INFO:teuthology.misc.health.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:33:00.842 INFO:teuthology.misc.health.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:33:00.842 INFO:teuthology.misc.health.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:33:00.842 INFO:teuthology.misc.health.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:33:00.842 INFO:teuthology.misc.health.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:33:00.842 INFO:teuthology.misc.health.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:33:00.842 INFO:teuthology.misc.health.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:33:00.842 INFO:teuthology.misc.health.smithi088.stderr: 2023-09-30T01:33:00.843 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:33:00.818+0000 7f9c36073700 10 monclient: _renew_subs 2023-09-30T01:33:00.843 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:33:00.818+0000 7f9c36073700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.88:3301/0 2023-09-30T01:33:00.843 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:33:00.818+0000 7f9c36073700 1 -- 172.21.15.88:0/2361314053 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c30097a40 con 0x7f9c300994e0 2023-09-30T01:33:00.843 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:33:00.818+0000 7f9c257fa700 1 -- 172.21.15.88:0/2361314053 <== mon.2 v2:172.21.15.88:3301/0 4 ==== mgrmap(e 4) v1 ==== 40088+0+0 (secure 0 0 0) 0x7f9c28012420 con 0x7f9c300994e0 2023-09-30T01:33:00.843 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:33:00.822+0000 7f9c257fa700 1 --2- 172.21.15.88:0/2361314053 >> v2:172.21.15.107:6812/29926 conn(0x7f9c140332d0 0x7f9c14035780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:33:00.843 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:33:00.822+0000 7f9c257fa700 1 -- 172.21.15.88:0/2361314053 <== mon.2 v2:172.21.15.88:3301/0 5 ==== osd_map(12..12 src has 1..12) v4 ==== 2763+0+0 (secure 0 0 0) 0x7f9c28043610 con 0x7f9c300994e0 2023-09-30T01:33:00.843 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:33:00.822+0000 7f9c36073700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:33:00.843 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:33:00.822+0000 7f9c34870700 1 --2- 172.21.15.88:0/2361314053 >> v2:172.21.15.107:6812/29926 conn(0x7f9c140332d0 0x7f9c14035780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:33:00.843 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:33:00.822+0000 7f9c36073700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:33:00.843 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:33:00.822+0000 7f9c36073700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.88:3301/0 2023-09-30T01:33:00.843 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:33:00.822+0000 7f9c36073700 1 -- 172.21.15.88:0/2361314053 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9c300719f0 con 0x7f9c300994e0 2023-09-30T01:33:00.843 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:33:00.822+0000 7f9c34870700 10 monclient: get_auth_request con 0x7f9c140332d0 auth_method 0 2023-09-30T01:33:00.844 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:33:00.822+0000 7f9c34870700 1 --2- 172.21.15.88:0/2361314053 >> v2:172.21.15.107:6812/29926 conn(0x7f9c140332d0 0x7f9c14035780 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f9c200091e0 tx=0x7f9c20017000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:33:00.844 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:33:00.826+0000 7f9c257fa700 1 -- 172.21.15.88:0/2361314053 <== mon.2 v2:172.21.15.88:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f9c2801a3a0 con 0x7f9c300994e0 2023-09-30T01:33:00.844 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:33:00.826+0000 7f9c257fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:33:00.844 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:33:00.826+0000 7f9c257fa700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:33:00.992 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:33:00.990+0000 7f9c36073700 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2023-09-30T01:33:00.992 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:33:00.990+0000 7f9c36073700 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2023-09-30T01:33:00.992 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:33:00.990+0000 7f9c36073700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.88:3301/0 2023-09-30T01:33:00.992 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:33:00.990+0000 7f9c36073700 1 -- 172.21.15.88:0/2361314053 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9c30137810 con 0x7f9c300994e0 2023-09-30T01:33:00.993 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:33:00.990+0000 7f9c257fa700 1 -- 172.21.15.88:0/2361314053 <== mon.2 v2:172.21.15.88:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v12) v1 ==== 74+0+6910 (secure 0 0 0) 0x7f9c280022b0 con 0x7f9c300994e0 2023-09-30T01:33:00.993 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:33:00.990+0000 7f9c257fa700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2023-09-30T01:33:00.993 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:33:00.990+0000 7f9c257fa700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:33:00.993 INFO:teuthology.misc.health.smithi088.stdout: 2023-09-30T01:33:00.994 INFO:teuthology.misc.health.smithi088.stdout:{"epoch":12,"fsid":"650badb5-899b-4472-88a6-cfdccee24d52","created":"2023-09-30T01:32:51.511154+0000","modified":"2023-09-30T01:33:00.753200+0000","last_up_change":"2023-09-30T01:32:59.732301+0000","last_in_change":"2023-09-30T01:32:53.067458+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"1243f825-b99c-4546-a77d-90b8637698b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6808","nonce":30177}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6809","nonce":30177}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6811","nonce":30177}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6810","nonce":30177}]},"public_addr":"172.21.15.88:6808/30177","cluster_addr":"172.21.15.88:6809/30177","heartbeat_back_addr":"172.21.15.88:6811/30177","heartbeat_front_addr":"172.21.15.88:6810/30177","state":["exists","up"]},{"osd":1,"uuid":"e9831b75-399b-4d6d-bf97-9ef104adfdb5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6804","nonce":30175}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6805","nonce":30175}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6807","nonce":30175}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6806","nonce":30175}]},"public_addr":"172.21.15.88:6804/30175","cluster_addr":"172.21.15.88:6805/30175","heartbeat_back_addr":"172.21.15.88:6807/30175","heartbeat_front_addr":"172.21.15.88:6806/30175","state":["exists","up"]},{"osd":2,"uuid":"fbc7be74-cd43-4c72-89c1-469c17055b85","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6800","nonce":30176}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6801","nonce":30176}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6803","nonce":30176}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6802","nonce":30176}]},"public_addr":"172.21.15.88:6800/30176","cluster_addr":"172.21.15.88:6801/30176","heartbeat_back_addr":"172.21.15.88:6803/30176","heartbeat_front_addr":"172.21.15.88:6802/30176","state":["exists","up"]},{"osd":3,"uuid":"ca888e00-8f75-4dc4-b70c-6cd46880fe74","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":30274}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6805","nonce":30274}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6807","nonce":30274}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":30274}]},"public_addr":"172.21.15.107:6804/30274","cluster_addr":"172.21.15.107:6805/30274","heartbeat_back_addr":"172.21.15.107:6807/30274","heartbeat_front_addr":"172.21.15.107:6806/30274","state":["exists","up"]},{"osd":4,"uuid":"c8c73bd1-7674-4529-9104-8690e25af2fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":30276}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6809","nonce":30276}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6811","nonce":30276}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":30276}]},"public_addr":"172.21.15.107:6808/30276","cluster_addr":"172.21.15.107:6809/30276","heartbeat_back_addr":"172.21.15.107:6811/30276","heartbeat_front_addr":"172.21.15.107:6810/30276","state":["exists","up"]},{"osd":5,"uuid":"a3994780-9d6a-4bcd-b718-09f9d8774b3d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":30275}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6801","nonce":30275}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6803","nonce":30275}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":30275}]},"public_addr":"172.21.15.107:6800/30275","cluster_addr":"172.21.15.107:6801/30275","heartbeat_back_addr":"172.21.15.107:6803/30275","heartbeat_front_addr":"172.21.15.107:6802/30275","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T01:32:58.527368+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T01:32:58.431507+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T01:32:58.347215+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T01:32:58.501409+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T01:32:58.565697+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T01:32:58.313349+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-09-30T01:33:00.996 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:33:00.994+0000 7f9c36073700 1 -- 172.21.15.88:0/2361314053 >> v2:172.21.15.107:6812/29926 conn(0x7f9c140332d0 msgr2=0x7f9c14035780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:33:00.997 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:33:00.994+0000 7f9c36073700 1 --2- 172.21.15.88:0/2361314053 >> v2:172.21.15.107:6812/29926 conn(0x7f9c140332d0 0x7f9c14035780 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f9c200091e0 tx=0x7f9c20017000 comp rx=0 tx=0).stop 2023-09-30T01:33:00.997 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:33:00.994+0000 7f9c36073700 10 monclient: shutdown 2023-09-30T01:33:00.997 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:33:00.994+0000 7f9c36073700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:33:00.997 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:33:00.994+0000 7f9c36073700 1 -- 172.21.15.88:0/2361314053 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f9c300994e0 msgr2=0x7f9c301343c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:33:00.997 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:33:00.994+0000 7f9c36073700 1 --2- 172.21.15.88:0/2361314053 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f9c300994e0 0x7f9c301343c0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f9c2800ccb0 tx=0x7f9c28001b00 comp rx=0 tx=0).stop 2023-09-30T01:33:00.997 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:33:00.994+0000 7f9c36073700 1 -- 172.21.15.88:0/2361314053 shutdown_connections 2023-09-30T01:33:00.997 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:33:00.994+0000 7f9c36073700 1 --2- 172.21.15.88:0/2361314053 >> v2:172.21.15.107:6812/29926 conn(0x7f9c140332d0 0x7f9c14035780 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:00.997 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:33:00.994+0000 7f9c36073700 1 --2- 172.21.15.88:0/2361314053 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9c30138670 0x7f9c30134e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:00.997 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:33:00.994+0000 7f9c36073700 1 --2- 172.21.15.88:0/2361314053 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9c300998b0 0x7f9c30134900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:00.997 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:33:00.994+0000 7f9c36073700 1 --2- 172.21.15.88:0/2361314053 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f9c300994e0 0x7f9c301343c0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:00.997 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:33:00.994+0000 7f9c36073700 1 -- 172.21.15.88:0/2361314053 >> 172.21.15.88:0/2361314053 conn(0x7f9c300a7150 msgr2=0x7f9c300690f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:33:00.998 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:33:00.994+0000 7f9c36073700 1 -- 172.21.15.88:0/2361314053 shutdown_connections 2023-09-30T01:33:00.998 INFO:teuthology.misc.health.smithi088.stderr:2023-09-30T01:33:00.994+0000 7f9c36073700 1 -- 172.21.15.88:0/2361314053 wait complete. 2023-09-30T01:33:01.006 DEBUG:teuthology.misc:6 of 6 OSDs are up 2023-09-30T01:33:01.007 INFO:tasks.ceph:Creating RBD pool 2023-09-30T01:33:01.007 DEBUG:teuthology.orchestra.run.smithi088:> sudo ceph --cluster ceph osd pool create rbd 8 2023-09-30T01:33:01.163 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:01.162+0000 7fa0c3fff700 10 monclient: get_monmap_and_config 2023-09-30T01:33:01.164 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:01.162+0000 7fa0c3fff700 10 monclient: build_initial_monmap 2023-09-30T01:33:01.164 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:01.162+0000 7fa0c3fff700 10 monclient: monmap: 2023-09-30T01:33:01.164 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2023-09-30T01:33:01.164 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:33:01.164 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:33:01.164576+0000 2023-09-30T01:33:01.164 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:33:01.164576+0000 2023-09-30T01:33:01.164 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2023-09-30T01:33:01.164 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:33:01.164 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a-legacy 2023-09-30T01:33:01.164 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.107:6789/0 mon.noname-f-legacy 2023-09-30T01:33:01.165 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.noname-b 2023-09-30T01:33:01.165 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.noname-g 2023-09-30T01:33:01.165 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.noname-c 2023-09-30T01:33:01.165 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.noname-h 2023-09-30T01:33:01.165 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.noname-d 2023-09-30T01:33:01.165 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.noname-i 2023-09-30T01:33:01.165 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.noname-e 2023-09-30T01:33:01.165 INFO:teuthology.orchestra.run.smithi088.stderr:9: v2:172.21.15.88:3300/0 mon.noname-a 2023-09-30T01:33:01.165 INFO:teuthology.orchestra.run.smithi088.stderr:10: v2:172.21.15.107:3300/0 mon.noname-f 2023-09-30T01:33:01.165 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:33:01.168 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:01.162+0000 7fa0c3fff700 1 Processor -- start 2023-09-30T01:33:01.168 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:01.162+0000 7fa0c3fff700 1 -- start start 2023-09-30T01:33:01.169 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:01.162+0000 7fa0c3fff700 10 monclient: init 2023-09-30T01:33:01.169 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:01.162+0000 7fa0c3fff700 10 monclient: _reopen_session rank -1 2023-09-30T01:33:01.169 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:01.162+0000 7fa0c3fff700 10 monclient: _add_conns ranks=[2,1,4,3,7,9,5,10,8,0,6] 2023-09-30T01:33:01.169 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:01.162+0000 7fa0c3fff700 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa0c40976f0 0x7fa0c4098de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:33:01.169 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:01.162+0000 7fa0c3fff700 10 monclient(hunting): picked mon.noname-b con 0x7fa0c40976f0 addr [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] 2023-09-30T01:33:01.169 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:01.162+0000 7fa0c3fff700 10 monclient(hunting): picked mon.noname-f-legacy con 0x7fa0c409ed80 addr v1:172.21.15.107:6789/0 2023-09-30T01:33:01.169 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:01.162+0000 7fa0c3fff700 1 --2- >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7fa0c409f860 0x7fa0c41377c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:33:01.169 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:01.162+0000 7fa0c3fff700 10 monclient(hunting): picked mon.noname-c con 0x7fa0c409f860 addr [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] 2023-09-30T01:33:01.169 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:01.162+0000 7fa0c3fff700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0c404bb20 con 0x7fa0c409ed80 2023-09-30T01:33:01.169 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:01.162+0000 7fa0c3fff700 10 monclient(hunting): start opening mon connection 2023-09-30T01:33:01.169 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:01.162+0000 7fa0c3fff700 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0c4099320 con 0x7fa0c40976f0 2023-09-30T01:33:01.170 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:01.162+0000 7fa0c3fff700 10 monclient(hunting): start opening mon connection 2023-09-30T01:33:01.170 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:01.162+0000 7fa0c3fff700 1 -- --> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] -- mon_getmap magic: 0 v1 -- 0x7fa0c4098400 con 0x7fa0c409f860 2023-09-30T01:33:01.170 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:01.162+0000 7fa0c3fff700 10 monclient(hunting): _renew_subs 2023-09-30T01:33:01.170 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:01.162+0000 7fa0c3fff700 10 monclient(hunting): authenticate will time out at 853.202963s 2023-09-30T01:33:01.170 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:01.162+0000 7fa0c37fe700 1 --2- >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7fa0c409f860 0x7fa0c41377c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:33:01.170 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:01.162+0000 7fa0c2ffd700 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa0c40976f0 0x7fa0c4098de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:33:01.170 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:01.162+0000 7fa0c2ffd700 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa0c40976f0 0x7fa0c4098de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3301/0 says I am v2:172.21.15.88:52384/0 (socket says 172.21.15.88:52384) 2023-09-30T01:33:01.170 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:01.162+0000 7fa0c37fe700 1 --2- >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7fa0c409f860 0x7fa0c41377c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3302/0 says I am v2:172.21.15.88:50342/0 (socket says 172.21.15.88:50342) 2023-09-30T01:33:01.170 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:01.162+0000 7fa0c2ffd700 1 -- 172.21.15.88:0/3156012342 learned_addr learned my addr 172.21.15.88:0/3156012342 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-09-30T01:33:01.170 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:01.162+0000 7fa0c2ffd700 10 monclient(hunting): get_auth_request con 0x7fa0c40976f0 auth_method 0 2023-09-30T01:33:01.170 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:01.162+0000 7fa0c2ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:33:01.171 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:01.162+0000 7fa0c2ffd700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:33:01.171 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:01.162+0000 7fa0c2ffd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:33:01.171 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:01.162+0000 7fa0c37fe700 10 monclient(hunting): get_auth_request con 0x7fa0c409f860 auth_method 0 2023-09-30T01:33:01.171 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:01.162+0000 7fa0c37fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:33:01.171 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:01.162+0000 7fa0c37fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:33:01.171 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:01.162+0000 7fa0c37fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:33:01.171 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:01.162+0000 7fa0c37fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:33:01.171 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:01.162+0000 7fa0c37fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:33:01.171 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:01.162+0000 7fa0c37fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:33:01.171 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:01.162+0000 7fa0c2ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:33:01.172 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:01.162+0000 7fa0c2ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:33:01.172 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:01.162+0000 7fa0c2ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:33:01.172 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:01.162+0000 7fa0c2ffd700 10 monclient(hunting): handle_auth_done global_id 4205 payload 274 2023-09-30T01:33:01.172 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:01.162+0000 7fa0c2ffd700 1 -- 172.21.15.88:0/3156012342 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7fa0c409f860 msgr2=0x7fa0c41377c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:33:01.172 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:01.162+0000 7fa0c2ffd700 1 --2- 172.21.15.88:0/3156012342 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7fa0c409f860 0x7fa0c41377c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:01.172 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:01.162+0000 7fa0c2ffd700 1 -- 172.21.15.88:0/3156012342 >> v1:172.21.15.107:6789/0 conn(0x7fa0c409ed80 legacy=0x7fa0c409f150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:33:01.172 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:01.162+0000 7fa0c2ffd700 10 monclient: _finish_hunting 0 2023-09-30T01:33:01.172 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:01.162+0000 7fa0c2ffd700 1 monclient: found mon.noname-b 2023-09-30T01:33:01.172 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:01.162+0000 7fa0c2ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:33:01.172 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:01.162+0000 7fa0c2ffd700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.88:3301/0 2023-09-30T01:33:01.172 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:01.162+0000 7fa0c2ffd700 1 -- 172.21.15.88:0/3156012342 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0c4098a30 con 0x7fa0c40976f0 2023-09-30T01:33:01.172 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:01.162+0000 7fa0c37fe700 1 --2- 172.21.15.88:0/3156012342 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7fa0c409f860 0x7fa0c41377c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:33:01.173 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:01.166+0000 7fa0c2ffd700 1 --2- 172.21.15.88:0/3156012342 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa0c40976f0 0x7fa0c4098de0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fa0bc009f40 tx=0x7fa0bc009840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a33bab06b7c94a3 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:33:01.173 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:01.166+0000 7fa0c1ffb700 1 -- 172.21.15.88:0/3156012342 <== mon.2 v2:172.21.15.88:3301/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7fa0bc00e070 con 0x7fa0c40976f0 2023-09-30T01:33:01.173 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:01.166+0000 7fa0c1ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:33:01.173 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:01.166+0000 7fa0c1ffb700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T01:33:01.173 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:01.166+0000 7fa0c1ffb700 10 monclient: dump: 2023-09-30T01:33:01.173 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:33:01.173 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:33:01.173 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:01.173 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:01.173 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:33:01.173 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:33:01.173 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:33:01.173 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:33:01.174 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:33:01.174 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:33:01.174 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:33:01.174 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:33:01.174 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:33:01.174 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:33:01.174 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:33:01.174 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:33:01.174 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:01.166+0000 7fa0c1ffb700 10 monclient: _finish_auth 0 2023-09-30T01:33:01.174 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:01.166+0000 7fa0c1ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:33:01.174 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:01.166+0000 7fa0c1ffb700 1 -- 172.21.15.88:0/3156012342 <== mon.2 v2:172.21.15.88:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa0bc007400 con 0x7fa0c40976f0 2023-09-30T01:33:01.174 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:01.166+0000 7fa0c1ffb700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:33:01.175 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:01.166+0000 7fa0c1ffb700 1 -- 172.21.15.88:0/3156012342 <== mon.2 v2:172.21.15.88:3301/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7fa0bc0092e0 con 0x7fa0c40976f0 2023-09-30T01:33:01.175 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:01.166+0000 7fa0c3fff700 5 monclient: authenticate success, global_id 4205 2023-09-30T01:33:01.175 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:01.166+0000 7fa0c3fff700 10 monclient: get_monmap_and_config success 2023-09-30T01:33:01.175 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:01.166+0000 7fa0c3fff700 4 set_mon_vals no callback set 2023-09-30T01:33:01.175 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:01.166+0000 7fa0c1ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:33:01.175 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:01.166+0000 7fa0c1ffb700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T01:33:01.175 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:01.166+0000 7fa0c1ffb700 10 monclient: dump: 2023-09-30T01:33:01.175 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:33:01.175 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:33:01.175 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:01.175 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:01.175 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:33:01.176 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:33:01.176 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:33:01.176 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:33:01.176 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:33:01.176 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:33:01.176 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:33:01.176 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:33:01.176 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:33:01.176 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:33:01.176 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:33:01.177 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:33:01.177 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:01.166+0000 7fa0c3fff700 10 monclient: shutdown 2023-09-30T01:33:01.177 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:01.166+0000 7fa0c3fff700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:33:01.177 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:01.166+0000 7fa0c3fff700 1 -- 172.21.15.88:0/3156012342 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa0c40976f0 msgr2=0x7fa0c4098de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:33:01.177 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:01.166+0000 7fa0c3fff700 1 --2- 172.21.15.88:0/3156012342 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa0c40976f0 0x7fa0c4098de0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fa0bc009f40 tx=0x7fa0bc009840 comp rx=0 tx=0).stop 2023-09-30T01:33:01.177 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:01.166+0000 7fa0c3fff700 1 -- 172.21.15.88:0/3156012342 shutdown_connections 2023-09-30T01:33:01.177 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:01.166+0000 7fa0c3fff700 1 --2- 172.21.15.88:0/3156012342 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7fa0c409f860 0x7fa0c41377c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:01.177 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:01.166+0000 7fa0c3fff700 1 --2- 172.21.15.88:0/3156012342 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa0c40976f0 0x7fa0c4098de0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:01.177 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:01.166+0000 7fa0c3fff700 1 -- 172.21.15.88:0/3156012342 >> 172.21.15.88:0/3156012342 conn(0x7fa0c40a0910 msgr2=0x7fa0c40a33c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:33:01.177 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:01.166+0000 7fa0c3fff700 1 -- 172.21.15.88:0/3156012342 shutdown_connections 2023-09-30T01:33:01.177 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:01.166+0000 7fa0c3fff700 1 -- 172.21.15.88:0/3156012342 wait complete. 2023-09-30T01:33:01.182 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:01.166+0000 7fa0c3fff700 10 monclient: build_initial_monmap 2023-09-30T01:33:01.182 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:01.166+0000 7fa0c3fff700 10 monclient: monmap: 2023-09-30T01:33:01.182 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2023-09-30T01:33:01.182 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:33:01.182 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 0.000000 2023-09-30T01:33:01.182 INFO:teuthology.orchestra.run.smithi088.stderr:created 0.000000 2023-09-30T01:33:01.182 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2023-09-30T01:33:01.183 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 1 2023-09-30T01:33:01.183 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.noname-a 2023-09-30T01:33:01.183 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.noname-f 2023-09-30T01:33:01.183 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.noname-b 2023-09-30T01:33:01.183 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.noname-g 2023-09-30T01:33:01.183 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.noname-c 2023-09-30T01:33:01.183 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.noname-h 2023-09-30T01:33:01.183 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.noname-d 2023-09-30T01:33:01.183 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.noname-i 2023-09-30T01:33:01.183 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.noname-e 2023-09-30T01:33:01.183 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:33:01.183 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:01.166+0000 7fa0c3fff700 1 Processor -- start 2023-09-30T01:33:01.184 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:01.166+0000 7fa0c3fff700 1 -- start start 2023-09-30T01:33:01.184 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:01.166+0000 7fa0c3fff700 10 monclient: init 2023-09-30T01:33:01.184 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:01.170+0000 7fa0c3fff700 10 monclient: _reopen_session rank -1 2023-09-30T01:33:01.184 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:01.170+0000 7fa0c3fff700 10 monclient: _add_conns ranks=[8,0,1,5,2,6,3,7,4] 2023-09-30T01:33:01.184 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:01.170+0000 7fa0c3fff700 1 --2- >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7fa0c40976f0 0x7fa0c408b6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:33:01.184 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:01.170+0000 7fa0c3fff700 10 monclient(hunting): picked mon.noname-e con 0x7fa0c40976f0 addr [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] 2023-09-30T01:33:01.184 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:01.170+0000 7fa0c3fff700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa0c409ed80 0x7fa0c408bc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:33:01.184 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:01.170+0000 7fa0c3fff700 10 monclient(hunting): picked mon.noname-a con 0x7fa0c409ed80 addr [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] 2023-09-30T01:33:01.184 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:01.170+0000 7fa0c3fff700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa0c409f860 0x7fa0c408c160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:33:01.184 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:01.170+0000 7fa0c3fff700 10 monclient(hunting): picked mon.noname-f con 0x7fa0c409f860 addr [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] 2023-09-30T01:33:01.184 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:01.170+0000 7fa0c3fff700 10 monclient(hunting): start opening mon connection 2023-09-30T01:33:01.184 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:01.170+0000 7fa0c3fff700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0c40962d0 con 0x7fa0c409ed80 2023-09-30T01:33:01.185 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:01.170+0000 7fa0c3fff700 10 monclient(hunting): start opening mon connection 2023-09-30T01:33:01.185 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:01.170+0000 7fa0c3fff700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0c4097d80 con 0x7fa0c409f860 2023-09-30T01:33:01.185 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:01.170+0000 7fa0c3fff700 10 monclient(hunting): start opening mon connection 2023-09-30T01:33:01.185 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:01.170+0000 7fa0c3fff700 1 -- --> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] -- mon_getmap magic: 0 v1 -- 0x7fa0c408eff0 con 0x7fa0c40976f0 2023-09-30T01:33:01.185 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:01.170+0000 7fa0c3fff700 10 monclient(hunting): _renew_subs 2023-09-30T01:33:01.185 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:01.170+0000 7fa0c3fff700 10 monclient(hunting): authenticate will time out at 853.208652s 2023-09-30T01:33:01.185 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:01.170+0000 7fa0c2ffd700 1 --2- >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7fa0c40976f0 0x7fa0c408b6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:33:01.185 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:01.170+0000 7fa0c27fc700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa0c409ed80 0x7fa0c408bc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:33:01.185 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:01.170+0000 7fa0c2ffd700 1 --2- >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7fa0c40976f0 0x7fa0c408b6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3304/0 says I am v2:172.21.15.88:36616/0 (socket says 172.21.15.88:36616) 2023-09-30T01:33:01.185 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:01.170+0000 7fa0c2ffd700 1 -- 172.21.15.88:0/2088123284 learned_addr learned my addr 172.21.15.88:0/2088123284 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-09-30T01:33:01.185 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:01.170+0000 7fa0c2ffd700 10 monclient(hunting): get_auth_request con 0x7fa0c40976f0 auth_method 0 2023-09-30T01:33:01.185 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:01.170+0000 7fa0c2ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:33:01.186 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:01.170+0000 7fa0c2ffd700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:33:01.186 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:01.170+0000 7fa0c2ffd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:33:01.186 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:01.170+0000 7fa0c27fc700 10 monclient(hunting): get_auth_request con 0x7fa0c409ed80 auth_method 0 2023-09-30T01:33:01.186 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:01.170+0000 7fa0c27fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:33:01.186 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:01.170+0000 7fa0c27fc700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:33:01.186 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:01.170+0000 7fa0c27fc700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:33:01.186 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:01.170+0000 7fa0c37fe700 1 --2- 172.21.15.88:0/2088123284 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa0c409f860 0x7fa0c408c160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:33:01.186 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:01.170+0000 7fa0c37fe700 10 monclient(hunting): get_auth_request con 0x7fa0c409f860 auth_method 0 2023-09-30T01:33:01.186 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:01.170+0000 7fa0c37fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:33:01.186 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:01.170+0000 7fa0c37fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:33:01.186 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:01.170+0000 7fa0c37fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:33:01.186 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:01.170+0000 7fa0c27fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:33:01.187 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:01.170+0000 7fa0c27fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:33:01.187 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:01.170+0000 7fa0c27fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:33:01.187 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:01.170+0000 7fa0c2ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:33:01.187 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:01.170+0000 7fa0c2ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:33:01.187 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:01.170+0000 7fa0c2ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:33:01.187 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:01.170+0000 7fa0c37fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:33:01.187 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:01.170+0000 7fa0c37fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:33:01.187 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:01.170+0000 7fa0c37fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:33:01.187 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:01.170+0000 7fa0c27fc700 10 monclient(hunting): handle_auth_done global_id 4239 payload 824 2023-09-30T01:33:01.187 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:01.170+0000 7fa0c27fc700 1 -- 172.21.15.88:0/2088123284 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7fa0c40976f0 msgr2=0x7fa0c408b6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:33:01.187 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:01.170+0000 7fa0c27fc700 1 --2- 172.21.15.88:0/2088123284 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7fa0c40976f0 0x7fa0c408b6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:01.187 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:01.170+0000 7fa0c27fc700 1 -- 172.21.15.88:0/2088123284 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa0c409f860 msgr2=0x7fa0c408c160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:33:01.187 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:01.170+0000 7fa0c27fc700 1 --2- 172.21.15.88:0/2088123284 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa0c409f860 0x7fa0c408c160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:01.188 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:01.170+0000 7fa0c27fc700 10 monclient: _finish_hunting 0 2023-09-30T01:33:01.188 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:01.170+0000 7fa0c27fc700 1 monclient: found mon.noname-a 2023-09-30T01:33:01.188 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:01.170+0000 7fa0c27fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:33:01.188 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:01.170+0000 7fa0c27fc700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.88:3300/0 2023-09-30T01:33:01.188 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:01.170+0000 7fa0c27fc700 1 -- 172.21.15.88:0/2088123284 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0b4009420 con 0x7fa0c409ed80 2023-09-30T01:33:01.188 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:01.170+0000 7fa0c27fc700 1 --2- 172.21.15.88:0/2088123284 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa0c409ed80 0x7fa0c408bc20 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fa0ac00c590 tx=0x7fa0ac007ec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:33:01.188 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:01.170+0000 7fa0baffd700 1 -- 172.21.15.88:0/2088123284 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7fa0ac001120 con 0x7fa0c409ed80 2023-09-30T01:33:01.188 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:01.170+0000 7fa0baffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:33:01.188 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:01.170+0000 7fa0baffd700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T01:33:01.188 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:01.170+0000 7fa0baffd700 10 monclient: dump: 2023-09-30T01:33:01.188 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:33:01.189 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:33:01.189 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:01.189 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:01.189 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:33:01.189 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:33:01.189 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:33:01.189 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:33:01.189 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:33:01.189 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:33:01.189 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:33:01.189 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:33:01.189 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:33:01.190 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:33:01.190 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:33:01.190 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:33:01.190 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:01.170+0000 7fa0baffd700 10 monclient: _finish_auth 0 2023-09-30T01:33:01.190 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:01.170+0000 7fa0baffd700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:33:01.190 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:01.170+0000 7fa0baffd700 1 -- 172.21.15.88:0/2088123284 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa0ac011a10 con 0x7fa0c409ed80 2023-09-30T01:33:01.190 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:01.170+0000 7fa0c3fff700 5 monclient: authenticate success, global_id 4239 2023-09-30T01:33:01.190 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:01.170+0000 7fa0baffd700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:33:01.190 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:01.170+0000 7fa0baffd700 1 -- 172.21.15.88:0/2088123284 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7fa0ac0102e0 con 0x7fa0c409ed80 2023-09-30T01:33:01.190 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:01.170+0000 7fa0baffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:33:01.190 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:01.170+0000 7fa0baffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:33:01.190 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:01.170+0000 7fa0baffd700 10 monclient: dump: 2023-09-30T01:33:01.191 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:33:01.191 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:33:01.191 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:01.191 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:01.191 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:33:01.191 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:33:01.191 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:33:01.191 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:33:01.191 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:33:01.191 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:33:01.191 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:33:01.191 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:33:01.192 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:33:01.192 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:33:01.192 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:33:01.192 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:33:01.192 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:01.170+0000 7fa0c3fff700 10 monclient: _renew_subs 2023-09-30T01:33:01.192 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:01.170+0000 7fa0c17fa700 4 set_mon_vals no callback set 2023-09-30T01:33:01.192 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:01.170+0000 7fa0c3fff700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.88:3300/0 2023-09-30T01:33:01.192 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:01.170+0000 7fa0c3fff700 1 -- 172.21.15.88:0/2088123284 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa0bc009a80 con 0x7fa0c409ed80 2023-09-30T01:33:01.192 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:01.170+0000 7fa0c3fff700 10 monclient: _renew_subs 2023-09-30T01:33:01.192 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:01.170+0000 7fa0c3fff700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.88:3300/0 2023-09-30T01:33:01.192 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:01.170+0000 7fa0c3fff700 1 -- 172.21.15.88:0/2088123284 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0c408ed10 con 0x7fa0c409ed80 2023-09-30T01:33:01.193 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:01.170+0000 7fa0baffd700 1 -- 172.21.15.88:0/2088123284 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 4) v1 ==== 40088+0+0 (secure 0 0 0) 0x7fa0ac011b80 con 0x7fa0c409ed80 2023-09-30T01:33:01.193 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:01.170+0000 7fa0baffd700 1 --2- 172.21.15.88:0/2088123284 >> v2:172.21.15.107:6812/29926 conn(0x7fa0a40332d0 0x7fa0a4035780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:33:01.193 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:01.170+0000 7fa0baffd700 1 -- 172.21.15.88:0/2088123284 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(12..12 src has 1..12) v4 ==== 2763+0+0 (secure 0 0 0) 0x7fa0ac041f90 con 0x7fa0c409ed80 2023-09-30T01:33:01.193 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:01.170+0000 7fa0c3fff700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:33:01.193 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:01.170+0000 7fa0c3fff700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:33:01.193 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:01.170+0000 7fa0c3fff700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.88:3300/0 2023-09-30T01:33:01.193 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:01.170+0000 7fa0c3fff700 1 -- 172.21.15.88:0/2088123284 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa0c4071960 con 0x7fa0c409ed80 2023-09-30T01:33:01.193 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:01.174+0000 7fa0c2ffd700 1 --2- 172.21.15.88:0/2088123284 >> v2:172.21.15.107:6812/29926 conn(0x7fa0a40332d0 0x7fa0a4035780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:33:01.193 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:01.174+0000 7fa0c2ffd700 10 monclient: get_auth_request con 0x7fa0a40332d0 auth_method 0 2023-09-30T01:33:01.193 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:01.174+0000 7fa0c2ffd700 1 --2- 172.21.15.88:0/2088123284 >> v2:172.21.15.107:6812/29926 conn(0x7fa0a40332d0 0x7fa0a4035780 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fa0bc009880 tx=0x7fa0bc018000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:33:01.193 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:01.178+0000 7fa0baffd700 1 -- 172.21.15.88:0/2088123284 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fa0ac017030 con 0x7fa0c409ed80 2023-09-30T01:33:01.193 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:01.178+0000 7fa0baffd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:33:01.194 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:01.178+0000 7fa0baffd700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:33:01.348 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:01.346+0000 7fa0c3fff700 10 start_mon_command cmd=[{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2023-09-30T01:33:01.349 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:01.346+0000 7fa0c3fff700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2023-09-30T01:33:01.349 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:01.346+0000 7fa0c3fff700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.88:3300/0 2023-09-30T01:33:01.349 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:01.346+0000 7fa0c3fff700 1 -- 172.21.15.88:0/2088123284 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7fa0c4067af0 con 0x7fa0c409ed80 2023-09-30T01:33:02.270 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:02.266+0000 7fa0baffd700 1 -- 172.21.15.88:0/2088123284 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}]=0 pool 'rbd' created v13) v1 ==== 109+0+0 (secure 0 0 0) 0x7fa0ac017020 con 0x7fa0c409ed80 2023-09-30T01:33:02.270 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:02.266+0000 7fa0baffd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2023-09-30T01:33:02.270 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:02.266+0000 7fa0baffd700 10 monclient: _finish_command 2 = system:0 pool 'rbd' created 2023-09-30T01:33:02.273 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:02.270+0000 7fa0c3fff700 1 -- 172.21.15.88:0/2088123284 >> v2:172.21.15.107:6812/29926 conn(0x7fa0a40332d0 msgr2=0x7fa0a4035780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:33:02.273 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:02.270+0000 7fa0c3fff700 1 --2- 172.21.15.88:0/2088123284 >> v2:172.21.15.107:6812/29926 conn(0x7fa0a40332d0 0x7fa0a4035780 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fa0bc009880 tx=0x7fa0bc018000 comp rx=0 tx=0).stop 2023-09-30T01:33:02.273 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:02.270+0000 7fa0c3fff700 10 monclient: shutdown 2023-09-30T01:33:02.273 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:02.270+0000 7fa0c3fff700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:33:02.273 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:02.270+0000 7fa0c3fff700 1 -- 172.21.15.88:0/2088123284 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa0c409ed80 msgr2=0x7fa0c408bc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:33:02.273 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:02.270+0000 7fa0c3fff700 1 --2- 172.21.15.88:0/2088123284 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa0c409ed80 0x7fa0c408bc20 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fa0ac00c590 tx=0x7fa0ac007ec0 comp rx=0 tx=0).stop 2023-09-30T01:33:02.274 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:02.270+0000 7fa0c3fff700 1 -- 172.21.15.88:0/2088123284 shutdown_connections 2023-09-30T01:33:02.274 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:02.270+0000 7fa0c3fff700 1 --2- 172.21.15.88:0/2088123284 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa0c409f860 0x7fa0c408c160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:02.274 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:02.270+0000 7fa0c3fff700 1 --2- 172.21.15.88:0/2088123284 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa0c409ed80 0x7fa0c408bc20 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:02.274 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:02.270+0000 7fa0c3fff700 1 --2- 172.21.15.88:0/2088123284 >> v2:172.21.15.107:6812/29926 conn(0x7fa0a40332d0 0x7fa0a4035780 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:02.274 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:02.270+0000 7fa0c3fff700 1 --2- 172.21.15.88:0/2088123284 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7fa0c40976f0 0x7fa0c408b6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:02.274 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:02.270+0000 7fa0c3fff700 1 -- 172.21.15.88:0/2088123284 >> 172.21.15.88:0/2088123284 conn(0x7fa0c40a0910 msgr2=0x7fa0c40a2ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:33:02.274 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:02.270+0000 7fa0c3fff700 1 -- 172.21.15.88:0/2088123284 shutdown_connections 2023-09-30T01:33:02.274 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:02.270+0000 7fa0c3fff700 1 -- 172.21.15.88:0/2088123284 wait complete. 2023-09-30T01:33:02.274 INFO:teuthology.orchestra.run.smithi088.stderr:pool 'rbd' created 2023-09-30T01:33:02.286 DEBUG:teuthology.orchestra.run.smithi088:> sudo ceph --cluster ceph osd pool application enable rbd rbd --yes-i-really-mean-it 2023-09-30T01:33:02.391 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:02.390+0000 7f0023fff700 10 monclient: get_monmap_and_config 2023-09-30T01:33:02.391 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:02.390+0000 7f0023fff700 10 monclient: build_initial_monmap 2023-09-30T01:33:02.391 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:02.390+0000 7f0023fff700 10 monclient: monmap: 2023-09-30T01:33:02.391 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2023-09-30T01:33:02.391 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:33:02.392 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:33:02.392051+0000 2023-09-30T01:33:02.392 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:33:02.392051+0000 2023-09-30T01:33:02.392 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2023-09-30T01:33:02.392 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:33:02.392 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a-legacy 2023-09-30T01:33:02.392 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.107:6789/0 mon.noname-f-legacy 2023-09-30T01:33:02.392 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.noname-b 2023-09-30T01:33:02.392 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.noname-g 2023-09-30T01:33:02.392 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.noname-c 2023-09-30T01:33:02.392 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.noname-h 2023-09-30T01:33:02.392 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.noname-d 2023-09-30T01:33:02.392 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.noname-i 2023-09-30T01:33:02.393 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.noname-e 2023-09-30T01:33:02.393 INFO:teuthology.orchestra.run.smithi088.stderr:9: v2:172.21.15.88:3300/0 mon.noname-a 2023-09-30T01:33:02.393 INFO:teuthology.orchestra.run.smithi088.stderr:10: v2:172.21.15.107:3300/0 mon.noname-f 2023-09-30T01:33:02.393 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:33:02.395 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:02.390+0000 7f0023fff700 1 Processor -- start 2023-09-30T01:33:02.395 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:02.390+0000 7f0023fff700 1 -- start start 2023-09-30T01:33:02.395 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:02.390+0000 7f0023fff700 10 monclient: init 2023-09-30T01:33:02.395 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:02.390+0000 7f0023fff700 10 monclient: _reopen_session rank -1 2023-09-30T01:33:02.395 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:02.390+0000 7f0023fff700 10 monclient: _add_conns ranks=[4,10,1,9,0,3,5,2,6,8,7] 2023-09-30T01:33:02.395 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:02.390+0000 7f0023fff700 1 --2- >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f002406ea70 0x7f002406c910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:33:02.395 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:02.390+0000 7f0023fff700 10 monclient(hunting): picked mon.noname-c con 0x7f002406ea70 addr [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] 2023-09-30T01:33:02.396 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:02.390+0000 7f0023fff700 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f002406ee40 0x7f002409d4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:33:02.396 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:02.390+0000 7f0023fff700 10 monclient(hunting): picked mon.noname-f con 0x7f002406ee40 addr v2:172.21.15.107:3300/0 2023-09-30T01:33:02.396 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:02.390+0000 7f0023fff700 10 monclient(hunting): picked mon.noname-f-legacy con 0x7f002409da30 addr v1:172.21.15.107:6789/0 2023-09-30T01:33:02.396 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:02.390+0000 7f0023fff700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f002404bb20 con 0x7f002409da30 2023-09-30T01:33:02.396 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:02.390+0000 7f0023fff700 10 monclient(hunting): start opening mon connection 2023-09-30T01:33:02.396 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:02.390+0000 7f0023fff700 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f002406c5f0 con 0x7f002406ee40 2023-09-30T01:33:02.396 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:02.390+0000 7f0023fff700 10 monclient(hunting): start opening mon connection 2023-09-30T01:33:02.396 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:02.390+0000 7f0023fff700 1 -- --> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] -- mon_getmap magic: 0 v1 -- 0x7f002406bf30 con 0x7f002406ea70 2023-09-30T01:33:02.396 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:02.390+0000 7f0023fff700 10 monclient(hunting): _renew_subs 2023-09-30T01:33:02.396 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:02.390+0000 7f0023fff700 10 monclient(hunting): authenticate will time out at 854.431255s 2023-09-30T01:33:02.396 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:02.390+0000 7f0022ffd700 1 --2- >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f002406ea70 0x7f002406c910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:33:02.397 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:02.390+0000 7f0022ffd700 1 --2- >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f002406ea70 0x7f002406c910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3302/0 says I am v2:172.21.15.88:50344/0 (socket says 172.21.15.88:50344) 2023-09-30T01:33:02.397 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:02.390+0000 7f0022ffd700 1 -- 172.21.15.88:0/2793502255 learned_addr learned my addr 172.21.15.88:0/2793502255 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-09-30T01:33:02.397 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:02.390+0000 7f0022ffd700 10 monclient(hunting): get_auth_request con 0x7f002406ea70 auth_method 0 2023-09-30T01:33:02.397 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:02.390+0000 7f0022ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:33:02.397 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:02.390+0000 7f0022ffd700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:33:02.397 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:02.390+0000 7f0022ffd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:33:02.397 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:02.390+0000 7f001a7fc700 1 --2- 172.21.15.88:0/2793502255 >> v2:172.21.15.107:3300/0 conn(0x7f002406ee40 0x7f002409d4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:33:02.397 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:02.390+0000 7f0022ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:33:02.397 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:02.390+0000 7f0022ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:33:02.398 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:02.390+0000 7f0022ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:33:02.398 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:02.390+0000 7f001a7fc700 10 monclient(hunting): get_auth_request con 0x7f002406ee40 auth_method 0 2023-09-30T01:33:02.398 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:02.390+0000 7f001a7fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:33:02.398 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:02.390+0000 7f001a7fc700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:33:02.398 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:02.390+0000 7f001a7fc700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:33:02.398 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:02.390+0000 7f001a7fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:33:02.398 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:02.390+0000 7f001a7fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:33:02.398 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:02.390+0000 7f001a7fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:33:02.398 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:02.394+0000 7f0022ffd700 10 monclient(hunting): handle_auth_done global_id 4225 payload 274 2023-09-30T01:33:02.398 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:02.394+0000 7f0022ffd700 1 -- 172.21.15.88:0/2793502255 >> v2:172.21.15.107:3300/0 conn(0x7f002406ee40 msgr2=0x7f002409d4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:33:02.398 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:02.394+0000 7f0022ffd700 1 --2- 172.21.15.88:0/2793502255 >> v2:172.21.15.107:3300/0 conn(0x7f002406ee40 0x7f002409d4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:02.398 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:02.394+0000 7f0022ffd700 1 -- 172.21.15.88:0/2793502255 >> v1:172.21.15.107:6789/0 conn(0x7f002409da30 legacy=0x7f0024136400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:33:02.399 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:02.394+0000 7f0022ffd700 10 monclient: _finish_hunting 0 2023-09-30T01:33:02.399 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:02.394+0000 7f0022ffd700 1 monclient: found mon.noname-c 2023-09-30T01:33:02.399 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:02.394+0000 7f0022ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:33:02.399 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:02.394+0000 7f0022ffd700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.88:3302/0 2023-09-30T01:33:02.399 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:02.394+0000 7f0022ffd700 1 -- 172.21.15.88:0/2793502255 --> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f002406ce50 con 0x7f002406ea70 2023-09-30T01:33:02.399 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:02.394+0000 7f0022ffd700 1 --2- 172.21.15.88:0/2793502255 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f002406ea70 0x7f002406c910 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f001400ab30 tx=0x7f001400a3c0 comp rx=0 tx=0).ready entity=mon.4 client_cookie=901f8bd04dac7b26 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:33:02.404 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:02.394+0000 7f00227fc700 1 -- 172.21.15.88:0/2793502255 <== mon.4 v2:172.21.15.88:3302/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f001400ccb0 con 0x7f002406ea70 2023-09-30T01:33:02.404 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:02.394+0000 7f00227fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:33:02.404 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:02.394+0000 7f00227fc700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T01:33:02.404 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:02.394+0000 7f00227fc700 10 monclient: dump: 2023-09-30T01:33:02.404 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:33:02.404 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:33:02.404 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:02.405 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:02.405 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:33:02.405 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:33:02.405 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:33:02.405 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:33:02.405 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:33:02.405 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:33:02.405 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:33:02.405 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:33:02.405 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:33:02.405 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:33:02.405 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:33:02.405 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:33:02.406 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:02.394+0000 7f00227fc700 10 monclient: _finish_auth 0 2023-09-30T01:33:02.406 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:02.394+0000 7f00227fc700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:33:02.406 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:02.394+0000 7f0023fff700 5 monclient: authenticate success, global_id 4225 2023-09-30T01:33:02.406 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:02.394+0000 7f00227fc700 1 -- 172.21.15.88:0/2793502255 <== mon.4 v2:172.21.15.88:3302/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f001400ce30 con 0x7f002406ea70 2023-09-30T01:33:02.406 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:02.394+0000 7f0023fff700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:33:02.406 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:02.394+0000 7f00227fc700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:33:02.406 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:02.394+0000 7f0023fff700 10 monclient: get_monmap_and_config success 2023-09-30T01:33:02.406 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:02.394+0000 7f0023fff700 4 set_mon_vals no callback set 2023-09-30T01:33:02.406 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:02.394+0000 7f00227fc700 1 -- 172.21.15.88:0/2793502255 <== mon.4 v2:172.21.15.88:3302/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f0014005c10 con 0x7f002406ea70 2023-09-30T01:33:02.406 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:02.394+0000 7f00227fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:33:02.406 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:02.394+0000 7f00227fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T01:33:02.406 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:02.394+0000 7f00227fc700 10 monclient: dump: 2023-09-30T01:33:02.407 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:33:02.407 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:33:02.407 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:02.407 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:02.407 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:33:02.407 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:33:02.407 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:33:02.407 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:33:02.407 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:33:02.407 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:33:02.407 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:33:02.407 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:33:02.408 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:33:02.408 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:33:02.408 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:33:02.408 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:33:02.408 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:02.394+0000 7f0023fff700 10 monclient: shutdown 2023-09-30T01:33:02.408 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:02.394+0000 7f0023fff700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:33:02.408 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:02.394+0000 7f0023fff700 1 -- 172.21.15.88:0/2793502255 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f002406ea70 msgr2=0x7f002406c910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:33:02.408 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:02.394+0000 7f0023fff700 1 --2- 172.21.15.88:0/2793502255 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f002406ea70 0x7f002406c910 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f001400ab30 tx=0x7f001400a3c0 comp rx=0 tx=0).stop 2023-09-30T01:33:02.408 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:02.394+0000 7f0023fff700 1 -- 172.21.15.88:0/2793502255 shutdown_connections 2023-09-30T01:33:02.408 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:02.394+0000 7f0023fff700 1 --2- 172.21.15.88:0/2793502255 >> v2:172.21.15.107:3300/0 conn(0x7f002406ee40 0x7f002409d4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:02.408 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:02.394+0000 7f0023fff700 1 --2- 172.21.15.88:0/2793502255 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f002406ea70 0x7f002406c910 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:02.408 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:02.394+0000 7f0023fff700 1 -- 172.21.15.88:0/2793502255 >> 172.21.15.88:0/2793502255 conn(0x7f00240a2c00 msgr2=0x7f00240a3000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:33:02.409 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:02.394+0000 7f0023fff700 1 -- 172.21.15.88:0/2793502255 shutdown_connections 2023-09-30T01:33:02.409 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:02.394+0000 7f0023fff700 1 -- 172.21.15.88:0/2793502255 wait complete. 2023-09-30T01:33:02.409 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:02.394+0000 7f0023fff700 10 monclient: build_initial_monmap 2023-09-30T01:33:02.409 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:02.394+0000 7f0023fff700 10 monclient: monmap: 2023-09-30T01:33:02.409 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2023-09-30T01:33:02.409 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:33:02.409 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 0.000000 2023-09-30T01:33:02.409 INFO:teuthology.orchestra.run.smithi088.stderr:created 0.000000 2023-09-30T01:33:02.409 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2023-09-30T01:33:02.409 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 1 2023-09-30T01:33:02.409 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.noname-a 2023-09-30T01:33:02.409 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.noname-f 2023-09-30T01:33:02.409 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.noname-b 2023-09-30T01:33:02.410 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.noname-g 2023-09-30T01:33:02.410 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.noname-c 2023-09-30T01:33:02.410 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.noname-h 2023-09-30T01:33:02.410 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.noname-d 2023-09-30T01:33:02.410 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.noname-i 2023-09-30T01:33:02.410 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.noname-e 2023-09-30T01:33:02.410 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:33:02.410 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:02.394+0000 7f0023fff700 1 Processor -- start 2023-09-30T01:33:02.410 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:02.394+0000 7f0023fff700 1 -- start start 2023-09-30T01:33:02.410 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:02.394+0000 7f0023fff700 10 monclient: init 2023-09-30T01:33:02.410 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:02.394+0000 7f0023fff700 10 monclient: _reopen_session rank -1 2023-09-30T01:33:02.410 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:02.394+0000 7f0023fff700 10 monclient: _add_conns ranks=[4,0,2,8,5,6,1,7,3] 2023-09-30T01:33:02.411 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:02.394+0000 7f0023fff700 1 --2- >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f002406ea70 0x7f0024142640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:33:02.411 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:02.394+0000 7f0023fff700 10 monclient(hunting): picked mon.noname-c con 0x7f002406ea70 addr [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] 2023-09-30T01:33:02.411 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:02.394+0000 7f0023fff700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f002406ee40 0x7f002412fb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:33:02.411 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:02.394+0000 7f0023fff700 10 monclient(hunting): picked mon.noname-a con 0x7f002406ee40 addr [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] 2023-09-30T01:33:02.411 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:02.394+0000 7f0023fff700 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f002409da30 0x7f0024098b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:33:02.411 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:02.394+0000 7f0023fff700 10 monclient(hunting): picked mon.noname-b con 0x7f002409da30 addr [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] 2023-09-30T01:33:02.411 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:02.394+0000 7f0023fff700 10 monclient(hunting): start opening mon connection 2023-09-30T01:33:02.411 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:02.394+0000 7f0023fff700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0024099ff0 con 0x7f002406ee40 2023-09-30T01:33:02.411 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:02.394+0000 7f0023fff700 10 monclient(hunting): start opening mon connection 2023-09-30T01:33:02.411 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:02.394+0000 7f0023fff700 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f002409c4b0 con 0x7f002409da30 2023-09-30T01:33:02.411 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:02.394+0000 7f0023fff700 10 monclient(hunting): start opening mon connection 2023-09-30T01:33:02.411 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:02.394+0000 7f0023fff700 1 -- --> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] -- mon_getmap magic: 0 v1 -- 0x7f002406b550 con 0x7f002406ea70 2023-09-30T01:33:02.412 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:02.394+0000 7f0023fff700 10 monclient(hunting): _renew_subs 2023-09-30T01:33:02.412 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:02.394+0000 7f0023fff700 10 monclient(hunting): authenticate will time out at 854.435397s 2023-09-30T01:33:02.412 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:02.394+0000 7f0022ffd700 1 --2- >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f002406ea70 0x7f0024142640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:33:02.412 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:02.394+0000 7f0022ffd700 1 --2- >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f002406ea70 0x7f0024142640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3302/0 says I am v2:172.21.15.88:50358/0 (socket says 172.21.15.88:50358) 2023-09-30T01:33:02.412 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:02.394+0000 7f0022ffd700 1 -- 172.21.15.88:0/2475167360 learned_addr learned my addr 172.21.15.88:0/2475167360 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-09-30T01:33:02.412 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:02.394+0000 7f0022ffd700 10 monclient(hunting): get_auth_request con 0x7f002406ea70 auth_method 0 2023-09-30T01:33:02.412 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:02.394+0000 7f0022ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:33:02.412 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:02.394+0000 7f0022ffd700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:33:02.412 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:02.394+0000 7f0022ffd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:33:02.412 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:02.394+0000 7f001a7fc700 1 --2- 172.21.15.88:0/2475167360 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f002406ee40 0x7f002412fb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:33:02.412 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:02.394+0000 7f001a7fc700 10 monclient(hunting): get_auth_request con 0x7f002406ee40 auth_method 0 2023-09-30T01:33:02.412 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:02.394+0000 7f001a7fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:33:02.413 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:02.394+0000 7f001a7fc700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:33:02.413 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:02.394+0000 7f001a7fc700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:33:02.413 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:02.394+0000 7f0022ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:33:02.413 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:02.394+0000 7f0022ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:33:02.413 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:02.394+0000 7f00237fe700 1 --2- 172.21.15.88:0/2475167360 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f002409da30 0x7f0024098b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:33:02.413 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:02.394+0000 7f0022ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:33:02.413 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:02.394+0000 7f00237fe700 10 monclient(hunting): get_auth_request con 0x7f002409da30 auth_method 0 2023-09-30T01:33:02.413 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:02.394+0000 7f00237fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:33:02.413 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:02.394+0000 7f00237fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:33:02.413 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:02.394+0000 7f00237fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:33:02.413 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:02.394+0000 7f001a7fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:33:02.413 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:02.394+0000 7f001a7fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:33:02.414 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:02.394+0000 7f001a7fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:33:02.414 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:02.398+0000 7f00237fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:33:02.414 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:02.398+0000 7f00237fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:33:02.414 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:02.398+0000 7f00237fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:33:02.414 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:02.398+0000 7f0022ffd700 10 monclient(hunting): handle_auth_done global_id 4234 payload 824 2023-09-30T01:33:02.414 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:02.398+0000 7f0022ffd700 1 -- 172.21.15.88:0/2475167360 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f002409da30 msgr2=0x7f0024098b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:33:02.414 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:02.398+0000 7f0022ffd700 1 --2- 172.21.15.88:0/2475167360 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f002409da30 0x7f0024098b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:02.414 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:02.398+0000 7f0022ffd700 1 -- 172.21.15.88:0/2475167360 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f002406ee40 msgr2=0x7f002412fb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:33:02.414 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:02.398+0000 7f0022ffd700 1 --2- 172.21.15.88:0/2475167360 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f002406ee40 0x7f002412fb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:02.414 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:02.398+0000 7f0022ffd700 10 monclient: _finish_hunting 0 2023-09-30T01:33:02.414 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:02.398+0000 7f0022ffd700 1 monclient: found mon.noname-c 2023-09-30T01:33:02.414 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:02.398+0000 7f0022ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:33:02.415 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:02.398+0000 7f0022ffd700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.88:3302/0 2023-09-30T01:33:02.415 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:02.398+0000 7f0022ffd700 1 -- 172.21.15.88:0/2475167360 --> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f001400a620 con 0x7f002406ea70 2023-09-30T01:33:02.415 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:02.398+0000 7f0022ffd700 1 --2- 172.21.15.88:0/2475167360 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f002406ea70 0x7f0024142640 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f001400c4d0 tx=0x7f0014019000 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:33:02.415 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:02.398+0000 7f001b7fe700 1 -- 172.21.15.88:0/2475167360 <== mon.4 v2:172.21.15.88:3302/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f001401f070 con 0x7f002406ea70 2023-09-30T01:33:02.415 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:02.398+0000 7f001b7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:33:02.415 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:02.398+0000 7f001b7fe700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T01:33:02.415 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:02.398+0000 7f001b7fe700 10 monclient: dump: 2023-09-30T01:33:02.415 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:33:02.415 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:33:02.415 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:02.415 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:02.415 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:33:02.415 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:33:02.416 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:33:02.416 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:33:02.416 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:33:02.416 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:33:02.416 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:33:02.416 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:33:02.416 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:33:02.416 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:33:02.416 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:33:02.416 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:33:02.416 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:02.398+0000 7f001b7fe700 10 monclient: _finish_auth 0 2023-09-30T01:33:02.416 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:02.398+0000 7f001b7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:33:02.416 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:02.398+0000 7f001b7fe700 1 -- 172.21.15.88:0/2475167360 <== mon.4 v2:172.21.15.88:3302/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f001401ae20 con 0x7f002406ea70 2023-09-30T01:33:02.417 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:02.398+0000 7f0023fff700 5 monclient: authenticate success, global_id 4234 2023-09-30T01:33:02.417 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:02.398+0000 7f0023fff700 10 monclient: _renew_subs 2023-09-30T01:33:02.417 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:02.398+0000 7f0023fff700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.88:3302/0 2023-09-30T01:33:02.417 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:02.398+0000 7f0023fff700 1 -- 172.21.15.88:0/2475167360 --> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f002406b010 con 0x7f002406ea70 2023-09-30T01:33:02.417 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:02.398+0000 7f001b7fe700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:33:02.417 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:02.398+0000 7f001b7fe700 1 -- 172.21.15.88:0/2475167360 <== mon.4 v2:172.21.15.88:3302/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f0014024d60 con 0x7f002406ea70 2023-09-30T01:33:02.417 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:02.398+0000 7f001b7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:33:02.417 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:02.398+0000 7f001b7fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T01:33:02.417 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:02.398+0000 7f001b7fe700 10 monclient: dump: 2023-09-30T01:33:02.417 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:33:02.417 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:33:02.417 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:02.417 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:02.418 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:33:02.418 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:33:02.418 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:33:02.418 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:33:02.418 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:33:02.418 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:33:02.418 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:33:02.418 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:33:02.418 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:33:02.418 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:33:02.418 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:33:02.419 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:33:02.419 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:02.398+0000 7f00217fa700 4 set_mon_vals no callback set 2023-09-30T01:33:02.419 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:02.398+0000 7f0023fff700 10 monclient: _renew_subs 2023-09-30T01:33:02.419 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:02.398+0000 7f0023fff700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.88:3302/0 2023-09-30T01:33:02.419 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:02.398+0000 7f0023fff700 1 -- 172.21.15.88:0/2475167360 --> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f002409a6a0 con 0x7f002406ea70 2023-09-30T01:33:02.419 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:02.398+0000 7f001b7fe700 1 -- 172.21.15.88:0/2475167360 <== mon.4 v2:172.21.15.88:3302/0 4 ==== mgrmap(e 4) v1 ==== 40088+0+0 (secure 0 0 0) 0x7f0014024460 con 0x7f002406ea70 2023-09-30T01:33:02.419 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:02.398+0000 7f001b7fe700 1 --2- 172.21.15.88:0/2475167360 >> v2:172.21.15.107:6812/29926 conn(0x7f00040332d0 0x7f0004035780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:33:02.419 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:02.398+0000 7f0023fff700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:33:02.419 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:02.398+0000 7f0023fff700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:33:02.419 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:02.398+0000 7f0023fff700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.88:3302/0 2023-09-30T01:33:02.419 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:02.398+0000 7f0023fff700 1 -- 172.21.15.88:0/2475167360 --> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0024071960 con 0x7f002406ea70 2023-09-30T01:33:02.419 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:02.398+0000 7f001b7fe700 1 -- 172.21.15.88:0/2475167360 <== mon.4 v2:172.21.15.88:3302/0 5 ==== osd_map(13..13 src has 1..13) v4 ==== 3480+0+0 (secure 0 0 0) 0x7f0014044700 con 0x7f002406ea70 2023-09-30T01:33:02.419 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:02.398+0000 7f001a7fc700 1 --2- 172.21.15.88:0/2475167360 >> v2:172.21.15.107:6812/29926 conn(0x7f00040332d0 0x7f0004035780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:33:02.420 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:02.398+0000 7f001a7fc700 10 monclient: get_auth_request con 0x7f00040332d0 auth_method 0 2023-09-30T01:33:02.420 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:02.398+0000 7f001a7fc700 1 --2- 172.21.15.88:0/2475167360 >> v2:172.21.15.107:6812/29926 conn(0x7f00040332d0 0x7f0004035780 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f000c009db0 tx=0x7f000c0091a0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:33:02.420 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:02.410+0000 7f001b7fe700 1 -- 172.21.15.88:0/2475167360 <== mon.4 v2:172.21.15.88:3302/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f001402d3f0 con 0x7f002406ea70 2023-09-30T01:33:02.420 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:02.410+0000 7f001b7fe700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:33:02.420 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:02.410+0000 7f001b7fe700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:33:02.575 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:02.574+0000 7f0023fff700 10 start_mon_command cmd=[{"prefix": "osd pool application enable", "pool": "rbd", "app": "rbd", "yes_i_really_mean_it": true}] 2023-09-30T01:33:02.575 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:02.574+0000 7f0023fff700 10 monclient: _send_command 2 [{"prefix": "osd pool application enable", "pool": "rbd", "app": "rbd", "yes_i_really_mean_it": true}] 2023-09-30T01:33:02.575 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:02.574+0000 7f0023fff700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.88:3302/0 2023-09-30T01:33:02.575 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:02.574+0000 7f0023fff700 1 -- 172.21.15.88:0/2475167360 --> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] -- mon_command({"prefix": "osd pool application enable", "pool": "rbd", "app": "rbd", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f002409be40 con 0x7f002406ea70 2023-09-30T01:33:03.284 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.282+0000 7f001b7fe700 1 -- 172.21.15.88:0/2475167360 <== mon.4 v2:172.21.15.88:3302/0 7 ==== mon_command_ack([{"prefix": "osd pool application enable", "pool": "rbd", "app": "rbd", "yes_i_really_mean_it": true}]=0 enabled application 'rbd' on pool 'rbd' v14) v1 ==== 173+0+0 (secure 0 0 0) 0x7f00140183b0 con 0x7f002406ea70 2023-09-30T01:33:03.284 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.282+0000 7f001b7fe700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool application enable", "pool": "rbd", "app": "rbd", "yes_i_really_mean_it": true}] 2023-09-30T01:33:03.284 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.282+0000 7f001b7fe700 10 monclient: _finish_command 2 = system:0 enabled application 'rbd' on pool 'rbd' 2023-09-30T01:33:03.286 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.282+0000 7f0023fff700 1 -- 172.21.15.88:0/2475167360 >> v2:172.21.15.107:6812/29926 conn(0x7f00040332d0 msgr2=0x7f0004035780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:33:03.287 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.282+0000 7f0023fff700 1 --2- 172.21.15.88:0/2475167360 >> v2:172.21.15.107:6812/29926 conn(0x7f00040332d0 0x7f0004035780 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f000c009db0 tx=0x7f000c0091a0 comp rx=0 tx=0).stop 2023-09-30T01:33:03.287 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.282+0000 7f0023fff700 10 monclient: shutdown 2023-09-30T01:33:03.287 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.282+0000 7f0023fff700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:33:03.287 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.282+0000 7f0023fff700 1 -- 172.21.15.88:0/2475167360 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f002406ea70 msgr2=0x7f0024142640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:33:03.287 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.282+0000 7f0023fff700 1 --2- 172.21.15.88:0/2475167360 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f002406ea70 0x7f0024142640 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f001400c4d0 tx=0x7f0014019000 comp rx=0 tx=0).stop 2023-09-30T01:33:03.287 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.282+0000 7f0023fff700 1 -- 172.21.15.88:0/2475167360 shutdown_connections 2023-09-30T01:33:03.287 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.282+0000 7f0023fff700 1 --2- 172.21.15.88:0/2475167360 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f002409da30 0x7f0024098b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:03.287 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.282+0000 7f0023fff700 1 --2- 172.21.15.88:0/2475167360 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f002406ee40 0x7f002412fb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:03.287 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.282+0000 7f0023fff700 1 --2- 172.21.15.88:0/2475167360 >> v2:172.21.15.107:6812/29926 conn(0x7f00040332d0 0x7f0004035780 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:03.288 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.282+0000 7f0023fff700 1 --2- 172.21.15.88:0/2475167360 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f002406ea70 0x7f0024142640 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:03.288 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.282+0000 7f0023fff700 1 -- 172.21.15.88:0/2475167360 >> 172.21.15.88:0/2475167360 conn(0x7f00240a2c00 msgr2=0x7f00240a1e20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:33:03.288 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.282+0000 7f0023fff700 1 -- 172.21.15.88:0/2475167360 shutdown_connections 2023-09-30T01:33:03.288 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.282+0000 7f0023fff700 1 -- 172.21.15.88:0/2475167360 wait complete. 2023-09-30T01:33:03.288 INFO:teuthology.orchestra.run.smithi088.stderr:enabled application 'rbd' on pool 'rbd' 2023-09-30T01:33:03.302 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2023-09-30T01:33:03.302 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2023-09-30T01:33:03.302 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2023-09-30T01:33:03.302 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2023-09-30T01:33:03.303 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2023-09-30T01:33:03.395 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.390+0000 7fe637009700 10 monclient: get_monmap_and_config 2023-09-30T01:33:03.395 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.390+0000 7fe637009700 10 monclient: build_initial_monmap 2023-09-30T01:33:03.395 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.390+0000 7fe637009700 10 monclient: monmap: 2023-09-30T01:33:03.395 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2023-09-30T01:33:03.395 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:33:03.395 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:33:03.395440+0000 2023-09-30T01:33:03.395 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:33:03.395440+0000 2023-09-30T01:33:03.395 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2023-09-30T01:33:03.395 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:33:03.396 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a-legacy 2023-09-30T01:33:03.396 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.107:6789/0 mon.noname-f-legacy 2023-09-30T01:33:03.396 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.noname-b 2023-09-30T01:33:03.396 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.noname-g 2023-09-30T01:33:03.396 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.noname-c 2023-09-30T01:33:03.396 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.noname-h 2023-09-30T01:33:03.396 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.noname-d 2023-09-30T01:33:03.396 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.noname-i 2023-09-30T01:33:03.396 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.noname-e 2023-09-30T01:33:03.396 INFO:teuthology.orchestra.run.smithi088.stderr:9: v2:172.21.15.88:3300/0 mon.noname-a 2023-09-30T01:33:03.396 INFO:teuthology.orchestra.run.smithi088.stderr:10: v2:172.21.15.107:3300/0 mon.noname-f 2023-09-30T01:33:03.396 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:33:03.399 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.394+0000 7fe637009700 1 Processor -- start 2023-09-30T01:33:03.400 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.394+0000 7fe637009700 1 -- start start 2023-09-30T01:33:03.400 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.394+0000 7fe637009700 10 monclient: init 2023-09-30T01:33:03.400 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.394+0000 7fe637009700 10 monclient: _reopen_session rank -1 2023-09-30T01:33:03.400 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.394+0000 7fe637009700 10 monclient: _add_conns ranks=[3,10,2,0,6,7,5,8,9,4,1] 2023-09-30T01:33:03.400 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.394+0000 7fe637009700 1 --2- >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7fe6300a3270 0x7fe6300a0160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:33:03.400 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.394+0000 7fe637009700 10 monclient(hunting): picked mon.noname-g con 0x7fe6300a3270 addr [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] 2023-09-30T01:33:03.400 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.394+0000 7fe637009700 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fe6300a4b00 0x7fe6300a06a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:33:03.400 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.394+0000 7fe637009700 10 monclient(hunting): picked mon.noname-f con 0x7fe6300a4b00 addr v2:172.21.15.107:3300/0 2023-09-30T01:33:03.400 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.394+0000 7fe637009700 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe6301333c0 0x7fe6301337d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:33:03.400 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.394+0000 7fe637009700 10 monclient(hunting): picked mon.noname-b con 0x7fe6301333c0 addr [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] 2023-09-30T01:33:03.400 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.394+0000 7fe637009700 10 monclient(hunting): start opening mon connection 2023-09-30T01:33:03.400 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.394+0000 7fe637009700 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe6300a4ed0 con 0x7fe6300a4b00 2023-09-30T01:33:03.401 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.394+0000 7fe637009700 10 monclient(hunting): start opening mon connection 2023-09-30T01:33:03.401 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.394+0000 7fe637009700 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6300a6330 con 0x7fe6301333c0 2023-09-30T01:33:03.401 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.394+0000 7fe637009700 10 monclient(hunting): start opening mon connection 2023-09-30T01:33:03.401 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.394+0000 7fe637009700 1 -- --> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe630098610 con 0x7fe6300a3270 2023-09-30T01:33:03.401 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.394+0000 7fe637009700 10 monclient(hunting): _renew_subs 2023-09-30T01:33:03.401 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.394+0000 7fe637009700 10 monclient(hunting): authenticate will time out at 855.433076s 2023-09-30T01:33:03.401 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.394+0000 7fe6355a6700 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe6301333c0 0x7fe6301337d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:33:03.401 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.394+0000 7fe6355a6700 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe6301333c0 0x7fe6301337d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3301/0 says I am v2:172.21.15.88:52410/0 (socket says 172.21.15.88:52410) 2023-09-30T01:33:03.401 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.394+0000 7fe6355a6700 1 -- 172.21.15.88:0/4276323252 learned_addr learned my addr 172.21.15.88:0/4276323252 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-09-30T01:33:03.401 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.394+0000 7fe6355a6700 10 monclient(hunting): get_auth_request con 0x7fe6301333c0 auth_method 0 2023-09-30T01:33:03.401 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.394+0000 7fe6355a6700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:33:03.402 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.394+0000 7fe6355a6700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:33:03.402 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.394+0000 7fe6355a6700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:33:03.402 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.394+0000 7fe634da5700 1 --2- 172.21.15.88:0/4276323252 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7fe6300a3270 0x7fe6300a0160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:33:03.402 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.394+0000 7fe62ffff700 1 --2- 172.21.15.88:0/4276323252 >> v2:172.21.15.107:3300/0 conn(0x7fe6300a4b00 0x7fe6300a06a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:33:03.402 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.394+0000 7fe62ffff700 10 monclient(hunting): get_auth_request con 0x7fe6300a4b00 auth_method 0 2023-09-30T01:33:03.402 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.394+0000 7fe62ffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:33:03.402 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.394+0000 7fe62ffff700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:33:03.402 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.394+0000 7fe62ffff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:33:03.402 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.394+0000 7fe6355a6700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:33:03.402 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.394+0000 7fe6355a6700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:33:03.402 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.394+0000 7fe6355a6700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:33:03.402 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.394+0000 7fe634da5700 10 monclient(hunting): get_auth_request con 0x7fe6300a3270 auth_method 0 2023-09-30T01:33:03.403 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.394+0000 7fe634da5700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:33:03.403 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.394+0000 7fe634da5700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:33:03.403 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.394+0000 7fe634da5700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:33:03.403 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.394+0000 7fe62ffff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:33:03.403 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.394+0000 7fe62ffff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:33:03.403 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.394+0000 7fe62ffff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:33:03.403 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.394+0000 7fe6355a6700 10 monclient(hunting): handle_auth_done global_id 4223 payload 274 2023-09-30T01:33:03.403 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.394+0000 7fe6355a6700 1 -- 172.21.15.88:0/4276323252 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7fe6300a3270 msgr2=0x7fe6300a0160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:33:03.403 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.394+0000 7fe6355a6700 1 --2- 172.21.15.88:0/4276323252 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7fe6300a3270 0x7fe6300a0160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:03.403 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.394+0000 7fe6355a6700 1 -- 172.21.15.88:0/4276323252 >> v2:172.21.15.107:3300/0 conn(0x7fe6300a4b00 msgr2=0x7fe6300a06a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:33:03.403 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.394+0000 7fe6355a6700 1 --2- 172.21.15.88:0/4276323252 >> v2:172.21.15.107:3300/0 conn(0x7fe6300a4b00 0x7fe6300a06a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:03.403 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.394+0000 7fe6355a6700 10 monclient: _finish_hunting 0 2023-09-30T01:33:03.403 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.394+0000 7fe6355a6700 1 monclient: found mon.noname-b 2023-09-30T01:33:03.404 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.394+0000 7fe6355a6700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:33:03.404 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.394+0000 7fe6355a6700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.88:3301/0 2023-09-30T01:33:03.404 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.394+0000 7fe6355a6700 1 -- 172.21.15.88:0/4276323252 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe63005ecb0 con 0x7fe6301333c0 2023-09-30T01:33:03.404 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.394+0000 7fe634da5700 1 --2- 172.21.15.88:0/4276323252 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7fe6300a3270 0x7fe6300a0160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-30T01:33:03.404 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.394+0000 7fe6355a6700 1 --2- 172.21.15.88:0/4276323252 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe6301333c0 0x7fe6301337d0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fe628009f20 tx=0x7fe628009840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=701c686ff1a93b9e server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:33:03.404 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.394+0000 7fe62f7fe700 1 -- 172.21.15.88:0/4276323252 <== mon.2 v2:172.21.15.88:3301/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7fe62800e070 con 0x7fe6301333c0 2023-09-30T01:33:03.404 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.394+0000 7fe62f7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:33:03.404 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.394+0000 7fe62f7fe700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T01:33:03.404 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.394+0000 7fe62f7fe700 10 monclient: dump: 2023-09-30T01:33:03.404 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:33:03.404 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:33:03.404 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:03.404 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:03.405 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:33:03.405 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:33:03.405 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:33:03.405 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:33:03.405 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:33:03.405 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:33:03.405 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:33:03.405 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:33:03.405 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:33:03.405 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:33:03.405 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:33:03.405 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:33:03.406 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.394+0000 7fe62f7fe700 10 monclient: _finish_auth 0 2023-09-30T01:33:03.406 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.394+0000 7fe62f7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:33:03.406 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.394+0000 7fe637009700 5 monclient: authenticate success, global_id 4223 2023-09-30T01:33:03.406 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.394+0000 7fe637009700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:33:03.406 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.394+0000 7fe62f7fe700 1 -- 172.21.15.88:0/4276323252 <== mon.2 v2:172.21.15.88:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe62800be50 con 0x7fe6301333c0 2023-09-30T01:33:03.406 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.394+0000 7fe62f7fe700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:33:03.406 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.394+0000 7fe637009700 10 monclient: get_monmap_and_config success 2023-09-30T01:33:03.406 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.394+0000 7fe637009700 4 set_mon_vals no callback set 2023-09-30T01:33:03.406 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.394+0000 7fe62f7fe700 1 -- 172.21.15.88:0/4276323252 <== mon.2 v2:172.21.15.88:3301/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7fe6280092e0 con 0x7fe6301333c0 2023-09-30T01:33:03.406 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.394+0000 7fe62f7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:33:03.406 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.394+0000 7fe62f7fe700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T01:33:03.406 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.394+0000 7fe62f7fe700 10 monclient: dump: 2023-09-30T01:33:03.406 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:33:03.407 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:33:03.407 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:03.407 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:03.407 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:33:03.407 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:33:03.407 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:33:03.407 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:33:03.407 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:33:03.407 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:33:03.407 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:33:03.407 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:33:03.407 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:33:03.408 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:33:03.408 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:33:03.408 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:33:03.408 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.394+0000 7fe637009700 10 monclient: shutdown 2023-09-30T01:33:03.408 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.394+0000 7fe637009700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:33:03.408 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.394+0000 7fe637009700 1 -- 172.21.15.88:0/4276323252 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe6301333c0 msgr2=0x7fe6301337d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:33:03.408 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.394+0000 7fe637009700 1 --2- 172.21.15.88:0/4276323252 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe6301333c0 0x7fe6301337d0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fe628009f20 tx=0x7fe628009840 comp rx=0 tx=0).stop 2023-09-30T01:33:03.408 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.394+0000 7fe637009700 1 -- 172.21.15.88:0/4276323252 shutdown_connections 2023-09-30T01:33:03.408 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.394+0000 7fe637009700 1 --2- 172.21.15.88:0/4276323252 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe6301333c0 0x7fe6301337d0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:03.408 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.394+0000 7fe637009700 1 --2- 172.21.15.88:0/4276323252 >> v2:172.21.15.107:3300/0 conn(0x7fe6300a4b00 0x7fe6300a06a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:03.408 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.394+0000 7fe637009700 1 --2- 172.21.15.88:0/4276323252 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7fe6300a3270 0x7fe6300a0160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:03.408 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.394+0000 7fe637009700 1 -- 172.21.15.88:0/4276323252 >> 172.21.15.88:0/4276323252 conn(0x7fe6300a67f0 msgr2=0x7fe6300a6bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:33:03.408 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.394+0000 7fe637009700 1 -- 172.21.15.88:0/4276323252 shutdown_connections 2023-09-30T01:33:03.409 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.394+0000 7fe637009700 1 -- 172.21.15.88:0/4276323252 wait complete. 2023-09-30T01:33:03.409 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.394+0000 7fe637009700 10 monclient: build_initial_monmap 2023-09-30T01:33:03.409 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.398+0000 7fe637009700 10 monclient: monmap: 2023-09-30T01:33:03.409 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2023-09-30T01:33:03.409 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:33:03.409 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 0.000000 2023-09-30T01:33:03.409 INFO:teuthology.orchestra.run.smithi088.stderr:created 0.000000 2023-09-30T01:33:03.409 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2023-09-30T01:33:03.409 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 1 2023-09-30T01:33:03.409 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.noname-a 2023-09-30T01:33:03.409 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.noname-f 2023-09-30T01:33:03.409 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.noname-b 2023-09-30T01:33:03.409 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.noname-g 2023-09-30T01:33:03.410 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.noname-c 2023-09-30T01:33:03.410 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.noname-h 2023-09-30T01:33:03.410 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.noname-d 2023-09-30T01:33:03.410 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.noname-i 2023-09-30T01:33:03.410 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.noname-e 2023-09-30T01:33:03.410 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:33:03.410 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.398+0000 7fe637009700 1 Processor -- start 2023-09-30T01:33:03.410 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.398+0000 7fe637009700 1 -- start start 2023-09-30T01:33:03.410 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.398+0000 7fe637009700 10 monclient: init 2023-09-30T01:33:03.414 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.398+0000 7fe637009700 10 monclient: _reopen_session rank -1 2023-09-30T01:33:03.414 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.398+0000 7fe637009700 10 monclient: _add_conns ranks=[1,2,3,8,0,5,7,6,4] 2023-09-30T01:33:03.414 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.398+0000 7fe637009700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe6300a3270 0x7fe6301420c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:33:03.414 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.398+0000 7fe637009700 10 monclient(hunting): picked mon.noname-f con 0x7fe6300a3270 addr [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] 2023-09-30T01:33:03.414 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.398+0000 7fe637009700 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe6300a4b00 0x7fe630130330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:33:03.414 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.398+0000 7fe637009700 10 monclient(hunting): picked mon.noname-b con 0x7fe6300a4b00 addr [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] 2023-09-30T01:33:03.415 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.398+0000 7fe637009700 1 --2- >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7fe6301333c0 0x7fe63009dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:33:03.415 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.398+0000 7fe637009700 10 monclient(hunting): picked mon.noname-g con 0x7fe6301333c0 addr [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] 2023-09-30T01:33:03.415 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.398+0000 7fe637009700 10 monclient(hunting): start opening mon connection 2023-09-30T01:33:03.415 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.398+0000 7fe637009700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe630097c70 con 0x7fe6300a3270 2023-09-30T01:33:03.415 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.398+0000 7fe637009700 10 monclient(hunting): start opening mon connection 2023-09-30T01:33:03.415 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.398+0000 7fe637009700 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe63009fed0 con 0x7fe6300a4b00 2023-09-30T01:33:03.415 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.398+0000 7fe637009700 10 monclient(hunting): start opening mon connection 2023-09-30T01:33:03.415 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.398+0000 7fe637009700 1 -- --> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe630142f30 con 0x7fe6301333c0 2023-09-30T01:33:03.415 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.398+0000 7fe637009700 10 monclient(hunting): _renew_subs 2023-09-30T01:33:03.415 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.398+0000 7fe637009700 10 monclient(hunting): authenticate will time out at 855.438056s 2023-09-30T01:33:03.415 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.398+0000 7fe62ffff700 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe6300a4b00 0x7fe630130330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:33:03.415 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.398+0000 7fe634da5700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe6300a3270 0x7fe6301420c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:33:03.416 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.398+0000 7fe634da5700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe6300a3270 0x7fe6301420c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.88:57548/0 (socket says 172.21.15.88:57548) 2023-09-30T01:33:03.416 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.398+0000 7fe634da5700 1 -- 172.21.15.88:0/553954116 learned_addr learned my addr 172.21.15.88:0/553954116 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-09-30T01:33:03.416 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.398+0000 7fe634da5700 10 monclient(hunting): get_auth_request con 0x7fe6300a3270 auth_method 0 2023-09-30T01:33:03.416 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.398+0000 7fe634da5700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:33:03.416 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.398+0000 7fe634da5700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:33:03.416 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.398+0000 7fe634da5700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:33:03.416 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.398+0000 7fe62ffff700 10 monclient(hunting): get_auth_request con 0x7fe6300a4b00 auth_method 0 2023-09-30T01:33:03.416 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.398+0000 7fe62ffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:33:03.416 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.398+0000 7fe62ffff700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:33:03.416 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.398+0000 7fe62ffff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:33:03.416 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.398+0000 7fe62ffff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:33:03.416 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.398+0000 7fe62ffff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:33:03.416 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.398+0000 7fe6355a6700 1 --2- 172.21.15.88:0/553954116 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7fe6301333c0 0x7fe63009dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:33:03.416 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.398+0000 7fe62ffff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:33:03.417 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.398+0000 7fe6355a6700 10 monclient(hunting): get_auth_request con 0x7fe6301333c0 auth_method 0 2023-09-30T01:33:03.417 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.398+0000 7fe6355a6700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:33:03.417 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.398+0000 7fe6355a6700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:33:03.417 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.398+0000 7fe6355a6700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:33:03.417 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.398+0000 7fe634da5700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:33:03.417 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.398+0000 7fe634da5700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:33:03.417 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.398+0000 7fe634da5700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:33:03.417 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.398+0000 7fe6355a6700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:33:03.417 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.398+0000 7fe6355a6700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:33:03.417 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.398+0000 7fe6355a6700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:33:03.417 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.398+0000 7fe62ffff700 10 monclient(hunting): handle_auth_done global_id 4232 payload 824 2023-09-30T01:33:03.417 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.398+0000 7fe62ffff700 1 -- 172.21.15.88:0/553954116 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7fe6301333c0 msgr2=0x7fe63009dcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:33:03.417 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.398+0000 7fe62ffff700 1 --2- 172.21.15.88:0/553954116 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7fe6301333c0 0x7fe63009dcd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:03.418 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.398+0000 7fe62ffff700 1 -- 172.21.15.88:0/553954116 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe6300a3270 msgr2=0x7fe6301420c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:33:03.418 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.398+0000 7fe62ffff700 1 --2- 172.21.15.88:0/553954116 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe6300a3270 0x7fe6301420c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:03.418 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.398+0000 7fe62ffff700 10 monclient: _finish_hunting 0 2023-09-30T01:33:03.418 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.398+0000 7fe62ffff700 1 monclient: found mon.noname-b 2023-09-30T01:33:03.418 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.398+0000 7fe62ffff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:33:03.418 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.398+0000 7fe62ffff700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.88:3301/0 2023-09-30T01:33:03.418 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.398+0000 7fe62ffff700 1 -- 172.21.15.88:0/553954116 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe628009ae0 con 0x7fe6300a4b00 2023-09-30T01:33:03.418 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.398+0000 7fe62ffff700 1 --2- 172.21.15.88:0/553954116 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe6300a4b00 0x7fe630130330 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fe61c00ea80 tx=0x7fe61c0091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:33:03.418 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.398+0000 7fe62cff9700 1 -- 172.21.15.88:0/553954116 <== mon.2 v2:172.21.15.88:3301/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7fe61c0095d0 con 0x7fe6300a4b00 2023-09-30T01:33:03.418 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.398+0000 7fe62cff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:33:03.418 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.398+0000 7fe62cff9700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T01:33:03.418 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.398+0000 7fe62cff9700 10 monclient: dump: 2023-09-30T01:33:03.419 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:33:03.419 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:33:03.419 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:03.419 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:03.419 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:33:03.419 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:33:03.419 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:33:03.419 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:33:03.419 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:33:03.419 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:33:03.419 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:33:03.419 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:33:03.419 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:33:03.419 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:33:03.420 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:33:03.420 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:33:03.420 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.398+0000 7fe62cff9700 10 monclient: _finish_auth 0 2023-09-30T01:33:03.420 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.398+0000 7fe62cff9700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:33:03.420 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.398+0000 7fe637009700 5 monclient: authenticate success, global_id 4232 2023-09-30T01:33:03.420 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.398+0000 7fe62cff9700 1 -- 172.21.15.88:0/553954116 <== mon.2 v2:172.21.15.88:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe61c00cc70 con 0x7fe6300a4b00 2023-09-30T01:33:03.420 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.398+0000 7fe62cff9700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:33:03.420 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.398+0000 7fe62cff9700 1 -- 172.21.15.88:0/553954116 <== mon.2 v2:172.21.15.88:3301/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7fe61c00cde0 con 0x7fe6300a4b00 2023-09-30T01:33:03.420 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.398+0000 7fe62cff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:33:03.420 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.398+0000 7fe62cff9700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T01:33:03.420 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.398+0000 7fe62cff9700 10 monclient: dump: 2023-09-30T01:33:03.420 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:33:03.421 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:33:03.421 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:03.421 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:03.421 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:33:03.421 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:33:03.421 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:33:03.421 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:33:03.421 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:33:03.421 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:33:03.421 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:33:03.421 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:33:03.421 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:33:03.421 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:33:03.422 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:33:03.422 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:33:03.422 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.398+0000 7fe62effd700 4 set_mon_vals no callback set 2023-09-30T01:33:03.422 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.398+0000 7fe637009700 10 monclient: _renew_subs 2023-09-30T01:33:03.422 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.398+0000 7fe637009700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.88:3301/0 2023-09-30T01:33:03.422 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.398+0000 7fe637009700 1 -- 172.21.15.88:0/553954116 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe63011d730 con 0x7fe6300a4b00 2023-09-30T01:33:03.422 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.398+0000 7fe637009700 10 monclient: _renew_subs 2023-09-30T01:33:03.422 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.398+0000 7fe637009700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.88:3301/0 2023-09-30T01:33:03.422 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.398+0000 7fe637009700 1 -- 172.21.15.88:0/553954116 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe63009f950 con 0x7fe6300a4b00 2023-09-30T01:33:03.422 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.402+0000 7fe637009700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:33:03.422 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.402+0000 7fe637009700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:33:03.423 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.402+0000 7fe637009700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.88:3301/0 2023-09-30T01:33:03.423 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.402+0000 7fe637009700 1 -- 172.21.15.88:0/553954116 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe630071960 con 0x7fe6300a4b00 2023-09-30T01:33:03.423 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.402+0000 7fe62cff9700 1 -- 172.21.15.88:0/553954116 <== mon.2 v2:172.21.15.88:3301/0 4 ==== mgrmap(e 4) v1 ==== 40088+0+0 (secure 0 0 0) 0x7fe61c00c7f0 con 0x7fe6300a4b00 2023-09-30T01:33:03.423 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.402+0000 7fe62cff9700 1 --2- 172.21.15.88:0/553954116 >> v2:172.21.15.107:6812/29926 conn(0x7fe614033300 0x7fe6140357b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:33:03.423 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.402+0000 7fe62cff9700 1 -- 172.21.15.88:0/553954116 <== mon.2 v2:172.21.15.88:3301/0 5 ==== osd_map(14..14 src has 1..14) v4 ==== 3502+0+0 (secure 0 0 0) 0x7fe61c0434f0 con 0x7fe6300a4b00 2023-09-30T01:33:03.423 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.402+0000 7fe634da5700 1 --2- 172.21.15.88:0/553954116 >> v2:172.21.15.107:6812/29926 conn(0x7fe614033300 0x7fe6140357b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:33:03.423 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.402+0000 7fe634da5700 10 monclient: get_auth_request con 0x7fe614033300 auth_method 0 2023-09-30T01:33:03.423 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.402+0000 7fe634da5700 1 --2- 172.21.15.88:0/553954116 >> v2:172.21.15.107:6812/29926 conn(0x7fe614033300 0x7fe6140357b0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fe624006dc0 tx=0x7fe624008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:33:03.423 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.410+0000 7fe62cff9700 1 -- 172.21.15.88:0/553954116 <== mon.2 v2:172.21.15.88:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fe61c0192c0 con 0x7fe6300a4b00 2023-09-30T01:33:03.423 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.410+0000 7fe62cff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:33:03.423 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.410+0000 7fe62cff9700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:33:03.618 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.614+0000 7fe637009700 10 start_mon_command cmd=[{"prefix": "mgr dump", "format": "json"}] 2023-09-30T01:33:03.618 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.614+0000 7fe637009700 10 monclient: _send_command 2 [{"prefix": "mgr dump", "format": "json"}] 2023-09-30T01:33:03.618 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.614+0000 7fe637009700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.88:3301/0 2023-09-30T01:33:03.619 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.614+0000 7fe637009700 1 -- 172.21.15.88:0/553954116 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) v1 -- 0x7fe630062c70 con 0x7fe6300a4b00 2023-09-30T01:33:03.620 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.618+0000 7fe62cff9700 1 -- 172.21.15.88:0/553954116 <== mon.2 v2:172.21.15.88:3301/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v4) v1 ==== 74+0+77509 (secure 0 0 0) 0x7fe61c01ae80 con 0x7fe6300a4b00 2023-09-30T01:33:03.620 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.618+0000 7fe62cff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "mgr dump", "format": "json"}] 2023-09-30T01:33:03.621 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.618+0000 7fe62cff9700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:33:03.622 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-09-30T01:33:03.624 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.622+0000 7fe637009700 1 -- 172.21.15.88:0/553954116 >> v2:172.21.15.107:6812/29926 conn(0x7fe614033300 msgr2=0x7fe6140357b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:33:03.624 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.622+0000 7fe637009700 1 --2- 172.21.15.88:0/553954116 >> v2:172.21.15.107:6812/29926 conn(0x7fe614033300 0x7fe6140357b0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fe624006dc0 tx=0x7fe624008040 comp rx=0 tx=0).stop 2023-09-30T01:33:03.624 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.622+0000 7fe637009700 10 monclient: shutdown 2023-09-30T01:33:03.624 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.622+0000 7fe637009700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:33:03.624 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.622+0000 7fe637009700 1 -- 172.21.15.88:0/553954116 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe6300a4b00 msgr2=0x7fe630130330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:33:03.624 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.622+0000 7fe637009700 1 --2- 172.21.15.88:0/553954116 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe6300a4b00 0x7fe630130330 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fe61c00ea80 tx=0x7fe61c0091a0 comp rx=0 tx=0).stop 2023-09-30T01:33:03.624 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.622+0000 7fe637009700 1 -- 172.21.15.88:0/553954116 shutdown_connections 2023-09-30T01:33:03.625 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.622+0000 7fe637009700 1 --2- 172.21.15.88:0/553954116 >> v2:172.21.15.107:6812/29926 conn(0x7fe614033300 0x7fe6140357b0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:03.625 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.622+0000 7fe637009700 1 --2- 172.21.15.88:0/553954116 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7fe6301333c0 0x7fe63009dcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:03.625 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.622+0000 7fe637009700 1 --2- 172.21.15.88:0/553954116 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe6300a4b00 0x7fe630130330 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:03.625 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.622+0000 7fe637009700 1 --2- 172.21.15.88:0/553954116 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe6300a3270 0x7fe6301420c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:03.625 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.622+0000 7fe637009700 1 -- 172.21.15.88:0/553954116 >> 172.21.15.88:0/553954116 conn(0x7fe6300a67f0 msgr2=0x7fe6300a20e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:33:03.625 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.622+0000 7fe637009700 1 -- 172.21.15.88:0/553954116 shutdown_connections 2023-09-30T01:33:03.625 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.622+0000 7fe637009700 1 -- 172.21.15.88:0/553954116 wait complete. 2023-09-30T01:33:03.640 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":4,"active_gid":4108,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":29926}]},"active_addr":"172.21.15.107:6812/29926","active_change":"2023-09-30T01:32:55.639354+0000","active_mgr_features":4540138320759226367,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":false,"error_string":"loading asyncssh library:No module named 'asyncssh'","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.1.5","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.107:0","nonce":586140942}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.107:0","nonce":3583015890}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.107:0","nonce":2034708167}]}]} 2023-09-30T01:33:03.642 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2023-09-30T01:33:03.642 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2023-09-30T01:33:03.643 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-09-30T01:33:03.737 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.734+0000 7f2c1f9a6700 10 monclient: get_monmap_and_config 2023-09-30T01:33:03.737 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.734+0000 7f2c1f9a6700 10 monclient: build_initial_monmap 2023-09-30T01:33:03.737 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.734+0000 7f2c1f9a6700 10 monclient: monmap: 2023-09-30T01:33:03.738 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2023-09-30T01:33:03.738 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:33:03.738 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:33:03.737849+0000 2023-09-30T01:33:03.738 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:33:03.737849+0000 2023-09-30T01:33:03.738 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2023-09-30T01:33:03.738 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:33:03.738 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a-legacy 2023-09-30T01:33:03.738 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.107:6789/0 mon.noname-f-legacy 2023-09-30T01:33:03.739 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.noname-b 2023-09-30T01:33:03.739 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.noname-g 2023-09-30T01:33:03.739 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.noname-c 2023-09-30T01:33:03.739 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.noname-h 2023-09-30T01:33:03.739 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.noname-d 2023-09-30T01:33:03.739 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.noname-i 2023-09-30T01:33:03.739 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.noname-e 2023-09-30T01:33:03.739 INFO:teuthology.orchestra.run.smithi088.stderr:9: v2:172.21.15.88:3300/0 mon.noname-a 2023-09-30T01:33:03.739 INFO:teuthology.orchestra.run.smithi088.stderr:10: v2:172.21.15.107:3300/0 mon.noname-f 2023-09-30T01:33:03.740 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:33:03.748 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.734+0000 7f2c1f9a6700 1 Processor -- start 2023-09-30T01:33:03.748 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.734+0000 7f2c1f9a6700 1 -- start start 2023-09-30T01:33:03.748 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.734+0000 7f2c1f9a6700 10 monclient: init 2023-09-30T01:33:03.749 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.734+0000 7f2c1f9a6700 10 monclient: _reopen_session rank -1 2023-09-30T01:33:03.749 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.734+0000 7f2c1f9a6700 10 monclient: _add_conns ranks=[2,4,8,10,3,7,5,0,1,9,6] 2023-09-30T01:33:03.749 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.734+0000 7f2c1f9a6700 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2c1809ddb0 0x7f2c18098ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:33:03.749 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.734+0000 7f2c1f9a6700 10 monclient(hunting): picked mon.noname-b con 0x7f2c1809ddb0 addr [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] 2023-09-30T01:33:03.749 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.734+0000 7f2c1f9a6700 1 --2- >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f2c1809e180 0x7f2c1809e570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:33:03.749 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.734+0000 7f2c1f9a6700 10 monclient(hunting): picked mon.noname-c con 0x7f2c1809e180 addr [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] 2023-09-30T01:33:03.749 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.734+0000 7f2c1f9a6700 1 --2- >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f2c181305f0 0x7f2c18130a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:33:03.749 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.734+0000 7f2c1f9a6700 10 monclient(hunting): picked mon.noname-e con 0x7f2c181305f0 addr [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] 2023-09-30T01:33:03.749 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.734+0000 7f2c1f9a6700 10 monclient(hunting): start opening mon connection 2023-09-30T01:33:03.749 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.734+0000 7f2c1f9a6700 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c18098200 con 0x7f2c1809ddb0 2023-09-30T01:33:03.750 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.734+0000 7f2c1f9a6700 10 monclient(hunting): start opening mon connection 2023-09-30T01:33:03.750 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.734+0000 7f2c1f9a6700 1 -- --> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] -- mon_getmap magic: 0 v1 -- 0x7f2c18098b10 con 0x7f2c1809e180 2023-09-30T01:33:03.750 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.734+0000 7f2c1f9a6700 10 monclient(hunting): start opening mon connection 2023-09-30T01:33:03.750 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.734+0000 7f2c1f9a6700 1 -- --> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] -- mon_getmap magic: 0 v1 -- 0x7f2c1809a2d0 con 0x7f2c181305f0 2023-09-30T01:33:03.750 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.734+0000 7f2c1f9a6700 10 monclient(hunting): _renew_subs 2023-09-30T01:33:03.750 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.734+0000 7f2c1f9a6700 10 monclient(hunting): authenticate will time out at 855.775399s 2023-09-30T01:33:03.750 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.734+0000 7f2c1cf41700 1 --2- >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f2c1809e180 0x7f2c1809e570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:33:03.750 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.734+0000 7f2c1d742700 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2c1809ddb0 0x7f2c18098ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:33:03.750 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.734+0000 7f2c1cf41700 1 --2- >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f2c1809e180 0x7f2c1809e570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3302/0 says I am v2:172.21.15.88:50370/0 (socket says 172.21.15.88:50370) 2023-09-30T01:33:03.750 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.734+0000 7f2c1cf41700 1 -- 172.21.15.88:0/3257955921 learned_addr learned my addr 172.21.15.88:0/3257955921 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-09-30T01:33:03.750 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.734+0000 7f2c1d742700 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2c1809ddb0 0x7f2c18098ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3301/0 says I am v2:172.21.15.88:52416/0 (socket says 172.21.15.88:52416) 2023-09-30T01:33:03.750 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.734+0000 7f2c1cf41700 10 monclient(hunting): get_auth_request con 0x7f2c1809e180 auth_method 0 2023-09-30T01:33:03.751 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.734+0000 7f2c1cf41700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:33:03.751 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.734+0000 7f2c1cf41700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:33:03.751 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.734+0000 7f2c1cf41700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:33:03.751 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.734+0000 7f2c1d742700 10 monclient(hunting): get_auth_request con 0x7f2c1809ddb0 auth_method 0 2023-09-30T01:33:03.751 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.734+0000 7f2c1d742700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:33:03.751 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.734+0000 7f2c1d742700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:33:03.751 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.734+0000 7f2c1d742700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:33:03.751 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.734+0000 7f2c1df43700 1 --2- 172.21.15.88:0/3257955921 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f2c181305f0 0x7f2c18130a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:33:03.751 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.734+0000 7f2c1cf41700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:33:03.751 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.734+0000 7f2c1cf41700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:33:03.751 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.734+0000 7f2c1cf41700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:33:03.751 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.734+0000 7f2c1d742700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:33:03.752 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.734+0000 7f2c1d742700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:33:03.752 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.734+0000 7f2c1d742700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:33:03.752 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.734+0000 7f2c1df43700 10 monclient(hunting): get_auth_request con 0x7f2c181305f0 auth_method 0 2023-09-30T01:33:03.752 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.734+0000 7f2c1df43700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:33:03.752 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.734+0000 7f2c1df43700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:33:03.752 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.734+0000 7f2c1df43700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:33:03.752 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.734+0000 7f2c1df43700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:33:03.752 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.738+0000 7f2c1df43700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:33:03.752 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.738+0000 7f2c1df43700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:33:03.752 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.738+0000 7f2c1cf41700 10 monclient(hunting): handle_auth_done global_id 4243 payload 274 2023-09-30T01:33:03.752 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.738+0000 7f2c1cf41700 1 -- 172.21.15.88:0/3257955921 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f2c181305f0 msgr2=0x7f2c18130a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:33:03.752 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.738+0000 7f2c1cf41700 1 --2- 172.21.15.88:0/3257955921 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f2c181305f0 0x7f2c18130a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:03.752 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.738+0000 7f2c1cf41700 1 -- 172.21.15.88:0/3257955921 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2c1809ddb0 msgr2=0x7f2c18098ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:33:03.753 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.738+0000 7f2c1cf41700 1 --2- 172.21.15.88:0/3257955921 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2c1809ddb0 0x7f2c18098ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:03.753 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.738+0000 7f2c1cf41700 10 monclient: _finish_hunting 0 2023-09-30T01:33:03.753 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.738+0000 7f2c1cf41700 1 monclient: found mon.noname-c 2023-09-30T01:33:03.753 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.738+0000 7f2c1cf41700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:33:03.753 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.738+0000 7f2c1cf41700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.88:3302/0 2023-09-30T01:33:03.753 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.738+0000 7f2c1cf41700 1 -- 172.21.15.88:0/3257955921 --> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c1805ecb0 con 0x7f2c1809e180 2023-09-30T01:33:03.753 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.738+0000 7f2c1d742700 1 --2- 172.21.15.88:0/3257955921 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2c1809ddb0 0x7f2c18098ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:33:03.753 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.738+0000 7f2c1cf41700 1 --2- 172.21.15.88:0/3257955921 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f2c1809e180 0x7f2c1809e570 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f2c0c00bc80 tx=0x7f2c0c00b4f0 comp rx=0 tx=0).ready entity=mon.4 client_cookie=feb81109d1f0e24e server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:33:03.753 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.738+0000 7f2c17fff700 1 -- 172.21.15.88:0/3257955921 <== mon.4 v2:172.21.15.88:3302/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f2c0c019480 con 0x7f2c1809e180 2023-09-30T01:33:03.753 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.738+0000 7f2c17fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:33:03.753 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.738+0000 7f2c17fff700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T01:33:03.754 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.738+0000 7f2c17fff700 10 monclient: dump: 2023-09-30T01:33:03.754 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:33:03.754 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:33:03.754 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:03.754 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:03.754 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:33:03.754 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:33:03.754 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:33:03.754 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:33:03.754 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:33:03.754 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:33:03.754 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:33:03.755 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:33:03.755 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:33:03.755 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:33:03.755 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:33:03.755 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:33:03.755 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.738+0000 7f2c17fff700 10 monclient: _finish_auth 0 2023-09-30T01:33:03.755 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.738+0000 7f2c17fff700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:33:03.755 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.738+0000 7f2c1f9a6700 5 monclient: authenticate success, global_id 4243 2023-09-30T01:33:03.755 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.738+0000 7f2c1f9a6700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:33:03.755 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.738+0000 7f2c17fff700 1 -- 172.21.15.88:0/3257955921 <== mon.4 v2:172.21.15.88:3302/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c0c019600 con 0x7f2c1809e180 2023-09-30T01:33:03.755 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.738+0000 7f2c17fff700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:33:03.755 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.738+0000 7f2c1f9a6700 10 monclient: get_monmap_and_config success 2023-09-30T01:33:03.756 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.738+0000 7f2c1f9a6700 4 set_mon_vals no callback set 2023-09-30T01:33:03.756 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.738+0000 7f2c17fff700 1 -- 172.21.15.88:0/3257955921 <== mon.4 v2:172.21.15.88:3302/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f2c0c019bf0 con 0x7f2c1809e180 2023-09-30T01:33:03.756 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.738+0000 7f2c17fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:33:03.756 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.738+0000 7f2c17fff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T01:33:03.756 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.738+0000 7f2c17fff700 10 monclient: dump: 2023-09-30T01:33:03.756 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:33:03.756 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:33:03.756 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:03.757 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:03.757 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:33:03.757 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:33:03.757 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:33:03.757 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:33:03.757 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:33:03.757 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:33:03.757 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:33:03.757 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:33:03.757 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:33:03.757 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:33:03.757 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:33:03.758 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:33:03.758 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.738+0000 7f2c1f9a6700 10 monclient: shutdown 2023-09-30T01:33:03.758 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.738+0000 7f2c1f9a6700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:33:03.758 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.738+0000 7f2c1f9a6700 1 -- 172.21.15.88:0/3257955921 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f2c1809e180 msgr2=0x7f2c1809e570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:33:03.758 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.738+0000 7f2c1f9a6700 1 --2- 172.21.15.88:0/3257955921 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f2c1809e180 0x7f2c1809e570 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f2c0c00bc80 tx=0x7f2c0c00b4f0 comp rx=0 tx=0).stop 2023-09-30T01:33:03.758 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.738+0000 7f2c1f9a6700 1 -- 172.21.15.88:0/3257955921 shutdown_connections 2023-09-30T01:33:03.758 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.738+0000 7f2c1f9a6700 1 --2- 172.21.15.88:0/3257955921 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f2c181305f0 0x7f2c18130a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:03.758 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.738+0000 7f2c1f9a6700 1 --2- 172.21.15.88:0/3257955921 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f2c1809e180 0x7f2c1809e570 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:03.758 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.738+0000 7f2c1f9a6700 1 --2- 172.21.15.88:0/3257955921 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2c1809ddb0 0x7f2c18098ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:03.758 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.738+0000 7f2c1f9a6700 1 -- 172.21.15.88:0/3257955921 >> 172.21.15.88:0/3257955921 conn(0x7f2c180a3440 msgr2=0x7f2c180a3840 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:33:03.758 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.738+0000 7f2c1f9a6700 1 -- 172.21.15.88:0/3257955921 shutdown_connections 2023-09-30T01:33:03.758 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.738+0000 7f2c1f9a6700 1 -- 172.21.15.88:0/3257955921 wait complete. 2023-09-30T01:33:03.759 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.738+0000 7f2c1f9a6700 10 monclient: build_initial_monmap 2023-09-30T01:33:03.759 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.738+0000 7f2c1f9a6700 10 monclient: monmap: 2023-09-30T01:33:03.759 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2023-09-30T01:33:03.759 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:33:03.759 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 0.000000 2023-09-30T01:33:03.759 INFO:teuthology.orchestra.run.smithi088.stderr:created 0.000000 2023-09-30T01:33:03.759 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2023-09-30T01:33:03.759 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 1 2023-09-30T01:33:03.759 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.noname-a 2023-09-30T01:33:03.759 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.noname-f 2023-09-30T01:33:03.759 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.noname-b 2023-09-30T01:33:03.759 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.noname-g 2023-09-30T01:33:03.760 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.noname-c 2023-09-30T01:33:03.760 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.noname-h 2023-09-30T01:33:03.760 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.noname-d 2023-09-30T01:33:03.760 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.noname-i 2023-09-30T01:33:03.760 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.noname-e 2023-09-30T01:33:03.760 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:33:03.760 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.738+0000 7f2c1f9a6700 1 Processor -- start 2023-09-30T01:33:03.760 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.738+0000 7f2c1f9a6700 1 -- start start 2023-09-30T01:33:03.760 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.738+0000 7f2c1f9a6700 10 monclient: init 2023-09-30T01:33:03.760 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.738+0000 7f2c1f9a6700 10 monclient: _reopen_session rank -1 2023-09-30T01:33:03.760 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.738+0000 7f2c1f9a6700 10 monclient: _add_conns ranks=[7,4,8,2,3,5,0,6,1] 2023-09-30T01:33:03.760 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.738+0000 7f2c1f9a6700 1 --2- >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7f2c1809ddb0 0x7f2c18096b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:33:03.760 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.738+0000 7f2c1f9a6700 10 monclient(hunting): picked mon.noname-i con 0x7f2c1809ddb0 addr [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] 2023-09-30T01:33:03.761 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.738+0000 7f2c1f9a6700 1 --2- >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f2c1809e180 0x7f2c1806d040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:33:03.761 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.738+0000 7f2c1f9a6700 10 monclient(hunting): picked mon.noname-c con 0x7f2c1809e180 addr [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] 2023-09-30T01:33:03.761 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.738+0000 7f2c1f9a6700 1 --2- >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f2c181305f0 0x7f2c1806d580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:33:03.761 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.738+0000 7f2c1f9a6700 10 monclient(hunting): picked mon.noname-e con 0x7f2c181305f0 addr [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] 2023-09-30T01:33:03.761 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.738+0000 7f2c1f9a6700 10 monclient(hunting): start opening mon connection 2023-09-30T01:33:03.761 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.738+0000 7f2c1f9a6700 1 -- --> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] -- mon_getmap magic: 0 v1 -- 0x7f2c18097df0 con 0x7f2c1809e180 2023-09-30T01:33:03.761 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.738+0000 7f2c1f9a6700 10 monclient(hunting): start opening mon connection 2023-09-30T01:33:03.761 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.738+0000 7f2c1f9a6700 1 -- --> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] -- mon_getmap magic: 0 v1 -- 0x7f2c1808a8b0 con 0x7f2c1809ddb0 2023-09-30T01:33:03.761 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.738+0000 7f2c1f9a6700 10 monclient(hunting): start opening mon connection 2023-09-30T01:33:03.761 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.738+0000 7f2c1f9a6700 1 -- --> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] -- mon_getmap magic: 0 v1 -- 0x7f2c1808a1b0 con 0x7f2c181305f0 2023-09-30T01:33:03.761 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.738+0000 7f2c1f9a6700 10 monclient(hunting): _renew_subs 2023-09-30T01:33:03.761 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.738+0000 7f2c1f9a6700 10 monclient(hunting): authenticate will time out at 855.779099s 2023-09-30T01:33:03.762 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.738+0000 7f2c1cf41700 1 --2- >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f2c1809e180 0x7f2c1806d040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:33:03.762 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.738+0000 7f2c1df43700 1 --2- >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f2c181305f0 0x7f2c1806d580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:33:03.762 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.738+0000 7f2c1cf41700 1 --2- >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f2c1809e180 0x7f2c1806d040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3302/0 says I am v2:172.21.15.88:50386/0 (socket says 172.21.15.88:50386) 2023-09-30T01:33:03.762 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.738+0000 7f2c1cf41700 1 -- 172.21.15.88:0/1959404386 learned_addr learned my addr 172.21.15.88:0/1959404386 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-09-30T01:33:03.762 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.738+0000 7f2c1cf41700 10 monclient(hunting): get_auth_request con 0x7f2c1809e180 auth_method 0 2023-09-30T01:33:03.762 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.738+0000 7f2c1cf41700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:33:03.762 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.738+0000 7f2c1cf41700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:33:03.762 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.738+0000 7f2c1cf41700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:33:03.762 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.738+0000 7f2c1df43700 10 monclient(hunting): get_auth_request con 0x7f2c181305f0 auth_method 0 2023-09-30T01:33:03.762 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.738+0000 7f2c1df43700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:33:03.762 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.738+0000 7f2c1df43700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:33:03.763 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.738+0000 7f2c1df43700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:33:03.763 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.738+0000 7f2c1cf41700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:33:03.763 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.738+0000 7f2c1cf41700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:33:03.763 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.738+0000 7f2c1cf41700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:33:03.763 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.738+0000 7f2c1d742700 1 --2- 172.21.15.88:0/1959404386 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7f2c1809ddb0 0x7f2c18096b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:33:03.763 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.738+0000 7f2c1df43700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:33:03.763 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.738+0000 7f2c1df43700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:33:03.763 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.738+0000 7f2c1df43700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:33:03.763 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.738+0000 7f2c1d742700 10 monclient(hunting): get_auth_request con 0x7f2c1809ddb0 auth_method 0 2023-09-30T01:33:03.763 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.738+0000 7f2c1d742700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:33:03.763 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.738+0000 7f2c1d742700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:33:03.763 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.738+0000 7f2c1d742700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:33:03.763 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.738+0000 7f2c1cf41700 10 monclient(hunting): handle_auth_done global_id 4252 payload 824 2023-09-30T01:33:03.764 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.738+0000 7f2c1cf41700 1 -- 172.21.15.88:0/1959404386 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f2c181305f0 msgr2=0x7f2c1806d580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:33:03.764 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.738+0000 7f2c1cf41700 1 --2- 172.21.15.88:0/1959404386 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f2c181305f0 0x7f2c1806d580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:03.764 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.738+0000 7f2c1cf41700 1 -- 172.21.15.88:0/1959404386 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7f2c1809ddb0 msgr2=0x7f2c18096b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:33:03.764 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.738+0000 7f2c1cf41700 1 --2- 172.21.15.88:0/1959404386 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7f2c1809ddb0 0x7f2c18096b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:03.764 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.738+0000 7f2c1cf41700 10 monclient: _finish_hunting 0 2023-09-30T01:33:03.764 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.738+0000 7f2c1cf41700 1 monclient: found mon.noname-c 2023-09-30T01:33:03.764 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.738+0000 7f2c1cf41700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:33:03.764 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.738+0000 7f2c1cf41700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.88:3302/0 2023-09-30T01:33:03.764 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.738+0000 7f2c1cf41700 1 -- 172.21.15.88:0/1959404386 --> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c08009a20 con 0x7f2c1809e180 2023-09-30T01:33:03.764 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.738+0000 7f2c1d742700 1 --2- 172.21.15.88:0/1959404386 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7f2c1809ddb0 0x7f2c18096b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-30T01:33:03.764 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.738+0000 7f2c1df43700 1 --2- 172.21.15.88:0/1959404386 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f2c181305f0 0x7f2c1806d580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:33:03.764 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.742+0000 7f2c1cf41700 1 --2- 172.21.15.88:0/1959404386 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f2c1809e180 0x7f2c1806d040 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f2c0c00bb20 tx=0x7f2c0c019000 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:33:03.765 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.742+0000 7f2c157fa700 1 -- 172.21.15.88:0/1959404386 <== mon.4 v2:172.21.15.88:3302/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f2c0c019480 con 0x7f2c1809e180 2023-09-30T01:33:03.765 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.742+0000 7f2c157fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:33:03.765 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.742+0000 7f2c157fa700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T01:33:03.765 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.742+0000 7f2c157fa700 10 monclient: dump: 2023-09-30T01:33:03.765 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:33:03.765 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:33:03.765 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:03.765 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:03.765 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:33:03.765 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:33:03.765 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:33:03.765 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:33:03.765 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:33:03.766 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:33:03.766 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:33:03.766 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:33:03.766 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:33:03.766 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:33:03.766 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:33:03.766 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:33:03.766 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.742+0000 7f2c157fa700 10 monclient: _finish_auth 0 2023-09-30T01:33:03.766 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.742+0000 7f2c157fa700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:33:03.766 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.742+0000 7f2c1f9a6700 5 monclient: authenticate success, global_id 4252 2023-09-30T01:33:03.766 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.742+0000 7f2c157fa700 1 -- 172.21.15.88:0/1959404386 <== mon.4 v2:172.21.15.88:3302/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c0c006ad0 con 0x7f2c1809e180 2023-09-30T01:33:03.766 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.742+0000 7f2c1f9a6700 10 monclient: _renew_subs 2023-09-30T01:33:03.767 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.742+0000 7f2c1f9a6700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.88:3302/0 2023-09-30T01:33:03.767 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.742+0000 7f2c1f9a6700 1 -- 172.21.15.88:0/1959404386 --> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2c0c00b790 con 0x7f2c1809e180 2023-09-30T01:33:03.767 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.742+0000 7f2c157fa700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:33:03.767 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.742+0000 7f2c157fa700 1 -- 172.21.15.88:0/1959404386 <== mon.4 v2:172.21.15.88:3302/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f2c0c0183e0 con 0x7f2c1809e180 2023-09-30T01:33:03.767 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.742+0000 7f2c157fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:33:03.767 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.742+0000 7f2c157fa700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T01:33:03.768 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.742+0000 7f2c177fe700 4 set_mon_vals no callback set 2023-09-30T01:33:03.768 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.742+0000 7f2c157fa700 10 monclient: dump: 2023-09-30T01:33:03.768 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:33:03.768 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:33:03.768 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:03.768 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:03.768 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:33:03.768 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:33:03.768 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:33:03.768 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:33:03.769 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:33:03.769 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:33:03.769 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:33:03.769 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:33:03.769 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:33:03.769 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:33:03.769 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:33:03.769 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:33:03.769 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.742+0000 7f2c1f9a6700 10 monclient: _renew_subs 2023-09-30T01:33:03.769 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.742+0000 7f2c1f9a6700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.88:3302/0 2023-09-30T01:33:03.769 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.742+0000 7f2c1f9a6700 1 -- 172.21.15.88:0/1959404386 --> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c1806e9a0 con 0x7f2c1809e180 2023-09-30T01:33:03.770 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.742+0000 7f2c157fa700 1 -- 172.21.15.88:0/1959404386 <== mon.4 v2:172.21.15.88:3302/0 4 ==== mgrmap(e 4) v1 ==== 40088+0+0 (secure 0 0 0) 0x7f2c0c010680 con 0x7f2c1809e180 2023-09-30T01:33:03.770 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.742+0000 7f2c1f9a6700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:33:03.770 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.742+0000 7f2c1f9a6700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:33:03.770 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.742+0000 7f2c1f9a6700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.88:3302/0 2023-09-30T01:33:03.770 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.742+0000 7f2c1f9a6700 1 -- 172.21.15.88:0/1959404386 --> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2c18071960 con 0x7f2c1809e180 2023-09-30T01:33:03.770 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.742+0000 7f2c157fa700 1 --2- 172.21.15.88:0/1959404386 >> v2:172.21.15.107:6812/29926 conn(0x7f2c040332d0 0x7f2c04035780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:33:03.770 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.742+0000 7f2c157fa700 1 -- 172.21.15.88:0/1959404386 <== mon.4 v2:172.21.15.88:3302/0 5 ==== osd_map(14..14 src has 1..14) v4 ==== 3502+0+0 (secure 0 0 0) 0x7f2c0c014070 con 0x7f2c1809e180 2023-09-30T01:33:03.770 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.742+0000 7f2c1d742700 1 --2- 172.21.15.88:0/1959404386 >> v2:172.21.15.107:6812/29926 conn(0x7f2c040332d0 0x7f2c04035780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:33:03.770 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.742+0000 7f2c1d742700 10 monclient: get_auth_request con 0x7f2c040332d0 auth_method 0 2023-09-30T01:33:03.770 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.742+0000 7f2c1d742700 1 --2- 172.21.15.88:0/1959404386 >> v2:172.21.15.107:6812/29926 conn(0x7f2c040332d0 0x7f2c04035780 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f2c080099f0 tx=0x7f2c0800b040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:33:03.771 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.750+0000 7f2c157fa700 1 -- 172.21.15.88:0/1959404386 <== mon.4 v2:172.21.15.88:3302/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f2c0c017de0 con 0x7f2c1809e180 2023-09-30T01:33:03.771 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.750+0000 7f2c157fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:33:03.771 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.750+0000 7f2c157fa700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:33:03.912 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.910+0000 7f2c1f9a6700 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2023-09-30T01:33:03.913 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.910+0000 7f2c1f9a6700 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2023-09-30T01:33:03.913 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.910+0000 7f2c1f9a6700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.88:3302/0 2023-09-30T01:33:03.913 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.910+0000 7f2c1f9a6700 1 -- 172.21.15.88:0/1959404386 --> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2c18064b80 con 0x7f2c1809e180 2023-09-30T01:33:03.913 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.910+0000 7f2c157fa700 1 -- 172.21.15.88:0/1959404386 <== mon.4 v2:172.21.15.88:3302/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v14) v1 ==== 74+0+10002 (secure 0 0 0) 0x7f2c0c027030 con 0x7f2c1809e180 2023-09-30T01:33:03.914 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.910+0000 7f2c157fa700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2023-09-30T01:33:03.914 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.910+0000 7f2c157fa700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:33:03.914 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-09-30T01:33:03.914 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":14,"fsid":"650badb5-899b-4472-88a6-cfdccee24d52","created":"2023-09-30T01:32:51.511154+0000","modified":"2023-09-30T01:33:03.267756+0000","last_up_change":"2023-09-30T01:32:59.732301+0000","last_in_change":"2023-09-30T01:32:53.067458+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":"rbd","create_time":"2023-09-30T01:33:01.387368+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":".mgr","create_time":"2023-09-30T01:33:01.712290+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"1243f825-b99c-4546-a77d-90b8637698b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6808","nonce":30177}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6809","nonce":30177}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6811","nonce":30177}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6810","nonce":30177}]},"public_addr":"172.21.15.88:6808/30177","cluster_addr":"172.21.15.88:6809/30177","heartbeat_back_addr":"172.21.15.88:6811/30177","heartbeat_front_addr":"172.21.15.88:6810/30177","state":["exists","up"]},{"osd":1,"uuid":"e9831b75-399b-4d6d-bf97-9ef104adfdb5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6804","nonce":30175}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6805","nonce":30175}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6807","nonce":30175}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6806","nonce":30175}]},"public_addr":"172.21.15.88:6804/30175","cluster_addr":"172.21.15.88:6805/30175","heartbeat_back_addr":"172.21.15.88:6807/30175","heartbeat_front_addr":"172.21.15.88:6806/30175","state":["exists","up"]},{"osd":2,"uuid":"fbc7be74-cd43-4c72-89c1-469c17055b85","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6800","nonce":30176}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6801","nonce":30176}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6803","nonce":30176}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6802","nonce":30176}]},"public_addr":"172.21.15.88:6800/30176","cluster_addr":"172.21.15.88:6801/30176","heartbeat_back_addr":"172.21.15.88:6803/30176","heartbeat_front_addr":"172.21.15.88:6802/30176","state":["exists","up"]},{"osd":3,"uuid":"ca888e00-8f75-4dc4-b70c-6cd46880fe74","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":30274}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6805","nonce":30274}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6807","nonce":30274}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":30274}]},"public_addr":"172.21.15.107:6804/30274","cluster_addr":"172.21.15.107:6805/30274","heartbeat_back_addr":"172.21.15.107:6807/30274","heartbeat_front_addr":"172.21.15.107:6806/30274","state":["exists","up"]},{"osd":4,"uuid":"c8c73bd1-7674-4529-9104-8690e25af2fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":30276}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6809","nonce":30276}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6811","nonce":30276}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":30276}]},"public_addr":"172.21.15.107:6808/30276","cluster_addr":"172.21.15.107:6809/30276","heartbeat_back_addr":"172.21.15.107:6811/30276","heartbeat_front_addr":"172.21.15.107:6810/30276","state":["exists","up"]},{"osd":5,"uuid":"a3994780-9d6a-4bcd-b718-09f9d8774b3d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":30275}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6801","nonce":30275}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6803","nonce":30275}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":30275}]},"public_addr":"172.21.15.107:6800/30275","cluster_addr":"172.21.15.107:6801/30275","heartbeat_back_addr":"172.21.15.107:6803/30275","heartbeat_front_addr":"172.21.15.107:6802/30275","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T01:32:58.527368+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T01:32:58.431507+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T01:32:58.347215+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T01:32:58.501409+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T01:32:58.565697+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T01:32:58.313349+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-09-30T01:33:03.916 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.914+0000 7f2c1f9a6700 1 -- 172.21.15.88:0/1959404386 >> v2:172.21.15.107:6812/29926 conn(0x7f2c040332d0 msgr2=0x7f2c04035780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:33:03.917 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.914+0000 7f2c1f9a6700 1 --2- 172.21.15.88:0/1959404386 >> v2:172.21.15.107:6812/29926 conn(0x7f2c040332d0 0x7f2c04035780 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f2c080099f0 tx=0x7f2c0800b040 comp rx=0 tx=0).stop 2023-09-30T01:33:03.917 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.914+0000 7f2c1f9a6700 10 monclient: shutdown 2023-09-30T01:33:03.917 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.914+0000 7f2c1f9a6700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:33:03.917 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.914+0000 7f2c1f9a6700 1 -- 172.21.15.88:0/1959404386 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f2c1809e180 msgr2=0x7f2c1806d040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:33:03.917 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.914+0000 7f2c1f9a6700 1 --2- 172.21.15.88:0/1959404386 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f2c1809e180 0x7f2c1806d040 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f2c0c00bb20 tx=0x7f2c0c019000 comp rx=0 tx=0).stop 2023-09-30T01:33:03.917 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.914+0000 7f2c1f9a6700 1 -- 172.21.15.88:0/1959404386 shutdown_connections 2023-09-30T01:33:03.917 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.914+0000 7f2c1f9a6700 1 --2- 172.21.15.88:0/1959404386 >> v2:172.21.15.107:6812/29926 conn(0x7f2c040332d0 0x7f2c04035780 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:03.917 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.914+0000 7f2c1f9a6700 1 --2- 172.21.15.88:0/1959404386 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f2c181305f0 0x7f2c1806d580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:03.917 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.914+0000 7f2c1f9a6700 1 --2- 172.21.15.88:0/1959404386 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f2c1809e180 0x7f2c1806d040 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:03.917 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.914+0000 7f2c1f9a6700 1 --2- 172.21.15.88:0/1959404386 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7f2c1809ddb0 0x7f2c18096b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:03.917 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.914+0000 7f2c1f9a6700 1 -- 172.21.15.88:0/1959404386 >> 172.21.15.88:0/1959404386 conn(0x7f2c180a3440 msgr2=0x7f2c1809c770 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:33:03.918 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.914+0000 7f2c1f9a6700 1 -- 172.21.15.88:0/1959404386 shutdown_connections 2023-09-30T01:33:03.918 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:03.914+0000 7f2c1f9a6700 1 -- 172.21.15.88:0/1959404386 wait complete. 2023-09-30T01:33:03.927 INFO:tasks.ceph.ceph_manager.ceph:all up! 2023-09-30T01:33:03.928 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-09-30T01:33:04.032 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.030+0000 7f77c7fff700 10 monclient: get_monmap_and_config 2023-09-30T01:33:04.032 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.030+0000 7f77c7fff700 10 monclient: build_initial_monmap 2023-09-30T01:33:04.032 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.030+0000 7f77c7fff700 10 monclient: monmap: 2023-09-30T01:33:04.032 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2023-09-30T01:33:04.032 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:33:04.032 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:33:04.032621+0000 2023-09-30T01:33:04.032 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:33:04.032621+0000 2023-09-30T01:33:04.032 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2023-09-30T01:33:04.032 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:33:04.033 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a-legacy 2023-09-30T01:33:04.033 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.107:6789/0 mon.noname-f-legacy 2023-09-30T01:33:04.033 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.noname-b 2023-09-30T01:33:04.033 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.noname-g 2023-09-30T01:33:04.033 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.noname-c 2023-09-30T01:33:04.033 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.noname-h 2023-09-30T01:33:04.033 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.noname-d 2023-09-30T01:33:04.033 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.noname-i 2023-09-30T01:33:04.033 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.noname-e 2023-09-30T01:33:04.033 INFO:teuthology.orchestra.run.smithi088.stderr:9: v2:172.21.15.88:3300/0 mon.noname-a 2023-09-30T01:33:04.033 INFO:teuthology.orchestra.run.smithi088.stderr:10: v2:172.21.15.107:3300/0 mon.noname-f 2023-09-30T01:33:04.033 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:33:04.041 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.030+0000 7f77c7fff700 1 Processor -- start 2023-09-30T01:33:04.041 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.030+0000 7f77c7fff700 1 -- start start 2023-09-30T01:33:04.041 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.030+0000 7f77c7fff700 10 monclient: init 2023-09-30T01:33:04.041 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.030+0000 7f77c7fff700 10 monclient: _reopen_session rank -1 2023-09-30T01:33:04.041 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.030+0000 7f77c7fff700 10 monclient: _add_conns ranks=[6,8,10,0,7,9,2,3,4,1,5] 2023-09-30T01:33:04.041 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.030+0000 7f77c7fff700 1 --2- >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7f77c8097a40 0x7f77c8132700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:33:04.041 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.030+0000 7f77c7fff700 10 monclient(hunting): picked mon.noname-d con 0x7f77c8097a40 addr [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] 2023-09-30T01:33:04.041 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.030+0000 7f77c7fff700 1 --2- >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f77c81311c0 0x7f77c8132c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:33:04.042 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.030+0000 7f77c7fff700 10 monclient(hunting): picked mon.noname-e con 0x7f77c81311c0 addr [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] 2023-09-30T01:33:04.042 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.030+0000 7f77c7fff700 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f77c81332e0 0x7f77c8133710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:33:04.042 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.030+0000 7f77c7fff700 10 monclient(hunting): picked mon.noname-f con 0x7f77c81332e0 addr v2:172.21.15.107:3300/0 2023-09-30T01:33:04.042 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.030+0000 7f77c7fff700 10 monclient(hunting): start opening mon connection 2023-09-30T01:33:04.042 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.030+0000 7f77c7fff700 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f77c8131590 con 0x7f77c81332e0 2023-09-30T01:33:04.042 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.030+0000 7f77c7fff700 10 monclient(hunting): start opening mon connection 2023-09-30T01:33:04.042 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.030+0000 7f77c7fff700 1 -- --> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] -- mon_getmap magic: 0 v1 -- 0x7f77c809a5d0 con 0x7f77c8097a40 2023-09-30T01:33:04.042 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.030+0000 7f77c7fff700 10 monclient(hunting): start opening mon connection 2023-09-30T01:33:04.042 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.030+0000 7f77c7fff700 1 -- --> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] -- mon_getmap magic: 0 v1 -- 0x7f77c8097e10 con 0x7f77c81311c0 2023-09-30T01:33:04.042 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.030+0000 7f77c7fff700 10 monclient(hunting): _renew_subs 2023-09-30T01:33:04.042 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.030+0000 7f77c7fff700 10 monclient(hunting): authenticate will time out at 856.070051s 2023-09-30T01:33:04.043 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.030+0000 7f77c6ffd700 1 --2- >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7f77c8097a40 0x7f77c8132700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:33:04.043 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.030+0000 7f77c67fc700 1 --2- >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f77c81311c0 0x7f77c8132c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:33:04.043 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.030+0000 7f77c6ffd700 1 --2- >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7f77c8097a40 0x7f77c8132700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3303/0 says I am v2:172.21.15.88:53522/0 (socket says 172.21.15.88:53522) 2023-09-30T01:33:04.043 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.030+0000 7f77c6ffd700 1 -- 172.21.15.88:0/1642660362 learned_addr learned my addr 172.21.15.88:0/1642660362 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-09-30T01:33:04.043 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.030+0000 7f77c6ffd700 10 monclient(hunting): get_auth_request con 0x7f77c8097a40 auth_method 0 2023-09-30T01:33:04.043 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.030+0000 7f77c6ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:33:04.043 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.030+0000 7f77c6ffd700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:33:04.043 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.030+0000 7f77c6ffd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:33:04.043 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.030+0000 7f77c67fc700 10 monclient(hunting): get_auth_request con 0x7f77c81311c0 auth_method 0 2023-09-30T01:33:04.043 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.030+0000 7f77c67fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:33:04.043 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.030+0000 7f77c67fc700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:33:04.044 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.030+0000 7f77c67fc700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:33:04.044 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.030+0000 7f77c6ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:33:04.044 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.030+0000 7f77c6ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:33:04.044 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.030+0000 7f77c6ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:33:04.044 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.030+0000 7f77c77fe700 1 --2- 172.21.15.88:0/1642660362 >> v2:172.21.15.107:3300/0 conn(0x7f77c81332e0 0x7f77c8133710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:33:04.044 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.030+0000 7f77c77fe700 10 monclient(hunting): get_auth_request con 0x7f77c81332e0 auth_method 0 2023-09-30T01:33:04.044 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.030+0000 7f77c77fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:33:04.044 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.030+0000 7f77c77fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:33:04.044 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.030+0000 7f77c77fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:33:04.044 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.030+0000 7f77c67fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:33:04.044 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.030+0000 7f77c67fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:33:04.044 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.030+0000 7f77c67fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:33:04.045 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.030+0000 7f77c6ffd700 10 monclient(hunting): handle_auth_done global_id 4191 payload 274 2023-09-30T01:33:04.045 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.030+0000 7f77c6ffd700 1 -- 172.21.15.88:0/1642660362 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f77c81311c0 msgr2=0x7f77c8132c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:33:04.045 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.030+0000 7f77c6ffd700 1 --2- 172.21.15.88:0/1642660362 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f77c81311c0 0x7f77c8132c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:04.045 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.030+0000 7f77c6ffd700 1 -- 172.21.15.88:0/1642660362 >> v2:172.21.15.107:3300/0 conn(0x7f77c81332e0 msgr2=0x7f77c8133710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:33:04.045 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.030+0000 7f77c6ffd700 1 --2- 172.21.15.88:0/1642660362 >> v2:172.21.15.107:3300/0 conn(0x7f77c81332e0 0x7f77c8133710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:04.045 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.030+0000 7f77c6ffd700 10 monclient: _finish_hunting 0 2023-09-30T01:33:04.045 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.030+0000 7f77c6ffd700 1 monclient: found mon.noname-d 2023-09-30T01:33:04.045 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.030+0000 7f77c6ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:33:04.045 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.030+0000 7f77c6ffd700 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.88:3303/0 2023-09-30T01:33:04.045 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.030+0000 7f77c6ffd700 1 -- 172.21.15.88:0/1642660362 --> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77c804bb20 con 0x7f77c8097a40 2023-09-30T01:33:04.045 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.030+0000 7f77c77fe700 1 --2- 172.21.15.88:0/1642660362 >> v2:172.21.15.107:3300/0 conn(0x7f77c81332e0 0x7f77c8133710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-30T01:33:04.046 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.030+0000 7f77c6ffd700 1 --2- 172.21.15.88:0/1642660362 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7f77c8097a40 0x7f77c8132700 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f77c0009f20 tx=0x7f77c0009840 comp rx=0 tx=0).ready entity=mon.6 client_cookie=3a02f1d8469c70e3 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:33:04.046 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.030+0000 7f77c5ffb700 1 -- 172.21.15.88:0/1642660362 <== mon.6 v2:172.21.15.88:3303/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f77c000e070 con 0x7f77c8097a40 2023-09-30T01:33:04.046 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.030+0000 7f77c5ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:33:04.046 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.030+0000 7f77c5ffb700 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2023-09-30T01:33:04.046 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.030+0000 7f77c5ffb700 10 monclient: dump: 2023-09-30T01:33:04.046 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:33:04.046 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:33:04.046 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:04.046 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:04.046 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:33:04.046 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:33:04.046 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:33:04.047 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:33:04.047 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:33:04.047 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:33:04.047 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:33:04.047 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:33:04.047 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:33:04.047 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:33:04.047 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:33:04.047 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:33:04.047 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.030+0000 7f77c5ffb700 10 monclient: _finish_auth 0 2023-09-30T01:33:04.047 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.030+0000 7f77c5ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:33:04.047 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.030+0000 7f77c7fff700 5 monclient: authenticate success, global_id 4191 2023-09-30T01:33:04.047 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.030+0000 7f77c5ffb700 1 -- 172.21.15.88:0/1642660362 <== mon.6 v2:172.21.15.88:3303/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f77c000be50 con 0x7f77c8097a40 2023-09-30T01:33:04.048 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.030+0000 7f77c5ffb700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:33:04.048 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.030+0000 7f77c7fff700 10 monclient: get_monmap_and_config success 2023-09-30T01:33:04.048 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.030+0000 7f77c5ffb700 1 -- 172.21.15.88:0/1642660362 <== mon.6 v2:172.21.15.88:3303/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f77c00092e0 con 0x7f77c8097a40 2023-09-30T01:33:04.048 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.030+0000 7f77c7fff700 4 set_mon_vals no callback set 2023-09-30T01:33:04.048 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.030+0000 7f77c5ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:33:04.048 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.030+0000 7f77c5ffb700 10 monclient: got monmap 1 from mon.d (according to old e1) 2023-09-30T01:33:04.048 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.030+0000 7f77c5ffb700 10 monclient: dump: 2023-09-30T01:33:04.048 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:33:04.048 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:33:04.048 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:04.048 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:04.048 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:33:04.049 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:33:04.049 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:33:04.049 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:33:04.049 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:33:04.049 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:33:04.049 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:33:04.049 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:33:04.049 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:33:04.049 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:33:04.049 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:33:04.049 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:33:04.049 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.030+0000 7f77c7fff700 10 monclient: shutdown 2023-09-30T01:33:04.049 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.030+0000 7f77c7fff700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:33:04.050 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.030+0000 7f77c7fff700 1 -- 172.21.15.88:0/1642660362 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7f77c8097a40 msgr2=0x7f77c8132700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:33:04.050 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.030+0000 7f77c7fff700 1 --2- 172.21.15.88:0/1642660362 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7f77c8097a40 0x7f77c8132700 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f77c0009f20 tx=0x7f77c0009840 comp rx=0 tx=0).stop 2023-09-30T01:33:04.050 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.030+0000 7f77c7fff700 1 -- 172.21.15.88:0/1642660362 shutdown_connections 2023-09-30T01:33:04.050 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.030+0000 7f77c7fff700 1 --2- 172.21.15.88:0/1642660362 >> v2:172.21.15.107:3300/0 conn(0x7f77c81332e0 0x7f77c8133710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:04.050 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.030+0000 7f77c7fff700 1 --2- 172.21.15.88:0/1642660362 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f77c81311c0 0x7f77c8132c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:04.050 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.030+0000 7f77c7fff700 1 --2- 172.21.15.88:0/1642660362 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7f77c8097a40 0x7f77c8132700 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:04.050 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.030+0000 7f77c7fff700 1 -- 172.21.15.88:0/1642660362 >> 172.21.15.88:0/1642660362 conn(0x7f77c806e5e0 msgr2=0x7f77c806e9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:33:04.050 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.030+0000 7f77c7fff700 1 -- 172.21.15.88:0/1642660362 shutdown_connections 2023-09-30T01:33:04.050 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.030+0000 7f77c7fff700 1 -- 172.21.15.88:0/1642660362 wait complete. 2023-09-30T01:33:04.050 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.034+0000 7f77c7fff700 10 monclient: build_initial_monmap 2023-09-30T01:33:04.050 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.034+0000 7f77c7fff700 10 monclient: monmap: 2023-09-30T01:33:04.050 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2023-09-30T01:33:04.050 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:33:04.051 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 0.000000 2023-09-30T01:33:04.051 INFO:teuthology.orchestra.run.smithi088.stderr:created 0.000000 2023-09-30T01:33:04.051 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2023-09-30T01:33:04.051 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 1 2023-09-30T01:33:04.051 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.noname-a 2023-09-30T01:33:04.051 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.noname-f 2023-09-30T01:33:04.051 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.noname-b 2023-09-30T01:33:04.051 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.noname-g 2023-09-30T01:33:04.051 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.noname-c 2023-09-30T01:33:04.051 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.noname-h 2023-09-30T01:33:04.051 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.noname-d 2023-09-30T01:33:04.051 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.noname-i 2023-09-30T01:33:04.052 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.noname-e 2023-09-30T01:33:04.052 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:33:04.052 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.034+0000 7f77c7fff700 1 Processor -- start 2023-09-30T01:33:04.052 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.034+0000 7f77c7fff700 1 -- start start 2023-09-30T01:33:04.052 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.034+0000 7f77c7fff700 10 monclient: init 2023-09-30T01:33:04.052 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.034+0000 7f77c7fff700 10 monclient: _reopen_session rank -1 2023-09-30T01:33:04.052 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.034+0000 7f77c7fff700 10 monclient: _add_conns ranks=[6,0,5,7,3,2,8,4,1] 2023-09-30T01:33:04.052 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.034+0000 7f77c7fff700 1 --2- >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7f77c8097a40 0x7f77c811d830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:33:04.052 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.034+0000 7f77c7fff700 10 monclient(hunting): picked mon.noname-d con 0x7f77c8097a40 addr [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] 2023-09-30T01:33:04.052 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.034+0000 7f77c7fff700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f77c81311c0 0x7f77c811dd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:33:04.052 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.034+0000 7f77c7fff700 10 monclient(hunting): picked mon.noname-a con 0x7f77c81311c0 addr [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] 2023-09-30T01:33:04.052 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.034+0000 7f77c7fff700 1 --2- >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7f77c81332e0 0x7f77c811e2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:33:04.052 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.034+0000 7f77c7fff700 10 monclient(hunting): picked mon.noname-h con 0x7f77c81332e0 addr [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] 2023-09-30T01:33:04.053 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.034+0000 7f77c7fff700 10 monclient(hunting): start opening mon connection 2023-09-30T01:33:04.053 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.034+0000 7f77c7fff700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77c8099c30 con 0x7f77c81311c0 2023-09-30T01:33:04.053 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.034+0000 7f77c7fff700 10 monclient(hunting): start opening mon connection 2023-09-30T01:33:04.053 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.034+0000 7f77c7fff700 1 -- --> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] -- mon_getmap magic: 0 v1 -- 0x7f77c80996b0 con 0x7f77c81332e0 2023-09-30T01:33:04.053 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.034+0000 7f77c7fff700 10 monclient(hunting): start opening mon connection 2023-09-30T01:33:04.053 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.034+0000 7f77c7fff700 1 -- --> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] -- mon_getmap magic: 0 v1 -- 0x7f77c8121040 con 0x7f77c8097a40 2023-09-30T01:33:04.053 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.034+0000 7f77c7fff700 10 monclient(hunting): _renew_subs 2023-09-30T01:33:04.053 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.034+0000 7f77c7fff700 10 monclient(hunting): authenticate will time out at 856.074056s 2023-09-30T01:33:04.053 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.034+0000 7f77c6ffd700 1 --2- >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7f77c8097a40 0x7f77c811d830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:33:04.053 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.034+0000 7f77c67fc700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f77c81311c0 0x7f77c811dd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:33:04.053 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.034+0000 7f77c67fc700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f77c81311c0 0x7f77c811dd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:57764/0 (socket says 172.21.15.88:57764) 2023-09-30T01:33:04.053 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.034+0000 7f77c67fc700 1 -- 172.21.15.88:0/3381877168 learned_addr learned my addr 172.21.15.88:0/3381877168 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-09-30T01:33:04.054 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.034+0000 7f77c67fc700 10 monclient(hunting): get_auth_request con 0x7f77c81311c0 auth_method 0 2023-09-30T01:33:04.054 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.034+0000 7f77c67fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:33:04.054 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.034+0000 7f77c67fc700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:33:04.054 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.034+0000 7f77c67fc700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:33:04.054 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.034+0000 7f77c77fe700 1 --2- 172.21.15.88:0/3381877168 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7f77c81332e0 0x7f77c811e2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:33:04.054 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.034+0000 7f77c67fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:33:04.054 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.034+0000 7f77c67fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:33:04.054 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.034+0000 7f77c67fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:33:04.054 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.034+0000 7f77c77fe700 10 monclient(hunting): get_auth_request con 0x7f77c81332e0 auth_method 0 2023-09-30T01:33:04.054 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.034+0000 7f77c77fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:33:04.054 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.034+0000 7f77c77fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:33:04.054 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.034+0000 7f77c77fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:33:04.055 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.034+0000 7f77c6ffd700 10 monclient(hunting): get_auth_request con 0x7f77c8097a40 auth_method 0 2023-09-30T01:33:04.055 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.034+0000 7f77c6ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:33:04.055 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.034+0000 7f77c6ffd700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:33:04.055 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.034+0000 7f77c6ffd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:33:04.055 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.034+0000 7f77c77fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:33:04.055 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.034+0000 7f77c77fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:33:04.055 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.034+0000 7f77c77fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:33:04.055 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.034+0000 7f77c67fc700 10 monclient(hunting): handle_auth_done global_id 4266 payload 824 2023-09-30T01:33:04.055 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.034+0000 7f77c67fc700 1 -- 172.21.15.88:0/3381877168 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7f77c8097a40 msgr2=0x7f77c811d830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:33:04.055 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.034+0000 7f77c67fc700 1 --2- 172.21.15.88:0/3381877168 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7f77c8097a40 0x7f77c811d830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:04.055 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.034+0000 7f77c67fc700 1 -- 172.21.15.88:0/3381877168 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7f77c81332e0 msgr2=0x7f77c811e2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:33:04.055 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.034+0000 7f77c67fc700 1 --2- 172.21.15.88:0/3381877168 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7f77c81332e0 0x7f77c811e2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:04.055 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.034+0000 7f77c67fc700 10 monclient: _finish_hunting 0 2023-09-30T01:33:04.056 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.034+0000 7f77c67fc700 1 monclient: found mon.noname-a 2023-09-30T01:33:04.056 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.034+0000 7f77c67fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:33:04.056 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.034+0000 7f77c67fc700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.88:3300/0 2023-09-30T01:33:04.056 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.034+0000 7f77c67fc700 1 -- 172.21.15.88:0/3381877168 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77c0009ae0 con 0x7f77c81311c0 2023-09-30T01:33:04.056 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.034+0000 7f77c6ffd700 1 --2- 172.21.15.88:0/3381877168 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7f77c8097a40 0x7f77c811d830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-30T01:33:04.056 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.034+0000 7f77c67fc700 1 --2- 172.21.15.88:0/3381877168 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f77c81311c0 0x7f77c811dd70 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f77b0009e30 tx=0x7f77b00091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:33:04.056 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.034+0000 7f77beffd700 1 -- 172.21.15.88:0/3381877168 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f77b0017c60 con 0x7f77c81311c0 2023-09-30T01:33:04.056 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.034+0000 7f77beffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:33:04.056 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.034+0000 7f77beffd700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T01:33:04.056 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.034+0000 7f77beffd700 10 monclient: dump: 2023-09-30T01:33:04.056 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:33:04.056 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:33:04.057 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:04.057 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:04.057 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:33:04.057 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:33:04.057 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:33:04.057 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:33:04.057 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:33:04.057 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:33:04.057 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:33:04.057 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:33:04.057 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:33:04.057 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:33:04.058 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:33:04.058 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:33:04.058 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.034+0000 7f77beffd700 10 monclient: _finish_auth 0 2023-09-30T01:33:04.058 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.034+0000 7f77beffd700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:33:04.058 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.034+0000 7f77beffd700 1 -- 172.21.15.88:0/3381877168 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f77b0017de0 con 0x7f77c81311c0 2023-09-30T01:33:04.058 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.034+0000 7f77c7fff700 5 monclient: authenticate success, global_id 4266 2023-09-30T01:33:04.058 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.034+0000 7f77c7fff700 10 monclient: _renew_subs 2023-09-30T01:33:04.058 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.034+0000 7f77c7fff700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.88:3300/0 2023-09-30T01:33:04.058 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.034+0000 7f77c7fff700 1 -- 172.21.15.88:0/3381877168 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f77c8130df0 con 0x7f77c81311c0 2023-09-30T01:33:04.058 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.034+0000 7f77beffd700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:33:04.058 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.038+0000 7f77beffd700 1 -- 172.21.15.88:0/3381877168 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f77b001e3a0 con 0x7f77c81311c0 2023-09-30T01:33:04.058 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.038+0000 7f77beffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:33:04.059 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.038+0000 7f77beffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:33:04.059 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.038+0000 7f77c57fa700 4 set_mon_vals no callback set 2023-09-30T01:33:04.059 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.038+0000 7f77beffd700 10 monclient: dump: 2023-09-30T01:33:04.059 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:33:04.059 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:33:04.059 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:04.059 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:04.059 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:33:04.059 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:33:04.059 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:33:04.059 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:33:04.059 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:33:04.059 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:33:04.060 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:33:04.060 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:33:04.060 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:33:04.060 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:33:04.060 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:33:04.060 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:33:04.060 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.038+0000 7f77c7fff700 10 monclient: _renew_subs 2023-09-30T01:33:04.060 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.038+0000 7f77c7fff700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.88:3300/0 2023-09-30T01:33:04.060 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.038+0000 7f77c7fff700 1 -- 172.21.15.88:0/3381877168 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f77c811f230 con 0x7f77c81311c0 2023-09-30T01:33:04.060 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.038+0000 7f77beffd700 1 -- 172.21.15.88:0/3381877168 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 4) v1 ==== 40088+0+0 (secure 0 0 0) 0x7f77b000c640 con 0x7f77c81311c0 2023-09-30T01:33:04.060 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.038+0000 7f77c7fff700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:33:04.060 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.038+0000 7f77c7fff700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:33:04.061 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.038+0000 7f77c7fff700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.88:3300/0 2023-09-30T01:33:04.061 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.038+0000 7f77c7fff700 1 -- 172.21.15.88:0/3381877168 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f77c81211b0 con 0x7f77c81311c0 2023-09-30T01:33:04.061 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.038+0000 7f77beffd700 1 --2- 172.21.15.88:0/3381877168 >> v2:172.21.15.107:6812/29926 conn(0x7f77a80332d0 0x7f77a8035780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:33:04.061 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.038+0000 7f77beffd700 1 -- 172.21.15.88:0/3381877168 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(14..14 src has 1..14) v4 ==== 3502+0+0 (secure 0 0 0) 0x7f77b004b4f0 con 0x7f77c81311c0 2023-09-30T01:33:04.061 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.038+0000 7f77c6ffd700 1 --2- 172.21.15.88:0/3381877168 >> v2:172.21.15.107:6812/29926 conn(0x7f77a80332d0 0x7f77a8035780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:33:04.062 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.038+0000 7f77c6ffd700 10 monclient: get_auth_request con 0x7f77a80332d0 auth_method 0 2023-09-30T01:33:04.062 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.038+0000 7f77c6ffd700 1 --2- 172.21.15.88:0/3381877168 >> v2:172.21.15.107:6812/29926 conn(0x7f77a80332d0 0x7f77a8035780 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f77c0000f80 tx=0x7f77c0018000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:33:04.062 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.046+0000 7f77beffd700 1 -- 172.21.15.88:0/3381877168 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f77b00268f0 con 0x7f77c81311c0 2023-09-30T01:33:04.062 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.046+0000 7f77beffd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:33:04.062 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.046+0000 7f77beffd700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:33:04.208 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.206+0000 7f77c7fff700 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2023-09-30T01:33:04.209 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.206+0000 7f77c7fff700 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2023-09-30T01:33:04.209 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.206+0000 7f77c7fff700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.88:3300/0 2023-09-30T01:33:04.209 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.206+0000 7f77c7fff700 1 -- 172.21.15.88:0/3381877168 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f77c8071960 con 0x7f77c81311c0 2023-09-30T01:33:04.210 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.206+0000 7f77beffd700 1 -- 172.21.15.88:0/3381877168 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v14) v1 ==== 74+0+10002 (secure 0 0 0) 0x7f77b0024020 con 0x7f77c81311c0 2023-09-30T01:33:04.210 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.206+0000 7f77beffd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2023-09-30T01:33:04.210 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.206+0000 7f77beffd700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:33:04.210 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-09-30T01:33:04.210 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":14,"fsid":"650badb5-899b-4472-88a6-cfdccee24d52","created":"2023-09-30T01:32:51.511154+0000","modified":"2023-09-30T01:33:03.267756+0000","last_up_change":"2023-09-30T01:32:59.732301+0000","last_in_change":"2023-09-30T01:32:53.067458+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":"rbd","create_time":"2023-09-30T01:33:01.387368+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":".mgr","create_time":"2023-09-30T01:33:01.712290+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"1243f825-b99c-4546-a77d-90b8637698b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6808","nonce":30177}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6809","nonce":30177}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6811","nonce":30177}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6810","nonce":30177}]},"public_addr":"172.21.15.88:6808/30177","cluster_addr":"172.21.15.88:6809/30177","heartbeat_back_addr":"172.21.15.88:6811/30177","heartbeat_front_addr":"172.21.15.88:6810/30177","state":["exists","up"]},{"osd":1,"uuid":"e9831b75-399b-4d6d-bf97-9ef104adfdb5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6804","nonce":30175}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6805","nonce":30175}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6807","nonce":30175}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6806","nonce":30175}]},"public_addr":"172.21.15.88:6804/30175","cluster_addr":"172.21.15.88:6805/30175","heartbeat_back_addr":"172.21.15.88:6807/30175","heartbeat_front_addr":"172.21.15.88:6806/30175","state":["exists","up"]},{"osd":2,"uuid":"fbc7be74-cd43-4c72-89c1-469c17055b85","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6800","nonce":30176}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6801","nonce":30176}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6803","nonce":30176}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6802","nonce":30176}]},"public_addr":"172.21.15.88:6800/30176","cluster_addr":"172.21.15.88:6801/30176","heartbeat_back_addr":"172.21.15.88:6803/30176","heartbeat_front_addr":"172.21.15.88:6802/30176","state":["exists","up"]},{"osd":3,"uuid":"ca888e00-8f75-4dc4-b70c-6cd46880fe74","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":30274}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6805","nonce":30274}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6807","nonce":30274}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":30274}]},"public_addr":"172.21.15.107:6804/30274","cluster_addr":"172.21.15.107:6805/30274","heartbeat_back_addr":"172.21.15.107:6807/30274","heartbeat_front_addr":"172.21.15.107:6806/30274","state":["exists","up"]},{"osd":4,"uuid":"c8c73bd1-7674-4529-9104-8690e25af2fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":30276}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6809","nonce":30276}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6811","nonce":30276}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":30276}]},"public_addr":"172.21.15.107:6808/30276","cluster_addr":"172.21.15.107:6809/30276","heartbeat_back_addr":"172.21.15.107:6811/30276","heartbeat_front_addr":"172.21.15.107:6810/30276","state":["exists","up"]},{"osd":5,"uuid":"a3994780-9d6a-4bcd-b718-09f9d8774b3d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":30275}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6801","nonce":30275}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6803","nonce":30275}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":30275}]},"public_addr":"172.21.15.107:6800/30275","cluster_addr":"172.21.15.107:6801/30275","heartbeat_back_addr":"172.21.15.107:6803/30275","heartbeat_front_addr":"172.21.15.107:6802/30275","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T01:32:58.527368+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T01:32:58.431507+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T01:32:58.347215+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T01:32:58.501409+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T01:32:58.565697+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T01:32:58.313349+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-09-30T01:33:04.212 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.210+0000 7f77c7fff700 1 -- 172.21.15.88:0/3381877168 >> v2:172.21.15.107:6812/29926 conn(0x7f77a80332d0 msgr2=0x7f77a8035780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:33:04.212 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.210+0000 7f77c7fff700 1 --2- 172.21.15.88:0/3381877168 >> v2:172.21.15.107:6812/29926 conn(0x7f77a80332d0 0x7f77a8035780 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f77c0000f80 tx=0x7f77c0018000 comp rx=0 tx=0).stop 2023-09-30T01:33:04.212 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.210+0000 7f77c7fff700 10 monclient: shutdown 2023-09-30T01:33:04.212 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.210+0000 7f77c7fff700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:33:04.212 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.210+0000 7f77c7fff700 1 -- 172.21.15.88:0/3381877168 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f77c81311c0 msgr2=0x7f77c811dd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:33:04.213 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.210+0000 7f77c7fff700 1 --2- 172.21.15.88:0/3381877168 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f77c81311c0 0x7f77c811dd70 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f77b0009e30 tx=0x7f77b00091a0 comp rx=0 tx=0).stop 2023-09-30T01:33:04.213 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.210+0000 7f77c7fff700 1 -- 172.21.15.88:0/3381877168 shutdown_connections 2023-09-30T01:33:04.213 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.210+0000 7f77c7fff700 1 --2- 172.21.15.88:0/3381877168 >> v2:172.21.15.107:6812/29926 conn(0x7f77a80332d0 0x7f77a8035780 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:04.213 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.210+0000 7f77c7fff700 1 --2- 172.21.15.88:0/3381877168 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f77c81311c0 0x7f77c811dd70 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:04.213 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.210+0000 7f77c7fff700 1 --2- 172.21.15.88:0/3381877168 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7f77c81332e0 0x7f77c811e2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:04.213 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.210+0000 7f77c7fff700 1 --2- 172.21.15.88:0/3381877168 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7f77c8097a40 0x7f77c811d830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:04.213 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.210+0000 7f77c7fff700 1 -- 172.21.15.88:0/3381877168 >> 172.21.15.88:0/3381877168 conn(0x7f77c806e5e0 msgr2=0x7f77c80a45d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:33:04.213 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.210+0000 7f77c7fff700 1 -- 172.21.15.88:0/3381877168 shutdown_connections 2023-09-30T01:33:04.213 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.210+0000 7f77c7fff700 1 -- 172.21.15.88:0/3381877168 wait complete. 2023-09-30T01:33:04.224 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2023-09-30T01:33:04.224 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2023-09-30T01:33:04.225 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2023-09-30T01:33:04.225 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2023-09-30T01:33:04.226 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2023-09-30T01:33:04.226 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2023-09-30T01:33:04.339 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.334+0000 7f037147f700 10 monclient: get_monmap_and_config 2023-09-30T01:33:04.339 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.334+0000 7f037147f700 10 monclient: build_initial_monmap 2023-09-30T01:33:04.339 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.334+0000 7fca12e64700 10 monclient: get_monmap_and_config 2023-09-30T01:33:04.340 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.334+0000 7fca12e64700 10 monclient: build_initial_monmap 2023-09-30T01:33:04.340 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.334+0000 7fca12e64700 10 monclient: monmap: 2023-09-30T01:33:04.340 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2023-09-30T01:33:04.340 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:33:04.340 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:33:04.339742+0000 2023-09-30T01:33:04.340 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:33:04.339742+0000 2023-09-30T01:33:04.340 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2023-09-30T01:33:04.340 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:33:04.340 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a-legacy 2023-09-30T01:33:04.340 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.107:6789/0 mon.noname-f-legacy 2023-09-30T01:33:04.341 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.noname-b 2023-09-30T01:33:04.341 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.noname-g 2023-09-30T01:33:04.341 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.noname-c 2023-09-30T01:33:04.341 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.noname-h 2023-09-30T01:33:04.341 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.noname-d 2023-09-30T01:33:04.341 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.noname-i 2023-09-30T01:33:04.341 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.noname-e 2023-09-30T01:33:04.341 INFO:teuthology.orchestra.run.smithi088.stderr:9: v2:172.21.15.88:3300/0 mon.noname-a 2023-09-30T01:33:04.341 INFO:teuthology.orchestra.run.smithi088.stderr:10: v2:172.21.15.107:3300/0 mon.noname-f 2023-09-30T01:33:04.342 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:33:04.342 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.334+0000 7f037147f700 10 monclient: monmap: 2023-09-30T01:33:04.342 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2023-09-30T01:33:04.342 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:33:04.342 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:33:04.338915+0000 2023-09-30T01:33:04.343 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:33:04.338915+0000 2023-09-30T01:33:04.343 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2023-09-30T01:33:04.343 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:33:04.343 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a-legacy 2023-09-30T01:33:04.343 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.107:6789/0 mon.noname-f-legacy 2023-09-30T01:33:04.343 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.noname-b 2023-09-30T01:33:04.343 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.noname-g 2023-09-30T01:33:04.343 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.noname-c 2023-09-30T01:33:04.343 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.noname-h 2023-09-30T01:33:04.343 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.noname-d 2023-09-30T01:33:04.344 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.noname-i 2023-09-30T01:33:04.344 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.noname-e 2023-09-30T01:33:04.344 INFO:teuthology.orchestra.run.smithi088.stderr:9: v2:172.21.15.88:3300/0 mon.noname-a 2023-09-30T01:33:04.344 INFO:teuthology.orchestra.run.smithi088.stderr:10: v2:172.21.15.107:3300/0 mon.noname-f 2023-09-30T01:33:04.344 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:33:04.344 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.338+0000 7f037147f700 1 Processor -- start 2023-09-30T01:33:04.344 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.338+0000 7f037147f700 1 -- start start 2023-09-30T01:33:04.344 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.338+0000 7f037147f700 10 monclient: init 2023-09-30T01:33:04.344 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.338+0000 7fca12e64700 1 Processor -- start 2023-09-30T01:33:04.345 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.338+0000 7f037147f700 10 monclient: _reopen_session rank -1 2023-09-30T01:33:04.345 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.338+0000 7f037147f700 10 monclient: _add_conns ranks=[6,1,4,3,2,10,9,7,8,5,0] 2023-09-30T01:33:04.345 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.338+0000 7fca12e64700 1 -- start start 2023-09-30T01:33:04.345 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.338+0000 7fca12e64700 10 monclient: init 2023-09-30T01:33:04.347 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.338+0000 7f037147f700 1 --2- >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7f036c09be60 0x7f036c12f800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:33:04.347 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.338+0000 7f037147f700 10 monclient(hunting): picked mon.noname-d con 0x7f036c09be60 addr [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] 2023-09-30T01:33:04.348 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.338+0000 7f037147f700 10 monclient(hunting): picked mon.noname-f-legacy con 0x7f036c09d7c0 addr v1:172.21.15.107:6789/0 2023-09-30T01:33:04.348 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.338+0000 7f037147f700 1 --2- >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f036c09cf30 0x7f036c137e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:33:04.348 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.338+0000 7f037147f700 10 monclient(hunting): picked mon.noname-c con 0x7f036c09cf30 addr [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] 2023-09-30T01:33:04.348 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.338+0000 7f037147f700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f036c04bb20 con 0x7f036c09d7c0 2023-09-30T01:33:04.348 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.338+0000 7f037147f700 10 monclient(hunting): start opening mon connection 2023-09-30T01:33:04.348 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.338+0000 7f037147f700 1 -- --> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] -- mon_getmap magic: 0 v1 -- 0x7f036c09c610 con 0x7f036c09cf30 2023-09-30T01:33:04.348 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.338+0000 7f037147f700 10 monclient(hunting): start opening mon connection 2023-09-30T01:33:04.348 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.338+0000 7f037147f700 1 -- --> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] -- mon_getmap magic: 0 v1 -- 0x7f036c09dbb0 con 0x7f036c09be60 2023-09-30T01:33:04.348 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.338+0000 7f037147f700 10 monclient(hunting): _renew_subs 2023-09-30T01:33:04.348 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.338+0000 7f037147f700 10 monclient(hunting): authenticate will time out at 856.377138s 2023-09-30T01:33:04.348 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.338+0000 7f0370c7e700 1 --2- >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f036c09cf30 0x7f036c137e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:33:04.349 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.338+0000 7f0370c7e700 1 --2- >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f036c09cf30 0x7f036c137e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3302/0 says I am v2:172.21.15.88:50398/0 (socket says 172.21.15.88:50398) 2023-09-30T01:33:04.349 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.338+0000 7f0370c7e700 1 -- 172.21.15.88:0/1081333788 learned_addr learned my addr 172.21.15.88:0/1081333788 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-09-30T01:33:04.349 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.338+0000 7f0370c7e700 10 monclient(hunting): get_auth_request con 0x7f036c09cf30 auth_method 0 2023-09-30T01:33:04.349 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.338+0000 7f0370c7e700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:33:04.349 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.338+0000 7f0370c7e700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:33:04.349 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.338+0000 7f0370c7e700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:33:04.349 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.338+0000 7f0370c7e700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:33:04.349 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.338+0000 7f0370c7e700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:33:04.349 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.338+0000 7f0370c7e700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:33:04.349 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.338+0000 7f0370c7e700 10 monclient(hunting): handle_auth_done global_id 4261 payload 274 2023-09-30T01:33:04.350 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.338+0000 7f0370c7e700 1 -- 172.21.15.88:0/1081333788 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7f036c09be60 msgr2=0x7f036c12f800 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-09-30T01:33:04.350 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.338+0000 7f036bfff700 1 --2- 172.21.15.88:0/1081333788 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7f036c09be60 0x7f036c12f800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:33:04.350 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.338+0000 7f0370c7e700 1 --2- 172.21.15.88:0/1081333788 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7f036c09be60 0x7f036c12f800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:04.350 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.338+0000 7f0370c7e700 1 -- 172.21.15.88:0/1081333788 >> v1:172.21.15.107:6789/0 conn(0x7f036c09d7c0 legacy=0x7f036c12fd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:33:04.350 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.338+0000 7f0370c7e700 10 monclient: _finish_hunting 0 2023-09-30T01:33:04.350 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.338+0000 7f0370c7e700 1 monclient: found mon.noname-c 2023-09-30T01:33:04.350 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.338+0000 7f0370c7e700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:33:04.350 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.338+0000 7f0370c7e700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.88:3302/0 2023-09-30T01:33:04.350 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.338+0000 7f0370c7e700 1 -- 172.21.15.88:0/1081333788 --> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f036c09b1b0 con 0x7f036c09cf30 2023-09-30T01:33:04.351 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.338+0000 7fca12e64700 10 monclient: _reopen_session rank -1 2023-09-30T01:33:04.351 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.338+0000 7fca12e64700 10 monclient: _add_conns ranks=[4,3,0,9,7,8,5,6,2,10,1] 2023-09-30T01:33:04.351 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.338+0000 7fca12e64700 1 --2- >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7fca0c0a3270 0x7fca0c0a0160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:33:04.351 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.338+0000 7fca12e64700 10 monclient(hunting): picked mon.noname-c con 0x7fca0c0a3270 addr [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] 2023-09-30T01:33:04.351 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.338+0000 7fca12e64700 1 --2- >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7fca0c0a4b00 0x7fca0c0a06a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:33:04.351 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.338+0000 7fca12e64700 10 monclient(hunting): picked mon.noname-g con 0x7fca0c0a4b00 addr [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] 2023-09-30T01:33:04.352 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.338+0000 7f0370c7e700 1 --2- 172.21.15.88:0/1081333788 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f036c09cf30 0x7f036c137e60 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f0364009f50 tx=0x7f0364009840 comp rx=0 tx=0).ready entity=mon.4 client_cookie=41460101b0c3a76 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:33:04.352 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.338+0000 7f036bfff700 10 monclient: get_auth_request con 0x7f036c09be60 auth_method 0 2023-09-30T01:33:04.352 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.338+0000 7f036bfff700 1 --2- 172.21.15.88:0/1081333788 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7f036c09be60 0x7f036c12f800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-09-30T01:33:04.352 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.338+0000 7f036affd700 1 -- 172.21.15.88:0/1081333788 <== mon.4 v2:172.21.15.88:3302/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f036400e070 con 0x7f036c09cf30 2023-09-30T01:33:04.352 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.338+0000 7f036affd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:33:04.353 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.338+0000 7fca11401700 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7fca0c1333c0 0x7fca0c1337b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:54094/0 (socket says 172.21.15.88:54094) 2023-09-30T01:33:04.353 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.338+0000 7fca11401700 1 -- 172.21.15.88:0/2943717464 learned_addr learned my addr 172.21.15.88:0/2943717464 (peer_addr_for_me v1:172.21.15.88:0/0) 2023-09-30T01:33:04.353 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.338+0000 7f036affd700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T01:33:04.354 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.338+0000 7fca12e64700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fca0c1333c0 addr v1:172.21.15.88:6789/0 2023-09-30T01:33:04.354 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.338+0000 7fca12e64700 1 -- 172.21.15.88:0/2943717464 --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca0c05ecb0 con 0x7fca0c1333c0 2023-09-30T01:33:04.354 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.338+0000 7fca12e64700 10 monclient(hunting): start opening mon connection 2023-09-30T01:33:04.354 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.338+0000 7fca12e64700 1 -- 172.21.15.88:0/2943717464 --> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca0c0a4ed0 con 0x7fca0c0a4b00 2023-09-30T01:33:04.354 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.338+0000 7fca12e64700 10 monclient(hunting): start opening mon connection 2023-09-30T01:33:04.354 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.338+0000 7fca12e64700 1 -- 172.21.15.88:0/2943717464 --> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] -- mon_getmap magic: 0 v1 -- 0x7fca0c0a6330 con 0x7fca0c0a3270 2023-09-30T01:33:04.354 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.338+0000 7fca12e64700 10 monclient(hunting): _renew_subs 2023-09-30T01:33:04.354 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.338+0000 7fca12e64700 10 monclient(hunting): authenticate will time out at 856.379634s 2023-09-30T01:33:04.355 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.338+0000 7f036affd700 10 monclient: dump: 2023-09-30T01:33:04.355 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:33:04.355 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:33:04.355 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:04.355 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:04.355 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:33:04.355 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:33:04.355 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:33:04.355 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:33:04.355 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:33:04.356 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:33:04.356 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:33:04.356 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:33:04.356 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:33:04.356 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:33:04.356 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:33:04.356 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:33:04.356 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.338+0000 7f036affd700 10 monclient: _finish_auth 0 2023-09-30T01:33:04.356 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.338+0000 7f036affd700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:33:04.356 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.338+0000 7f036affd700 1 -- 172.21.15.88:0/1081333788 <== mon.4 v2:172.21.15.88:3302/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0364007400 con 0x7f036c09cf30 2023-09-30T01:33:04.356 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.338+0000 7f037147f700 5 monclient: authenticate success, global_id 4261 2023-09-30T01:33:04.357 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.338+0000 7f037147f700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:33:04.357 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.338+0000 7f036affd700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:33:04.357 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.338+0000 7f036affd700 1 -- 172.21.15.88:0/1081333788 <== mon.4 v2:172.21.15.88:3302/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f03640092e0 con 0x7f036c09cf30 2023-09-30T01:33:04.357 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.338+0000 7f036affd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:33:04.357 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.338+0000 7f036affd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T01:33:04.357 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.338+0000 7f036affd700 10 monclient: dump: 2023-09-30T01:33:04.357 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:33:04.357 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:33:04.357 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:04.357 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:04.357 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:33:04.358 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:33:04.358 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:33:04.358 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:33:04.358 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:33:04.358 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:33:04.358 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:33:04.358 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:33:04.358 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:33:04.358 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:33:04.358 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:33:04.358 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:33:04.359 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.338+0000 7f037147f700 10 monclient: get_monmap_and_config success 2023-09-30T01:33:04.359 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.338+0000 7f037147f700 4 set_mon_vals no callback set 2023-09-30T01:33:04.359 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.342+0000 7fca10c00700 1 --2- 172.21.15.88:0/2943717464 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7fca0c0a3270 0x7fca0c0a0160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:33:04.359 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.342+0000 7fca10c00700 10 monclient(hunting): get_auth_request con 0x7fca0c0a3270 auth_method 0 2023-09-30T01:33:04.359 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.342+0000 7fca10c00700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:33:04.360 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.342+0000 7fca10c00700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:33:04.360 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.342+0000 7fca10c00700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:33:04.360 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.342+0000 7fca0bfff700 1 --2- 172.21.15.88:0/2943717464 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7fca0c0a4b00 0x7fca0c0a06a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:33:04.360 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.342+0000 7fca0bfff700 10 monclient(hunting): get_auth_request con 0x7fca0c0a4b00 auth_method 0 2023-09-30T01:33:04.360 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.342+0000 7fca0bfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:33:04.360 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.342+0000 7fca0bfff700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:33:04.360 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.342+0000 7fca0bfff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:33:04.360 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.342+0000 7fca10c00700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:33:04.360 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.342+0000 7fca10c00700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:33:04.360 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.342+0000 7fca10c00700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:33:04.361 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.342+0000 7f037147f700 10 monclient: shutdown 2023-09-30T01:33:04.361 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.342+0000 7f037147f700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:33:04.361 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.342+0000 7f037147f700 1 -- 172.21.15.88:0/1081333788 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f036c09cf30 msgr2=0x7f036c137e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:33:04.361 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.342+0000 7f037147f700 1 --2- 172.21.15.88:0/1081333788 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f036c09cf30 0x7f036c137e60 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f0364009f50 tx=0x7f0364009840 comp rx=0 tx=0).stop 2023-09-30T01:33:04.361 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.342+0000 7fca0bfff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:33:04.362 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.342+0000 7fca0bfff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:33:04.362 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.342+0000 7fca0bfff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:33:04.362 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.342+0000 7fca10c00700 10 monclient(hunting): handle_auth_done global_id 4270 payload 274 2023-09-30T01:33:04.362 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.342+0000 7fca10c00700 1 -- 172.21.15.88:0/2943717464 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7fca0c0a4b00 msgr2=0x7fca0c0a06a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:33:04.362 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.342+0000 7fca10c00700 1 --2- 172.21.15.88:0/2943717464 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7fca0c0a4b00 0x7fca0c0a06a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:04.362 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.342+0000 7fca10c00700 1 -- 172.21.15.88:0/2943717464 >> v1:172.21.15.88:6789/0 conn(0x7fca0c1333c0 legacy=0x7fca0c1337b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:33:04.362 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.342+0000 7fca10c00700 10 monclient: _finish_hunting 0 2023-09-30T01:33:04.362 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.342+0000 7fca10c00700 1 monclient: found mon.noname-c 2023-09-30T01:33:04.362 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.342+0000 7fca10c00700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:33:04.362 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.342+0000 7fca10c00700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.88:3302/0 2023-09-30T01:33:04.363 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.342+0000 7fca10c00700 1 -- 172.21.15.88:0/2943717464 --> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca0c098610 con 0x7fca0c0a3270 2023-09-30T01:33:04.363 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.342+0000 7fca10c00700 1 --2- 172.21.15.88:0/2943717464 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7fca0c0a3270 0x7fca0c0a0160 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fca00009910 tx=0x7fca000091a0 comp rx=0 tx=0).ready entity=mon.4 client_cookie=a576a0cb889595f server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:33:04.363 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.342+0000 7f037147f700 1 -- 172.21.15.88:0/1081333788 shutdown_connections 2023-09-30T01:33:04.363 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.342+0000 7f037147f700 1 --2- 172.21.15.88:0/1081333788 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f036c09cf30 0x7f036c137e60 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:04.363 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.342+0000 7f037147f700 1 --2- 172.21.15.88:0/1081333788 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7f036c09be60 0x7f036c12f800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:04.363 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.342+0000 7f037147f700 1 -- 172.21.15.88:0/1081333788 >> 172.21.15.88:0/1081333788 conn(0x7f036c0a2540 msgr2=0x7f036c0a2910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:33:04.364 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.342+0000 7f037147f700 1 -- 172.21.15.88:0/1081333788 shutdown_connections 2023-09-30T01:33:04.364 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.342+0000 7f037147f700 1 -- 172.21.15.88:0/1081333788 wait complete. 2023-09-30T01:33:04.365 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.342+0000 7fca0b7fe700 1 -- 172.21.15.88:0/2943717464 <== mon.4 v2:172.21.15.88:3302/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7fca0000bb00 con 0x7fca0c0a3270 2023-09-30T01:33:04.365 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.342+0000 7fca0b7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:33:04.365 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.342+0000 7fca0b7fe700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T01:33:04.365 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.342+0000 7fca0b7fe700 10 monclient: dump: 2023-09-30T01:33:04.365 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:33:04.365 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:33:04.365 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:04.366 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:04.366 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:33:04.366 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:33:04.366 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:33:04.366 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:33:04.366 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:33:04.366 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:33:04.366 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:33:04.366 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:33:04.366 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:33:04.366 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:33:04.367 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:33:04.367 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:33:04.367 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.342+0000 7fca0b7fe700 10 monclient: _finish_auth 0 2023-09-30T01:33:04.367 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.342+0000 7fca0b7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:33:04.367 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.342+0000 7fca0b7fe700 1 -- 172.21.15.88:0/2943717464 <== mon.4 v2:172.21.15.88:3302/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca0000bc80 con 0x7fca0c0a3270 2023-09-30T01:33:04.367 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.342+0000 7fca12e64700 5 monclient: authenticate success, global_id 4270 2023-09-30T01:33:04.367 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.342+0000 7fca12e64700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:33:04.367 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.342+0000 7fca0b7fe700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:33:04.367 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.342+0000 7fca12e64700 10 monclient: get_monmap_and_config success 2023-09-30T01:33:04.368 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.342+0000 7fca12e64700 4 set_mon_vals no callback set 2023-09-30T01:33:04.368 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.342+0000 7fca0b7fe700 1 -- 172.21.15.88:0/2943717464 <== mon.4 v2:172.21.15.88:3302/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7fca0000bdf0 con 0x7fca0c0a3270 2023-09-30T01:33:04.368 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.342+0000 7fca0b7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:33:04.368 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.342+0000 7fca0b7fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T01:33:04.368 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.342+0000 7fca0b7fe700 10 monclient: dump: 2023-09-30T01:33:04.368 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:33:04.368 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:33:04.368 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:04.368 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:04.368 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:33:04.368 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:33:04.368 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:33:04.369 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:33:04.369 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:33:04.369 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:33:04.369 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:33:04.369 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:33:04.369 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:33:04.369 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:33:04.369 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:33:04.369 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:33:04.370 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.342+0000 7f037147f700 10 monclient: build_initial_monmap 2023-09-30T01:33:04.370 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.342+0000 7f037147f700 10 monclient: monmap: 2023-09-30T01:33:04.370 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2023-09-30T01:33:04.370 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:33:04.370 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 0.000000 2023-09-30T01:33:04.370 INFO:teuthology.orchestra.run.smithi088.stderr:created 0.000000 2023-09-30T01:33:04.370 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2023-09-30T01:33:04.370 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 1 2023-09-30T01:33:04.370 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.noname-a 2023-09-30T01:33:04.371 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.noname-f 2023-09-30T01:33:04.371 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.noname-b 2023-09-30T01:33:04.371 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.noname-g 2023-09-30T01:33:04.371 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.noname-c 2023-09-30T01:33:04.371 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.noname-h 2023-09-30T01:33:04.371 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.noname-d 2023-09-30T01:33:04.371 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.noname-i 2023-09-30T01:33:04.371 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.noname-e 2023-09-30T01:33:04.371 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:33:04.371 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.342+0000 7fca12e64700 10 monclient: shutdown 2023-09-30T01:33:04.372 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.342+0000 7fca12e64700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:33:04.372 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.342+0000 7fca12e64700 1 -- 172.21.15.88:0/2943717464 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7fca0c0a3270 msgr2=0x7fca0c0a0160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:33:04.372 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.342+0000 7fca12e64700 1 --2- 172.21.15.88:0/2943717464 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7fca0c0a3270 0x7fca0c0a0160 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fca00009910 tx=0x7fca000091a0 comp rx=0 tx=0).stop 2023-09-30T01:33:04.372 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.342+0000 7f037147f700 1 Processor -- start 2023-09-30T01:33:04.373 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.342+0000 7fca12e64700 1 -- 172.21.15.88:0/2943717464 shutdown_connections 2023-09-30T01:33:04.373 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.342+0000 7fca12e64700 1 --2- 172.21.15.88:0/2943717464 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7fca0c0a4b00 0x7fca0c0a06a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:04.373 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.342+0000 7fca12e64700 1 --2- 172.21.15.88:0/2943717464 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7fca0c0a3270 0x7fca0c0a0160 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:04.373 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.342+0000 7fca12e64700 1 -- 172.21.15.88:0/2943717464 >> 172.21.15.88:0/2943717464 conn(0x7fca0c0a67f0 msgr2=0x7fca0c0a6bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:33:04.374 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.342+0000 7f037147f700 1 -- start start 2023-09-30T01:33:04.374 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.342+0000 7f037147f700 10 monclient: init 2023-09-30T01:33:04.374 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.346+0000 7f037147f700 10 monclient: _reopen_session rank -1 2023-09-30T01:33:04.375 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.346+0000 7f037147f700 10 monclient: _add_conns ranks=[7,3,0,2,1,5,8,6,4] 2023-09-30T01:33:04.375 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.346+0000 7f037147f700 1 --2- >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7f036c09cf30 0x7f036c08cb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:33:04.375 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.346+0000 7f037147f700 10 monclient(hunting): picked mon.noname-i con 0x7f036c09cf30 addr [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] 2023-09-30T01:33:04.375 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.346+0000 7f037147f700 1 --2- >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7f036c09d7c0 0x7f036c08d0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:33:04.375 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.346+0000 7f037147f700 10 monclient(hunting): picked mon.noname-g con 0x7f036c09d7c0 addr [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] 2023-09-30T01:33:04.375 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.346+0000 7f037147f700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f036c08d5e0 0x7f036c1467d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:33:04.375 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.346+0000 7f037147f700 10 monclient(hunting): picked mon.noname-a con 0x7f036c08d5e0 addr [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] 2023-09-30T01:33:04.375 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.346+0000 7f037147f700 10 monclient(hunting): start opening mon connection 2023-09-30T01:33:04.375 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.346+0000 7f037147f700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f036c08ea60 con 0x7f036c08d5e0 2023-09-30T01:33:04.375 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.346+0000 7f037147f700 10 monclient(hunting): start opening mon connection 2023-09-30T01:33:04.376 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.346+0000 7f037147f700 1 -- --> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f036c090280 con 0x7f036c09d7c0 2023-09-30T01:33:04.376 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.346+0000 7f037147f700 10 monclient(hunting): start opening mon connection 2023-09-30T01:33:04.376 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.346+0000 7f037147f700 1 -- --> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] -- mon_getmap magic: 0 v1 -- 0x7f036c09a560 con 0x7f036c09cf30 2023-09-30T01:33:04.376 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.346+0000 7f037147f700 10 monclient(hunting): _renew_subs 2023-09-30T01:33:04.376 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.346+0000 7f037147f700 10 monclient(hunting): authenticate will time out at 856.384618s 2023-09-30T01:33:04.376 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.346+0000 7fca12e64700 1 -- 172.21.15.88:0/2943717464 shutdown_connections 2023-09-30T01:33:04.376 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.346+0000 7fca12e64700 1 -- 172.21.15.88:0/2943717464 wait complete. 2023-09-30T01:33:04.377 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.346+0000 7f036b7fe700 1 --2- >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7f036c09d7c0 0x7f036c08d0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:33:04.377 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.346+0000 7f0370c7e700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f036c08d5e0 0x7f036c1467d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:33:04.377 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.346+0000 7f036b7fe700 1 --2- >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7f036c09d7c0 0x7f036c08d0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3301/0 says I am v2:172.21.15.88:48286/0 (socket says 172.21.15.88:48286) 2023-09-30T01:33:04.377 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.346+0000 7f036b7fe700 1 -- 172.21.15.88:0/3802989775 learned_addr learned my addr 172.21.15.88:0/3802989775 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-09-30T01:33:04.377 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.346+0000 7f036b7fe700 10 monclient(hunting): get_auth_request con 0x7f036c09d7c0 auth_method 0 2023-09-30T01:33:04.377 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.346+0000 7f036b7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:33:04.377 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.346+0000 7f036b7fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:33:04.377 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.346+0000 7f036b7fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:33:04.378 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.346+0000 7fca12e64700 10 monclient: build_initial_monmap 2023-09-30T01:33:04.378 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.346+0000 7f0370c7e700 10 monclient(hunting): get_auth_request con 0x7f036c08d5e0 auth_method 0 2023-09-30T01:33:04.378 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.346+0000 7f0370c7e700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:33:04.378 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.346+0000 7f0370c7e700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:33:04.379 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.346+0000 7f0370c7e700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:33:04.379 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.346+0000 7f036b7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:33:04.379 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.346+0000 7f036b7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:33:04.379 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.346+0000 7f036b7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:33:04.379 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.346+0000 7f0370c7e700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:33:04.379 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.346+0000 7f0370c7e700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:33:04.379 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.346+0000 7f0370c7e700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:33:04.380 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.346+0000 7f036bfff700 1 --2- 172.21.15.88:0/3802989775 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7f036c09cf30 0x7f036c08cb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:33:04.380 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.346+0000 7f036bfff700 10 monclient(hunting): get_auth_request con 0x7f036c09cf30 auth_method 0 2023-09-30T01:33:04.380 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.346+0000 7f036bfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:33:04.380 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.346+0000 7f036bfff700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:33:04.380 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.346+0000 7f036bfff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:33:04.380 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.346+0000 7f0370c7e700 10 monclient(hunting): handle_auth_done global_id 4284 payload 824 2023-09-30T01:33:04.380 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.346+0000 7f0370c7e700 1 -- 172.21.15.88:0/3802989775 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7f036c09cf30 msgr2=0x7f036c08cb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:33:04.380 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.346+0000 7f0370c7e700 1 --2- 172.21.15.88:0/3802989775 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7f036c09cf30 0x7f036c08cb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:04.380 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.346+0000 7f0370c7e700 1 -- 172.21.15.88:0/3802989775 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7f036c09d7c0 msgr2=0x7f036c08d0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:33:04.380 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.346+0000 7f0370c7e700 1 --2- 172.21.15.88:0/3802989775 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7f036c09d7c0 0x7f036c08d0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:04.380 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.346+0000 7f0370c7e700 10 monclient: _finish_hunting 0 2023-09-30T01:33:04.381 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.346+0000 7f0370c7e700 1 monclient: found mon.noname-a 2023-09-30T01:33:04.381 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.346+0000 7f0370c7e700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:33:04.381 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.346+0000 7f0370c7e700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.88:3300/0 2023-09-30T01:33:04.381 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.346+0000 7f0370c7e700 1 -- 172.21.15.88:0/3802989775 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0364009ae0 con 0x7f036c08d5e0 2023-09-30T01:33:04.381 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.346+0000 7f0370c7e700 1 --2- 172.21.15.88:0/3802989775 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f036c08d5e0 0x7f036c1467d0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f0364014000 tx=0x7f0364017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:33:04.383 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.346+0000 7fca12e64700 10 monclient: monmap: 2023-09-30T01:33:04.383 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2023-09-30T01:33:04.383 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:33:04.383 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 0.000000 2023-09-30T01:33:04.384 INFO:teuthology.orchestra.run.smithi088.stderr:created 0.000000 2023-09-30T01:33:04.384 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2023-09-30T01:33:04.384 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 1 2023-09-30T01:33:04.384 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.noname-a 2023-09-30T01:33:04.384 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.noname-f 2023-09-30T01:33:04.384 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.noname-b 2023-09-30T01:33:04.384 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.noname-g 2023-09-30T01:33:04.384 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.noname-c 2023-09-30T01:33:04.384 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.noname-h 2023-09-30T01:33:04.384 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.noname-d 2023-09-30T01:33:04.384 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.noname-i 2023-09-30T01:33:04.384 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.noname-e 2023-09-30T01:33:04.385 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:33:04.385 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.346+0000 7fca12e64700 1 Processor -- start 2023-09-30T01:33:04.385 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.346+0000 7fca12e64700 1 -- start start 2023-09-30T01:33:04.385 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.346+0000 7fca12e64700 10 monclient: init 2023-09-30T01:33:04.385 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.346+0000 7fca12e64700 10 monclient: _reopen_session rank -1 2023-09-30T01:33:04.385 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.346+0000 7fca12e64700 10 monclient: _add_conns ranks=[0,8,7,2,3,6,1,4,5] 2023-09-30T01:33:04.385 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.346+0000 7fca12e64700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fca0c0a3270 0x7fca0c142880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:33:04.385 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.346+0000 7fca12e64700 10 monclient(hunting): picked mon.noname-a con 0x7fca0c0a3270 addr [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] 2023-09-30T01:33:04.385 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.346+0000 7fca12e64700 1 --2- >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7fca0c0a4b00 0x7fca0c142dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:33:04.385 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.346+0000 7fca12e64700 10 monclient(hunting): picked mon.noname-e con 0x7fca0c0a4b00 addr [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] 2023-09-30T01:33:04.385 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.346+0000 7fca12e64700 1 --2- >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7fca0c1333c0 0x7fca0c143300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:33:04.385 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.346+0000 7fca12e64700 10 monclient(hunting): picked mon.noname-i con 0x7fca0c1333c0 addr [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] 2023-09-30T01:33:04.386 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.346+0000 7fca12e64700 10 monclient(hunting): start opening mon connection 2023-09-30T01:33:04.386 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.346+0000 7fca12e64700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca0c0a0e80 con 0x7fca0c0a3270 2023-09-30T01:33:04.386 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.346+0000 7fca12e64700 10 monclient(hunting): start opening mon connection 2023-09-30T01:33:04.386 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.346+0000 7fca12e64700 1 -- --> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] -- mon_getmap magic: 0 v1 -- 0x7fca0c097c70 con 0x7fca0c1333c0 2023-09-30T01:33:04.386 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.346+0000 7fca12e64700 10 monclient(hunting): start opening mon connection 2023-09-30T01:33:04.386 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.346+0000 7fca12e64700 1 -- --> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] -- mon_getmap magic: 0 v1 -- 0x7fca0c120180 con 0x7fca0c0a4b00 2023-09-30T01:33:04.386 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.346+0000 7fca12e64700 10 monclient(hunting): _renew_subs 2023-09-30T01:33:04.386 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.346+0000 7fca12e64700 10 monclient(hunting): authenticate will time out at 856.386848s 2023-09-30T01:33:04.386 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.346+0000 7fca10c00700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fca0c0a3270 0x7fca0c142880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:33:04.386 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.346+0000 7fca10c00700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fca0c0a3270 0x7fca0c142880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:57796/0 (socket says 172.21.15.88:57796) 2023-09-30T01:33:04.386 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.346+0000 7fca10c00700 1 -- 172.21.15.88:0/3275358350 learned_addr learned my addr 172.21.15.88:0/3275358350 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-09-30T01:33:04.387 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.346+0000 7fca10c00700 10 monclient(hunting): get_auth_request con 0x7fca0c0a3270 auth_method 0 2023-09-30T01:33:04.387 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.346+0000 7fca10c00700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:33:04.387 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.346+0000 7fca10c00700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:33:04.387 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.346+0000 7fca10c00700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:33:04.387 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.346+0000 7fca0bfff700 1 --2- 172.21.15.88:0/3275358350 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7fca0c0a4b00 0x7fca0c142dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:33:04.387 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.346+0000 7fca0bfff700 10 monclient(hunting): get_auth_request con 0x7fca0c0a4b00 auth_method 0 2023-09-30T01:33:04.387 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.346+0000 7fca0bfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:33:04.387 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.346+0000 7fca0bfff700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:33:04.387 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.346+0000 7fca0bfff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:33:04.387 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.346+0000 7fca10c00700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:33:04.387 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.346+0000 7fca10c00700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:33:04.387 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.346+0000 7fca11401700 1 --2- 172.21.15.88:0/3275358350 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7fca0c1333c0 0x7fca0c143300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:33:04.388 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.346+0000 7fca10c00700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:33:04.388 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.346+0000 7fca11401700 10 monclient(hunting): get_auth_request con 0x7fca0c1333c0 auth_method 0 2023-09-30T01:33:04.388 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.346+0000 7fca11401700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:33:04.388 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.346+0000 7fca11401700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:33:04.388 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.346+0000 7fca11401700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:33:04.388 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.346+0000 7fca0bfff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:33:04.388 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.346+0000 7fca0bfff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:33:04.388 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.346+0000 7fca0bfff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:33:04.388 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.346+0000 7fca11401700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:33:04.388 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.346+0000 7fca11401700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:33:04.388 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.346+0000 7fca11401700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:33:04.388 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.346+0000 7fca10c00700 10 monclient(hunting): handle_auth_done global_id 4293 payload 824 2023-09-30T01:33:04.389 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.346+0000 7fca10c00700 1 -- 172.21.15.88:0/3275358350 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7fca0c0a4b00 msgr2=0x7fca0c142dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:33:04.389 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.346+0000 7fca10c00700 1 --2- 172.21.15.88:0/3275358350 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7fca0c0a4b00 0x7fca0c142dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:04.389 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.346+0000 7fca10c00700 1 -- 172.21.15.88:0/3275358350 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7fca0c1333c0 msgr2=0x7fca0c143300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:33:04.389 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.346+0000 7fca10c00700 1 --2- 172.21.15.88:0/3275358350 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7fca0c1333c0 0x7fca0c143300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:04.389 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.346+0000 7fca10c00700 10 monclient: _finish_hunting 0 2023-09-30T01:33:04.389 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.346+0000 7fca10c00700 1 monclient: found mon.noname-a 2023-09-30T01:33:04.389 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.346+0000 7fca10c00700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:33:04.389 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.346+0000 7fca10c00700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.88:3300/0 2023-09-30T01:33:04.389 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.346+0000 7fca10c00700 1 -- 172.21.15.88:0/3275358350 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca00009420 con 0x7fca0c0a3270 2023-09-30T01:33:04.389 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.346+0000 7fca10c00700 1 --2- 172.21.15.88:0/3275358350 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fca0c0a3270 0x7fca0c142880 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fca00013000 tx=0x7fca00017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:33:04.393 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.354+0000 7f66d1c91700 10 monclient: get_monmap_and_config 2023-09-30T01:33:04.393 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.354+0000 7f66d1c91700 10 monclient: build_initial_monmap 2023-09-30T01:33:04.393 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.354+0000 7f66d1c91700 10 monclient: monmap: 2023-09-30T01:33:04.393 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2023-09-30T01:33:04.393 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:33:04.393 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:33:04.358433+0000 2023-09-30T01:33:04.393 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:33:04.358433+0000 2023-09-30T01:33:04.393 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2023-09-30T01:33:04.394 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:33:04.394 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a-legacy 2023-09-30T01:33:04.394 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.107:6789/0 mon.noname-f-legacy 2023-09-30T01:33:04.394 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.noname-b 2023-09-30T01:33:04.394 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.noname-g 2023-09-30T01:33:04.394 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.noname-c 2023-09-30T01:33:04.394 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.noname-h 2023-09-30T01:33:04.394 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.noname-d 2023-09-30T01:33:04.394 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.noname-i 2023-09-30T01:33:04.394 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.noname-e 2023-09-30T01:33:04.394 INFO:teuthology.orchestra.run.smithi088.stderr:9: v2:172.21.15.88:3300/0 mon.noname-a 2023-09-30T01:33:04.395 INFO:teuthology.orchestra.run.smithi088.stderr:10: v2:172.21.15.107:3300/0 mon.noname-f 2023-09-30T01:33:04.395 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:33:04.395 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.354+0000 7f66d1c91700 1 Processor -- start 2023-09-30T01:33:04.395 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.354+0000 7f66d1c91700 1 -- start start 2023-09-30T01:33:04.395 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.354+0000 7f66d1c91700 10 monclient: init 2023-09-30T01:33:04.395 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.358+0000 7f66d1c91700 10 monclient: _reopen_session rank -1 2023-09-30T01:33:04.395 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.358+0000 7f66d1c91700 10 monclient: _add_conns ranks=[7,6,1,3,8,2,5,10,4,0,9] 2023-09-30T01:33:04.395 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.358+0000 7f66d1c91700 1 --2- >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7f66cc0966a0 0x7f66cc096a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:33:04.395 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.358+0000 7f66d1c91700 10 monclient(hunting): picked mon.noname-i con 0x7f66cc0966a0 addr [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] 2023-09-30T01:33:04.395 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.358+0000 7f66d1c91700 1 --2- >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7f66cc097e40 0x7f66cc098210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:33:04.395 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.358+0000 7f66d1c91700 10 monclient(hunting): picked mon.noname-d con 0x7f66cc097e40 addr [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] 2023-09-30T01:33:04.395 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.358+0000 7f66d1c91700 10 monclient(hunting): picked mon.noname-f-legacy con 0x7f66cc09dd70 addr v1:172.21.15.107:6789/0 2023-09-30T01:33:04.396 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.358+0000 7f66d1c91700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66cc04bb20 con 0x7f66cc09dd70 2023-09-30T01:33:04.396 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.358+0000 7f66d1c91700 10 monclient(hunting): start opening mon connection 2023-09-30T01:33:04.396 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.358+0000 7f66d1c91700 1 -- --> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] -- mon_getmap magic: 0 v1 -- 0x7f66cc063c40 con 0x7f66cc097e40 2023-09-30T01:33:04.396 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.358+0000 7f66d1c91700 10 monclient(hunting): start opening mon connection 2023-09-30T01:33:04.396 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.358+0000 7f66d1c91700 1 -- --> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] -- mon_getmap magic: 0 v1 -- 0x7f66cc097460 con 0x7f66cc0966a0 2023-09-30T01:33:04.396 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.358+0000 7f66d1c91700 10 monclient(hunting): _renew_subs 2023-09-30T01:33:04.396 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.358+0000 7f66d1c91700 10 monclient(hunting): authenticate will time out at 856.396249s 2023-09-30T01:33:04.396 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.358+0000 7f66cbfff700 1 --2- >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7f66cc097e40 0x7f66cc098210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:33:04.396 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.358+0000 7f66cbfff700 1 --2- >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7f66cc097e40 0x7f66cc098210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3303/0 says I am v2:172.21.15.88:53548/0 (socket says 172.21.15.88:53548) 2023-09-30T01:33:04.396 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.358+0000 7f66cbfff700 1 -- 172.21.15.88:0/2739325911 learned_addr learned my addr 172.21.15.88:0/2739325911 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-09-30T01:33:04.396 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.358+0000 7f66cbfff700 10 monclient(hunting): get_auth_request con 0x7f66cc097e40 auth_method 0 2023-09-30T01:33:04.396 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.358+0000 7f66cbfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:33:04.397 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.358+0000 7f66cbfff700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:33:04.397 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.358+0000 7f66cbfff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:33:04.397 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.358+0000 7f66cbfff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:33:04.397 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.358+0000 7f66cbfff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:33:04.397 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.358+0000 7f66cbfff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:33:04.397 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.358+0000 7f66d0c8f700 1 --2- 172.21.15.88:0/2739325911 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7f66cc0966a0 0x7f66cc096a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:33:04.397 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.358+0000 7f66d0c8f700 10 monclient(hunting): get_auth_request con 0x7f66cc0966a0 auth_method 0 2023-09-30T01:33:04.397 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.358+0000 7f66d0c8f700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:33:04.397 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.358+0000 7f66d0c8f700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:33:04.397 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.358+0000 7f66d0c8f700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:33:04.397 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.358+0000 7f66cbfff700 10 monclient(hunting): handle_auth_done global_id 4209 payload 274 2023-09-30T01:33:04.397 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.358+0000 7f66cbfff700 1 -- 172.21.15.88:0/2739325911 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7f66cc0966a0 msgr2=0x7f66cc096a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:33:04.398 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.358+0000 7f66cbfff700 1 --2- 172.21.15.88:0/2739325911 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7f66cc0966a0 0x7f66cc096a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:04.398 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.358+0000 7f66cbfff700 1 -- 172.21.15.88:0/2739325911 >> v1:172.21.15.107:6789/0 conn(0x7f66cc09dd70 legacy=0x7f66cc09e1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:33:04.398 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.358+0000 7f66cbfff700 10 monclient: _finish_hunting 0 2023-09-30T01:33:04.398 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.358+0000 7f66cbfff700 1 monclient: found mon.noname-d 2023-09-30T01:33:04.398 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.358+0000 7f66cbfff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:33:04.398 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.358+0000 7f66cbfff700 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.88:3303/0 2023-09-30T01:33:04.398 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.358+0000 7f66cbfff700 1 -- 172.21.15.88:0/2739325911 --> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66cc063950 con 0x7f66cc097e40 2023-09-30T01:33:04.398 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.358+0000 7f66cbfff700 1 --2- 172.21.15.88:0/2739325911 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7f66cc097e40 0x7f66cc098210 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f66c00099e0 tx=0x7f66c0009250 comp rx=0 tx=0).ready entity=mon.6 client_cookie=5ea1ed83375a5a2e server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:33:04.398 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.358+0000 7f66cb7fe700 1 -- 172.21.15.88:0/2739325911 <== mon.6 v2:172.21.15.88:3303/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f66c000bce0 con 0x7f66cc097e40 2023-09-30T01:33:04.398 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.358+0000 7f66cb7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:33:04.398 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.358+0000 7f66cb7fe700 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2023-09-30T01:33:04.398 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.358+0000 7f66cb7fe700 10 monclient: dump: 2023-09-30T01:33:04.399 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:33:04.399 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:33:04.399 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:04.399 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:04.399 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:33:04.399 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:33:04.399 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:33:04.399 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:33:04.399 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:33:04.399 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:33:04.399 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:33:04.399 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:33:04.400 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:33:04.400 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:33:04.400 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:33:04.400 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:33:04.400 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.358+0000 7f66cb7fe700 10 monclient: _finish_auth 0 2023-09-30T01:33:04.400 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.358+0000 7f66cb7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:33:04.400 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.358+0000 7f66cb7fe700 1 -- 172.21.15.88:0/2739325911 <== mon.6 v2:172.21.15.88:3303/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f66c000be60 con 0x7f66cc097e40 2023-09-30T01:33:04.400 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.358+0000 7f66d1c91700 5 monclient: authenticate success, global_id 4209 2023-09-30T01:33:04.400 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.358+0000 7f66d1c91700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:33:04.400 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.358+0000 7f66cb7fe700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:33:04.400 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.358+0000 7f66d1c91700 10 monclient: get_monmap_and_config success 2023-09-30T01:33:04.400 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.358+0000 7f66d1c91700 4 set_mon_vals no callback set 2023-09-30T01:33:04.401 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.358+0000 7f66cb7fe700 1 -- 172.21.15.88:0/2739325911 <== mon.6 v2:172.21.15.88:3303/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f66c00118a0 con 0x7f66cc097e40 2023-09-30T01:33:04.401 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.358+0000 7f66cb7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:33:04.401 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.358+0000 7f66cb7fe700 10 monclient: got monmap 1 from mon.d (according to old e1) 2023-09-30T01:33:04.401 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.358+0000 7f66cb7fe700 10 monclient: dump: 2023-09-30T01:33:04.401 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:33:04.401 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:33:04.401 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:04.401 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:04.401 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:33:04.401 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:33:04.401 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:33:04.402 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:33:04.402 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:33:04.402 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:33:04.402 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:33:04.402 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:33:04.402 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:33:04.402 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:33:04.402 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:33:04.402 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:33:04.402 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.358+0000 7f66d1c91700 10 monclient: shutdown 2023-09-30T01:33:04.402 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.358+0000 7f66d1c91700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:33:04.402 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.358+0000 7f66d1c91700 1 -- 172.21.15.88:0/2739325911 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7f66cc097e40 msgr2=0x7f66cc098210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:33:04.403 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.358+0000 7f66d1c91700 1 --2- 172.21.15.88:0/2739325911 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7f66cc097e40 0x7f66cc098210 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f66c00099e0 tx=0x7f66c0009250 comp rx=0 tx=0).stop 2023-09-30T01:33:04.403 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.362+0000 7f66d1c91700 1 -- 172.21.15.88:0/2739325911 shutdown_connections 2023-09-30T01:33:04.403 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.362+0000 7f66d1c91700 1 --2- 172.21.15.88:0/2739325911 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7f66cc097e40 0x7f66cc098210 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:04.403 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.362+0000 7f66d1c91700 1 --2- 172.21.15.88:0/2739325911 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7f66cc0966a0 0x7f66cc096a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:04.403 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.362+0000 7f66d1c91700 1 -- 172.21.15.88:0/2739325911 >> 172.21.15.88:0/2739325911 conn(0x7f66cc09fe70 msgr2=0x7f66cc0a0270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:33:04.403 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.362+0000 7f66d1c91700 1 -- 172.21.15.88:0/2739325911 shutdown_connections 2023-09-30T01:33:04.403 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.362+0000 7f66d1c91700 1 -- 172.21.15.88:0/2739325911 wait complete. 2023-09-30T01:33:04.403 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.362+0000 7f66d1c91700 10 monclient: build_initial_monmap 2023-09-30T01:33:04.403 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.362+0000 7f66d1c91700 10 monclient: monmap: 2023-09-30T01:33:04.403 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2023-09-30T01:33:04.403 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:33:04.403 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 0.000000 2023-09-30T01:33:04.404 INFO:teuthology.orchestra.run.smithi088.stderr:created 0.000000 2023-09-30T01:33:04.404 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2023-09-30T01:33:04.404 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 1 2023-09-30T01:33:04.404 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.noname-a 2023-09-30T01:33:04.404 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.noname-f 2023-09-30T01:33:04.404 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.noname-b 2023-09-30T01:33:04.404 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.noname-g 2023-09-30T01:33:04.404 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.noname-c 2023-09-30T01:33:04.404 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.noname-h 2023-09-30T01:33:04.404 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.noname-d 2023-09-30T01:33:04.404 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.noname-i 2023-09-30T01:33:04.404 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.noname-e 2023-09-30T01:33:04.405 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:33:04.405 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.362+0000 7f66d1c91700 1 Processor -- start 2023-09-30T01:33:04.405 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.362+0000 7f66d1c91700 1 -- start start 2023-09-30T01:33:04.405 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.362+0000 7f66d1c91700 10 monclient: init 2023-09-30T01:33:04.405 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.362+0000 7f2b67fff700 10 monclient: get_monmap_and_config 2023-09-30T01:33:04.405 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.362+0000 7f2b67fff700 10 monclient: build_initial_monmap 2023-09-30T01:33:04.406 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.362+0000 7f2b67fff700 10 monclient: monmap: 2023-09-30T01:33:04.406 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2023-09-30T01:33:04.406 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:33:04.406 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:33:04.364515+0000 2023-09-30T01:33:04.406 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:33:04.364515+0000 2023-09-30T01:33:04.406 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2023-09-30T01:33:04.406 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:33:04.406 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a-legacy 2023-09-30T01:33:04.406 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.107:6789/0 mon.noname-f-legacy 2023-09-30T01:33:04.406 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.noname-b 2023-09-30T01:33:04.406 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.noname-g 2023-09-30T01:33:04.407 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.noname-c 2023-09-30T01:33:04.407 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.noname-h 2023-09-30T01:33:04.407 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.noname-d 2023-09-30T01:33:04.407 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.noname-i 2023-09-30T01:33:04.407 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.noname-e 2023-09-30T01:33:04.407 INFO:teuthology.orchestra.run.smithi088.stderr:9: v2:172.21.15.88:3300/0 mon.noname-a 2023-09-30T01:33:04.407 INFO:teuthology.orchestra.run.smithi088.stderr:10: v2:172.21.15.107:3300/0 mon.noname-f 2023-09-30T01:33:04.407 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:33:04.409 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.362+0000 7f66d1c91700 10 monclient: _reopen_session rank -1 2023-09-30T01:33:04.410 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.362+0000 7f66d1c91700 10 monclient: _add_conns ranks=[3,0,1,6,2,8,4,5,7] 2023-09-30T01:33:04.410 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.362+0000 7f66d1c91700 1 --2- >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7f66cc0966a0 0x7f66cc1428c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:33:04.410 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.362+0000 7f66d1c91700 10 monclient(hunting): picked mon.noname-g con 0x7f66cc0966a0 addr [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] 2023-09-30T01:33:04.410 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.362+0000 7f66d1c91700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f66cc097e40 0x7f66cc142e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:33:04.410 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.362+0000 7f66d1c91700 10 monclient(hunting): picked mon.noname-a con 0x7f66cc097e40 addr [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] 2023-09-30T01:33:04.410 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.362+0000 7f66d1c91700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f66cc09dd70 0x7f66cc08ddd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:33:04.410 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.362+0000 7f66d1c91700 10 monclient(hunting): picked mon.noname-f con 0x7f66cc09dd70 addr [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] 2023-09-30T01:33:04.410 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.362+0000 7f66d1c91700 10 monclient(hunting): start opening mon connection 2023-09-30T01:33:04.410 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.362+0000 7f66d1c91700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66cc099830 con 0x7f66cc097e40 2023-09-30T01:33:04.410 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.362+0000 7f66d1c91700 10 monclient(hunting): start opening mon connection 2023-09-30T01:33:04.410 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.362+0000 7f66d1c91700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66cc09f740 con 0x7f66cc09dd70 2023-09-30T01:33:04.411 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.362+0000 7f66d1c91700 10 monclient(hunting): start opening mon connection 2023-09-30T01:33:04.411 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.362+0000 7f66d1c91700 1 -- --> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66cc08fef0 con 0x7f66cc0966a0 2023-09-30T01:33:04.411 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.362+0000 7f66d1c91700 10 monclient(hunting): _renew_subs 2023-09-30T01:33:04.411 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.362+0000 7f66d1c91700 10 monclient(hunting): authenticate will time out at 856.402354s 2023-09-30T01:33:04.411 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.362+0000 7f66cbfff700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f66cc097e40 0x7f66cc142e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:33:04.411 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.362+0000 7f66cbfff700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f66cc097e40 0x7f66cc142e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:57806/0 (socket says 172.21.15.88:57806) 2023-09-30T01:33:04.411 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.362+0000 7f66cbfff700 1 -- 172.21.15.88:0/669031240 learned_addr learned my addr 172.21.15.88:0/669031240 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-09-30T01:33:04.411 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.362+0000 7f66cbfff700 10 monclient(hunting): get_auth_request con 0x7f66cc097e40 auth_method 0 2023-09-30T01:33:04.411 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.362+0000 7f66cbfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:33:04.411 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.362+0000 7f66cbfff700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:33:04.411 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.362+0000 7f66cbfff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:33:04.411 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.362+0000 7f66d0c8f700 1 --2- 172.21.15.88:0/669031240 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7f66cc0966a0 0x7f66cc1428c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:33:04.412 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.362+0000 7f66cbfff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:33:04.412 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.362+0000 7f66cbfff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:33:04.412 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.362+0000 7f66cbfff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:33:04.412 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.362+0000 7f66d0c8f700 10 monclient(hunting): get_auth_request con 0x7f66cc0966a0 auth_method 0 2023-09-30T01:33:04.412 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.362+0000 7f66d0c8f700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:33:04.412 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.362+0000 7f66d0c8f700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:33:04.412 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.362+0000 7f66d0c8f700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:33:04.412 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.362+0000 7f66d1490700 1 --2- 172.21.15.88:0/669031240 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f66cc09dd70 0x7f66cc08ddd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:33:04.412 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.362+0000 7f66d1490700 10 monclient(hunting): get_auth_request con 0x7f66cc09dd70 auth_method 0 2023-09-30T01:33:04.412 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.362+0000 7f66d1490700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:33:04.413 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.362+0000 7f66d1490700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:33:04.413 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.362+0000 7f66d1490700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:33:04.413 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.362+0000 7f66d0c8f700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:33:04.413 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.362+0000 7f66d0c8f700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:33:04.413 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.362+0000 7f66d0c8f700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:33:04.413 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.362+0000 7f66d1490700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:33:04.413 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.362+0000 7f66d1490700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:33:04.413 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.362+0000 7f66d1490700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:33:04.413 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.362+0000 7f66cbfff700 10 monclient(hunting): handle_auth_done global_id 4302 payload 824 2023-09-30T01:33:04.413 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.362+0000 7f66cbfff700 1 -- 172.21.15.88:0/669031240 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7f66cc0966a0 msgr2=0x7f66cc1428c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:33:04.413 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.362+0000 7f66cbfff700 1 --2- 172.21.15.88:0/669031240 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7f66cc0966a0 0x7f66cc1428c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:04.413 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.362+0000 7f66cbfff700 1 -- 172.21.15.88:0/669031240 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f66cc09dd70 msgr2=0x7f66cc08ddd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:33:04.413 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.362+0000 7f66cbfff700 1 --2- 172.21.15.88:0/669031240 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f66cc09dd70 0x7f66cc08ddd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:04.414 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.362+0000 7f66cbfff700 10 monclient: _finish_hunting 0 2023-09-30T01:33:04.414 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.362+0000 7f66cbfff700 1 monclient: found mon.noname-a 2023-09-30T01:33:04.414 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.362+0000 7f66cbfff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:33:04.414 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.362+0000 7f66cbfff700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.88:3300/0 2023-09-30T01:33:04.414 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.362+0000 7f66cbfff700 1 -- 172.21.15.88:0/669031240 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66c00094f0 con 0x7f66cc097e40 2023-09-30T01:33:04.414 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.362+0000 7f66cbfff700 1 --2- 172.21.15.88:0/669031240 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f66cc097e40 0x7f66cc142e00 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f66cc0987b0 tx=0x7f66c0017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:33:04.418 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.366+0000 7f2b67fff700 1 Processor -- start 2023-09-30T01:33:04.419 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.366+0000 7f2b67fff700 1 -- start start 2023-09-30T01:33:04.419 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.366+0000 7f2b67fff700 10 monclient: init 2023-09-30T01:33:04.419 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.366+0000 7f2b67fff700 10 monclient: _reopen_session rank -1 2023-09-30T01:33:04.419 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.366+0000 7f2b67fff700 10 monclient: _add_conns ranks=[2,6,8,9,7,1,3,4,0,10,5] 2023-09-30T01:33:04.419 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.366+0000 7f2b67fff700 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2b6809cf30 0x7f2b68096e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:33:04.419 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.366+0000 7f2b67fff700 10 monclient(hunting): picked mon.noname-b con 0x7f2b6809cf30 addr [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] 2023-09-30T01:33:04.419 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.366+0000 7f2b67fff700 1 --2- >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7f2b6809d300 0x7f2b6809d6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:33:04.419 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.366+0000 7f2b67fff700 10 monclient(hunting): picked mon.noname-d con 0x7f2b6809d300 addr [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] 2023-09-30T01:33:04.419 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.366+0000 7f2b67fff700 1 --2- >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f2b681356d0 0x7f2b68137ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:33:04.419 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.366+0000 7f2b67fff700 10 monclient(hunting): picked mon.noname-e con 0x7f2b681356d0 addr [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] 2023-09-30T01:33:04.419 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.366+0000 7f2b67fff700 10 monclient(hunting): start opening mon connection 2023-09-30T01:33:04.420 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.366+0000 7f2b67fff700 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b68096b00 con 0x7f2b6809cf30 2023-09-30T01:33:04.420 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.366+0000 7f2b67fff700 10 monclient(hunting): start opening mon connection 2023-09-30T01:33:04.420 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.366+0000 7f2b67fff700 1 -- --> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] -- mon_getmap magic: 0 v1 -- 0x7f2b6809c530 con 0x7f2b6809d300 2023-09-30T01:33:04.420 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.366+0000 7f2b67fff700 10 monclient(hunting): start opening mon connection 2023-09-30T01:33:04.420 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.366+0000 7f2b67fff700 1 -- --> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] -- mon_getmap magic: 0 v1 -- 0x7f2b6809c290 con 0x7f2b681356d0 2023-09-30T01:33:04.420 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.366+0000 7f2b67fff700 10 monclient(hunting): _renew_subs 2023-09-30T01:33:04.420 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.366+0000 7f2b67fff700 10 monclient(hunting): authenticate will time out at 856.404711s 2023-09-30T01:33:04.420 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.366+0000 7f2b66ffd700 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2b6809cf30 0x7f2b68096e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:33:04.420 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.366+0000 7f2b677fe700 1 --2- >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f2b681356d0 0x7f2b68137ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:33:04.420 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.366+0000 7f2b667fc700 1 --2- >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7f2b6809d300 0x7f2b6809d6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:33:04.420 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.366+0000 7f2b66ffd700 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2b6809cf30 0x7f2b68096e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3301/0 says I am v2:172.21.15.88:52432/0 (socket says 172.21.15.88:52432) 2023-09-30T01:33:04.421 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.366+0000 7f2b66ffd700 1 -- 172.21.15.88:0/2761573468 learned_addr learned my addr 172.21.15.88:0/2761573468 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-09-30T01:33:04.421 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.366+0000 7f2b66ffd700 10 monclient(hunting): get_auth_request con 0x7f2b6809cf30 auth_method 0 2023-09-30T01:33:04.421 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.366+0000 7f2b66ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:33:04.421 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.366+0000 7f2b66ffd700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:33:04.421 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.366+0000 7f2b66ffd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:33:04.421 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.366+0000 7f2b667fc700 10 monclient(hunting): get_auth_request con 0x7f2b6809d300 auth_method 0 2023-09-30T01:33:04.421 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.366+0000 7f2b667fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:33:04.421 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.366+0000 7f2b667fc700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:33:04.421 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.366+0000 7f2b667fc700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:33:04.421 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.366+0000 7f2b66ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:33:04.421 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.366+0000 7f2b66ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:33:04.421 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.366+0000 7f2b66ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:33:04.422 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.366+0000 7f2b677fe700 10 monclient(hunting): get_auth_request con 0x7f2b681356d0 auth_method 0 2023-09-30T01:33:04.422 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.366+0000 7f2b677fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:33:04.422 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.366+0000 7f2b677fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:33:04.422 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.366+0000 7f2b677fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:33:04.422 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.366+0000 7f2b677fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:33:04.422 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.366+0000 7f2b677fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:33:04.422 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.366+0000 7f2b677fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:33:04.422 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.366+0000 7f2b66ffd700 10 monclient(hunting): handle_auth_done global_id 4259 payload 274 2023-09-30T01:33:04.422 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.366+0000 7f2b66ffd700 1 -- 172.21.15.88:0/2761573468 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f2b681356d0 msgr2=0x7f2b68137ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:33:04.422 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.366+0000 7f2b66ffd700 1 --2- 172.21.15.88:0/2761573468 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f2b681356d0 0x7f2b68137ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:04.422 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.366+0000 7f2b66ffd700 1 -- 172.21.15.88:0/2761573468 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7f2b6809d300 msgr2=0x7f2b6809d6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:33:04.422 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.366+0000 7f2b66ffd700 1 --2- 172.21.15.88:0/2761573468 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7f2b6809d300 0x7f2b6809d6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:04.423 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.366+0000 7f2b66ffd700 10 monclient: _finish_hunting 0 2023-09-30T01:33:04.423 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.366+0000 7f2b66ffd700 1 monclient: found mon.noname-b 2023-09-30T01:33:04.423 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.366+0000 7f2b66ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:33:04.423 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.366+0000 7f2b66ffd700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.88:3301/0 2023-09-30T01:33:04.423 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.366+0000 7f2b66ffd700 1 -- 172.21.15.88:0/2761573468 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b6804bb20 con 0x7f2b6809cf30 2023-09-30T01:33:04.423 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.366+0000 7f2b66ffd700 1 --2- 172.21.15.88:0/2761573468 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2b6809cf30 0x7f2b68096e20 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f2b5c009a10 tx=0x7f2b5c009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bf7b4f80f356cfe4 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:33:04.423 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.366+0000 7f2b65ffb700 1 -- 172.21.15.88:0/2761573468 <== mon.2 v2:172.21.15.88:3301/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f2b5c00bca0 con 0x7f2b6809cf30 2023-09-30T01:33:04.423 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.366+0000 7f2b65ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:33:04.423 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.366+0000 7f2b65ffb700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T01:33:04.423 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.366+0000 7f2b65ffb700 10 monclient: dump: 2023-09-30T01:33:04.423 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:33:04.424 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:33:04.424 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:04.424 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:04.424 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:33:04.424 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:33:04.424 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:33:04.424 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:33:04.424 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:33:04.424 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:33:04.424 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:33:04.424 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:33:04.424 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:33:04.425 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:33:04.425 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:33:04.425 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:33:04.425 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.366+0000 7f2b65ffb700 10 monclient: _finish_auth 0 2023-09-30T01:33:04.425 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.366+0000 7f2b65ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:33:04.425 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.366+0000 7f2b65ffb700 1 -- 172.21.15.88:0/2761573468 <== mon.2 v2:172.21.15.88:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b5c00be20 con 0x7f2b6809cf30 2023-09-30T01:33:04.425 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.366+0000 7f2b67fff700 5 monclient: authenticate success, global_id 4259 2023-09-30T01:33:04.425 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.366+0000 7f2b67fff700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:33:04.425 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.366+0000 7f2b65ffb700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:33:04.425 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.366+0000 7f2b67fff700 10 monclient: get_monmap_and_config success 2023-09-30T01:33:04.425 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.366+0000 7f2b67fff700 4 set_mon_vals no callback set 2023-09-30T01:33:04.425 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.366+0000 7f2b65ffb700 1 -- 172.21.15.88:0/2761573468 <== mon.2 v2:172.21.15.88:3301/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f2b5c0118a0 con 0x7f2b6809cf30 2023-09-30T01:33:04.426 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.366+0000 7f2b65ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:33:04.426 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.366+0000 7f2b65ffb700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T01:33:04.426 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.366+0000 7f2b65ffb700 10 monclient: dump: 2023-09-30T01:33:04.426 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:33:04.426 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:33:04.426 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:04.426 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:04.426 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:33:04.426 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:33:04.426 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:33:04.426 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:33:04.426 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:33:04.427 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:33:04.427 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:33:04.427 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:33:04.427 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:33:04.427 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:33:04.427 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:33:04.427 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:33:04.427 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.366+0000 7f2b67fff700 10 monclient: shutdown 2023-09-30T01:33:04.427 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.366+0000 7f2b67fff700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:33:04.427 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.366+0000 7f2b67fff700 1 -- 172.21.15.88:0/2761573468 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2b6809cf30 msgr2=0x7f2b68096e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:33:04.427 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.366+0000 7f2b67fff700 1 --2- 172.21.15.88:0/2761573468 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2b6809cf30 0x7f2b68096e20 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f2b5c009a10 tx=0x7f2b5c009250 comp rx=0 tx=0).stop 2023-09-30T01:33:04.427 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.366+0000 7f2b67fff700 1 -- 172.21.15.88:0/2761573468 shutdown_connections 2023-09-30T01:33:04.428 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.366+0000 7f2b67fff700 1 --2- 172.21.15.88:0/2761573468 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f2b681356d0 0x7f2b68137ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:04.428 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.366+0000 7f2b67fff700 1 --2- 172.21.15.88:0/2761573468 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7f2b6809d300 0x7f2b6809d6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:04.428 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.366+0000 7f2b67fff700 1 --2- 172.21.15.88:0/2761573468 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2b6809cf30 0x7f2b68096e20 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:04.428 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.366+0000 7f2b67fff700 1 -- 172.21.15.88:0/2761573468 >> 172.21.15.88:0/2761573468 conn(0x7f2b680a2540 msgr2=0x7f2b680a2910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:33:04.428 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.366+0000 7f2b67fff700 1 -- 172.21.15.88:0/2761573468 shutdown_connections 2023-09-30T01:33:04.428 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.366+0000 7f2b67fff700 1 -- 172.21.15.88:0/2761573468 wait complete. 2023-09-30T01:33:04.428 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.370+0000 7f2b67fff700 10 monclient: build_initial_monmap 2023-09-30T01:33:04.428 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.370+0000 7f2b67fff700 10 monclient: monmap: 2023-09-30T01:33:04.428 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2023-09-30T01:33:04.428 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:33:04.428 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 0.000000 2023-09-30T01:33:04.428 INFO:teuthology.orchestra.run.smithi088.stderr:created 0.000000 2023-09-30T01:33:04.429 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2023-09-30T01:33:04.429 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 1 2023-09-30T01:33:04.429 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.noname-a 2023-09-30T01:33:04.429 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.noname-f 2023-09-30T01:33:04.429 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.noname-b 2023-09-30T01:33:04.429 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.noname-g 2023-09-30T01:33:04.429 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.noname-c 2023-09-30T01:33:04.429 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.noname-h 2023-09-30T01:33:04.429 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.noname-d 2023-09-30T01:33:04.429 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.noname-i 2023-09-30T01:33:04.429 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.noname-e 2023-09-30T01:33:04.429 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:33:04.430 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.370+0000 7f2b67fff700 1 Processor -- start 2023-09-30T01:33:04.430 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.370+0000 7f2b67fff700 1 -- start start 2023-09-30T01:33:04.430 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.370+0000 7f2b67fff700 10 monclient: init 2023-09-30T01:33:04.430 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.370+0000 7f2b67fff700 10 monclient: _reopen_session rank -1 2023-09-30T01:33:04.430 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.370+0000 7f2b67fff700 10 monclient: _add_conns ranks=[4,3,6,7,8,0,2,1,5] 2023-09-30T01:33:04.430 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.370+0000 7f2b67fff700 1 --2- >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f2b6809cf30 0x7f2b6806edf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:33:04.430 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.370+0000 7f2b67fff700 10 monclient(hunting): picked mon.noname-c con 0x7f2b6809cf30 addr [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] 2023-09-30T01:33:04.430 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.370+0000 7f2b67fff700 1 --2- >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7f2b6809d300 0x7f2b68127610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:33:04.430 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.370+0000 7f2b67fff700 10 monclient(hunting): picked mon.noname-g con 0x7f2b6809d300 addr [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] 2023-09-30T01:33:04.430 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.370+0000 7f2b67fff700 1 --2- >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7f2b681356d0 0x7f2b68127b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:33:04.430 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.370+0000 7f2b67fff700 10 monclient(hunting): picked mon.noname-d con 0x7f2b681356d0 addr [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] 2023-09-30T01:33:04.430 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.370+0000 7f2b67fff700 10 monclient(hunting): start opening mon connection 2023-09-30T01:33:04.431 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.370+0000 7f2b67fff700 1 -- --> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b68128ad0 con 0x7f2b6809d300 2023-09-30T01:33:04.431 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.370+0000 7f2b67fff700 10 monclient(hunting): start opening mon connection 2023-09-30T01:33:04.431 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.370+0000 7f2b67fff700 1 -- --> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] -- mon_getmap magic: 0 v1 -- 0x7f2b68098810 con 0x7f2b6809cf30 2023-09-30T01:33:04.431 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.370+0000 7f2b67fff700 10 monclient(hunting): start opening mon connection 2023-09-30T01:33:04.431 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.370+0000 7f2b67fff700 1 -- --> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] -- mon_getmap magic: 0 v1 -- 0x7f2b6812a5d0 con 0x7f2b681356d0 2023-09-30T01:33:04.431 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.370+0000 7f2b67fff700 10 monclient(hunting): _renew_subs 2023-09-30T01:33:04.431 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.370+0000 7f2b67fff700 10 monclient(hunting): authenticate will time out at 856.409888s 2023-09-30T01:33:04.431 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.370+0000 7f2b66ffd700 1 --2- >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f2b6809cf30 0x7f2b6806edf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:33:04.431 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.370+0000 7f2b66ffd700 1 --2- >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f2b6809cf30 0x7f2b6806edf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3302/0 says I am v2:172.21.15.88:50416/0 (socket says 172.21.15.88:50416) 2023-09-30T01:33:04.431 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.370+0000 7f2b677fe700 1 --2- >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7f2b681356d0 0x7f2b68127b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:33:04.431 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.370+0000 7f2b66ffd700 1 -- 172.21.15.88:0/3788500524 learned_addr learned my addr 172.21.15.88:0/3788500524 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-09-30T01:33:04.431 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.370+0000 7f2b66ffd700 10 monclient(hunting): get_auth_request con 0x7f2b6809cf30 auth_method 0 2023-09-30T01:33:04.432 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.370+0000 7f2b66ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:33:04.432 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.370+0000 7f2b66ffd700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:33:04.432 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.370+0000 7f2b66ffd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:33:04.432 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.370+0000 7f378ec82700 10 monclient: get_monmap_and_config 2023-09-30T01:33:04.432 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.370+0000 7f378ec82700 10 monclient: build_initial_monmap 2023-09-30T01:33:04.432 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.370+0000 7f378ec82700 10 monclient: monmap: 2023-09-30T01:33:04.432 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2023-09-30T01:33:04.433 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:33:04.433 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:33:04.373880+0000 2023-09-30T01:33:04.433 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:33:04.373880+0000 2023-09-30T01:33:04.433 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2023-09-30T01:33:04.433 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:33:04.433 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a-legacy 2023-09-30T01:33:04.433 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.107:6789/0 mon.noname-f-legacy 2023-09-30T01:33:04.433 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.noname-b 2023-09-30T01:33:04.433 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.noname-g 2023-09-30T01:33:04.433 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.noname-c 2023-09-30T01:33:04.433 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.noname-h 2023-09-30T01:33:04.433 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.noname-d 2023-09-30T01:33:04.434 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.noname-i 2023-09-30T01:33:04.434 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.noname-e 2023-09-30T01:33:04.434 INFO:teuthology.orchestra.run.smithi088.stderr:9: v2:172.21.15.88:3300/0 mon.noname-a 2023-09-30T01:33:04.434 INFO:teuthology.orchestra.run.smithi088.stderr:10: v2:172.21.15.107:3300/0 mon.noname-f 2023-09-30T01:33:04.434 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:33:04.435 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.370+0000 7f2b677fe700 10 monclient(hunting): get_auth_request con 0x7f2b681356d0 auth_method 0 2023-09-30T01:33:04.435 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.370+0000 7f2b677fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:33:04.435 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.370+0000 7f2b677fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:33:04.435 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.370+0000 7f2b677fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:33:04.435 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.370+0000 7f2b667fc700 1 --2- 172.21.15.88:0/3788500524 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7f2b6809d300 0x7f2b68127610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:33:04.435 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.370+0000 7f2b66ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:33:04.436 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.370+0000 7f2b66ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:33:04.436 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.370+0000 7f2b66ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:33:04.436 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.370+0000 7f2b667fc700 10 monclient(hunting): get_auth_request con 0x7f2b6809d300 auth_method 0 2023-09-30T01:33:04.436 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.370+0000 7f2b667fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:33:04.436 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.370+0000 7f2b667fc700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:33:04.436 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.370+0000 7f2b667fc700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:33:04.436 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.370+0000 7f2b677fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:33:04.436 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.370+0000 7f2b677fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:33:04.436 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.370+0000 7f2b677fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:33:04.436 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.370+0000 7f2b667fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:33:04.436 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.370+0000 7f2b667fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:33:04.437 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.370+0000 7f2b667fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:33:04.437 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.370+0000 7f2b677fe700 10 monclient(hunting): handle_auth_done global_id 4227 payload 824 2023-09-30T01:33:04.437 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.370+0000 7f2b677fe700 1 -- 172.21.15.88:0/3788500524 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f2b6809cf30 msgr2=0x7f2b6806edf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:33:04.437 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.370+0000 7f2b677fe700 1 --2- 172.21.15.88:0/3788500524 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f2b6809cf30 0x7f2b6806edf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:04.437 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.370+0000 7f2b677fe700 1 -- 172.21.15.88:0/3788500524 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7f2b6809d300 msgr2=0x7f2b68127610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:33:04.437 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.370+0000 7f2b677fe700 1 --2- 172.21.15.88:0/3788500524 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7f2b6809d300 0x7f2b68127610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:04.437 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.370+0000 7f2b677fe700 10 monclient: _finish_hunting 0 2023-09-30T01:33:04.437 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.370+0000 7f2b677fe700 1 monclient: found mon.noname-d 2023-09-30T01:33:04.437 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.370+0000 7f2b677fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:33:04.437 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.370+0000 7f2b677fe700 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.88:3303/0 2023-09-30T01:33:04.437 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.370+0000 7f2b677fe700 1 -- 172.21.15.88:0/3788500524 --> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b5c0094f0 con 0x7f2b681356d0 2023-09-30T01:33:04.437 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.370+0000 7f2b66ffd700 1 --2- 172.21.15.88:0/3788500524 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f2b6809cf30 0x7f2b6806edf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:33:04.438 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.370+0000 7f2b677fe700 1 --2- 172.21.15.88:0/3788500524 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7f2b681356d0 0x7f2b68127b50 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f2b6806a120 tx=0x7f2b6000b040 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:33:04.438 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.370+0000 7f378ec82700 1 Processor -- start 2023-09-30T01:33:04.438 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.370+0000 7f2b4effd700 1 -- 172.21.15.88:0/3788500524 <== mon.6 v2:172.21.15.88:3303/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f2b6000cce0 con 0x7f2b681356d0 2023-09-30T01:33:04.438 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.370+0000 7f2b4effd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:33:04.438 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.370+0000 7f2b4effd700 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2023-09-30T01:33:04.439 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.370+0000 7f378ec82700 1 -- start start 2023-09-30T01:33:04.439 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.370+0000 7f378ec82700 10 monclient: init 2023-09-30T01:33:04.440 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.370+0000 7f2b4effd700 10 monclient: dump: 2023-09-30T01:33:04.440 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:33:04.440 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:33:04.440 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:04.440 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:04.440 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:33:04.440 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:33:04.440 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:33:04.440 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:33:04.441 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:33:04.441 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:33:04.441 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:33:04.441 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:33:04.441 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:33:04.441 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:33:04.441 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:33:04.441 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:33:04.441 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.370+0000 7f2b4effd700 10 monclient: _finish_auth 0 2023-09-30T01:33:04.441 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.370+0000 7f2b4effd700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:33:04.441 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.370+0000 7f2b67fff700 5 monclient: authenticate success, global_id 4227 2023-09-30T01:33:04.441 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.370+0000 7f2b4effd700 1 -- 172.21.15.88:0/3788500524 <== mon.6 v2:172.21.15.88:3303/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b6000f040 con 0x7f2b681356d0 2023-09-30T01:33:04.442 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.370+0000 7f2b4effd700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:33:04.442 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.370+0000 7f2b4effd700 1 -- 172.21.15.88:0/3788500524 <== mon.6 v2:172.21.15.88:3303/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f2b6000ce60 con 0x7f2b681356d0 2023-09-30T01:33:04.442 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.370+0000 7f2b67fff700 10 monclient: _renew_subs 2023-09-30T01:33:04.442 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.370+0000 7f2b67fff700 10 monclient: _send_mon_message to mon.d at v2:172.21.15.88:3303/0 2023-09-30T01:33:04.442 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.370+0000 7f2b67fff700 1 -- 172.21.15.88:0/3788500524 --> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b681304e0 con 0x7f2b681356d0 2023-09-30T01:33:04.442 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.370+0000 7f2b64ff9700 4 set_mon_vals no callback set 2023-09-30T01:33:04.442 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.370+0000 7f2b67fff700 10 monclient: _renew_subs 2023-09-30T01:33:04.442 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.370+0000 7f2b67fff700 10 monclient: _send_mon_message to mon.d at v2:172.21.15.88:3303/0 2023-09-30T01:33:04.442 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.370+0000 7f2b67fff700 1 -- 172.21.15.88:0/3788500524 --> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b68129120 con 0x7f2b681356d0 2023-09-30T01:33:04.442 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.370+0000 7f2b4effd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:33:04.443 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.370+0000 7f378ec82700 10 monclient: _reopen_session rank -1 2023-09-30T01:33:04.443 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.370+0000 7f378ec82700 10 monclient: _add_conns ranks=[8,9,10,3,2,4,5,1,0,6,7] 2023-09-30T01:33:04.443 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.370+0000 7f2b4effd700 10 monclient: got monmap 1 from mon.d (according to old e1) 2023-09-30T01:33:04.443 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.370+0000 7f2b4effd700 10 monclient: dump: 2023-09-30T01:33:04.443 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:33:04.443 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:33:04.443 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:04.443 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:04.444 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:33:04.444 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:33:04.444 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:33:04.444 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:33:04.444 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:33:04.444 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:33:04.444 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:33:04.444 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:33:04.444 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:33:04.444 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:33:04.444 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:33:04.444 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:33:04.446 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.370+0000 7f378ec82700 1 --2- >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f3788097a40 0x7f3788132700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:33:04.446 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.370+0000 7f378ec82700 10 monclient(hunting): picked mon.noname-e con 0x7f3788097a40 addr [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] 2023-09-30T01:33:04.447 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.370+0000 7f378ec82700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f37881311c0 0x7f3788132c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:33:04.447 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.370+0000 7f378ec82700 10 monclient(hunting): picked mon.noname-a con 0x7f37881311c0 addr v2:172.21.15.88:3300/0 2023-09-30T01:33:04.447 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.370+0000 7f378ec82700 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f37881332e0 0x7f3788133740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:33:04.447 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.370+0000 7f378ec82700 10 monclient(hunting): picked mon.noname-f con 0x7f37881332e0 addr v2:172.21.15.107:3300/0 2023-09-30T01:33:04.447 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.370+0000 7f378ec82700 10 monclient(hunting): start opening mon connection 2023-09-30T01:33:04.447 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.374+0000 7f378ec82700 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3788131590 con 0x7f37881311c0 2023-09-30T01:33:04.447 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.374+0000 7f378ec82700 10 monclient(hunting): start opening mon connection 2023-09-30T01:33:04.447 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.374+0000 7f378ec82700 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f378809a5d0 con 0x7f37881332e0 2023-09-30T01:33:04.447 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.374+0000 7f378ec82700 10 monclient(hunting): start opening mon connection 2023-09-30T01:33:04.447 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.374+0000 7f378ec82700 1 -- --> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] -- mon_getmap magic: 0 v1 -- 0x7f3788097e10 con 0x7f3788097a40 2023-09-30T01:33:04.447 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.374+0000 7f378ec82700 10 monclient(hunting): _renew_subs 2023-09-30T01:33:04.448 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.374+0000 7f378ec82700 10 monclient(hunting): authenticate will time out at 856.412063s 2023-09-30T01:33:04.448 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.374+0000 7f378dc80700 1 --2- >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f3788097a40 0x7f3788132700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:33:04.448 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.374+0000 7f378dc80700 1 --2- >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f3788097a40 0x7f3788132700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3304/0 says I am v2:172.21.15.88:36676/0 (socket says 172.21.15.88:36676) 2023-09-30T01:33:04.448 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.374+0000 7f378dc80700 1 -- 172.21.15.88:0/4094278472 learned_addr learned my addr 172.21.15.88:0/4094278472 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-09-30T01:33:04.448 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.374+0000 7f378dc80700 10 monclient(hunting): get_auth_request con 0x7f3788097a40 auth_method 0 2023-09-30T01:33:04.448 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.374+0000 7f378dc80700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:33:04.448 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.374+0000 7f378dc80700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:33:04.448 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.374+0000 7f378dc80700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:33:04.448 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.374+0000 7f378e481700 1 --2- 172.21.15.88:0/4094278472 >> v2:172.21.15.107:3300/0 conn(0x7f37881332e0 0x7f3788133740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:33:04.448 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.374+0000 7f378e481700 10 monclient(hunting): get_auth_request con 0x7f37881332e0 auth_method 0 2023-09-30T01:33:04.448 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.374+0000 7f378e481700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:33:04.448 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.374+0000 7f378e481700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:33:04.449 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.374+0000 7f378e481700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:33:04.449 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.374+0000 7f378dc80700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:33:04.449 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.374+0000 7f378dc80700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:33:04.449 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.374+0000 7f378d47f700 1 --2- 172.21.15.88:0/4094278472 >> v2:172.21.15.88:3300/0 conn(0x7f37881311c0 0x7f3788132c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:33:04.449 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.374+0000 7f378dc80700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:33:04.449 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.374+0000 7f378d47f700 10 monclient(hunting): get_auth_request con 0x7f37881311c0 auth_method 0 2023-09-30T01:33:04.449 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.374+0000 7f378d47f700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:33:04.449 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.374+0000 7f378d47f700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:33:04.449 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.374+0000 7f378d47f700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:33:04.449 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.374+0000 7f378e481700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:33:04.449 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.374+0000 7f378e481700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:33:04.449 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.374+0000 7f378e481700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:33:04.450 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.374+0000 7f378d47f700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:33:04.450 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.374+0000 7f378d47f700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:33:04.450 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.374+0000 7f378d47f700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:33:04.450 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.374+0000 7f378dc80700 10 monclient(hunting): handle_auth_done global_id 4265 payload 274 2023-09-30T01:33:04.450 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.374+0000 7f378dc80700 1 -- 172.21.15.88:0/4094278472 >> v2:172.21.15.107:3300/0 conn(0x7f37881332e0 msgr2=0x7f3788133740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:33:04.450 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.374+0000 7f378dc80700 1 --2- 172.21.15.88:0/4094278472 >> v2:172.21.15.107:3300/0 conn(0x7f37881332e0 0x7f3788133740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:04.450 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.374+0000 7f378dc80700 1 -- 172.21.15.88:0/4094278472 >> v2:172.21.15.88:3300/0 conn(0x7f37881311c0 msgr2=0x7f3788132c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:33:04.450 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.374+0000 7f2b4effd700 1 -- 172.21.15.88:0/3788500524 <== mon.6 v2:172.21.15.88:3303/0 4 ==== mgrmap(e 4) v1 ==== 40088+0+0 (secure 0 0 0) 0x7f2b6000c860 con 0x7f2b681356d0 2023-09-30T01:33:04.451 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.374+0000 7f2b4effd700 1 --2- 172.21.15.88:0/3788500524 >> v2:172.21.15.107:6812/29926 conn(0x7f2b540332d0 0x7f2b54035780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:33:04.451 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.374+0000 7f2b4effd700 1 -- 172.21.15.88:0/3788500524 <== mon.6 v2:172.21.15.88:3303/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 3502+0+0 (secure 0 0 0) 0x7f2b60043980 con 0x7f2b681356d0 2023-09-30T01:33:04.451 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.374+0000 7f378dc80700 1 --2- 172.21.15.88:0/4094278472 >> v2:172.21.15.88:3300/0 conn(0x7f37881311c0 0x7f3788132c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:04.451 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.374+0000 7f378dc80700 10 monclient: _finish_hunting 0 2023-09-30T01:33:04.451 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.374+0000 7f378dc80700 1 monclient: found mon.noname-e 2023-09-30T01:33:04.451 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.374+0000 7f378dc80700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:33:04.451 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.374+0000 7f378dc80700 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.88:3304/0 2023-09-30T01:33:04.452 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.374+0000 7f378dc80700 1 -- 172.21.15.88:0/4094278472 --> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f378804bb20 con 0x7f3788097a40 2023-09-30T01:33:04.452 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.374+0000 7f378d47f700 1 --2- 172.21.15.88:0/4094278472 >> v2:172.21.15.88:3300/0 conn(0x7f37881311c0 0x7f3788132c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:33:04.452 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.374+0000 7f2b66ffd700 1 --2- 172.21.15.88:0/3788500524 >> v2:172.21.15.107:6812/29926 conn(0x7f2b540332d0 0x7f2b54035780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:33:04.452 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.374+0000 7f2b66ffd700 10 monclient: get_auth_request con 0x7f2b540332d0 auth_method 0 2023-09-30T01:33:04.453 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.374+0000 7f378dc80700 1 --2- 172.21.15.88:0/4094278472 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f3788097a40 0x7f3788132700 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f37880a3a30 tx=0x7f3778009250 comp rx=0 tx=0).ready entity=mon.8 client_cookie=e3e1108f8a8bf3dd server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:33:04.453 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.374+0000 7f378cc7e700 1 -- 172.21.15.88:0/4094278472 <== mon.8 v2:172.21.15.88:3304/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f377800bac0 con 0x7f3788097a40 2023-09-30T01:33:04.453 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.374+0000 7f378cc7e700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:33:04.453 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.374+0000 7f378cc7e700 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2023-09-30T01:33:04.453 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.374+0000 7f378cc7e700 10 monclient: dump: 2023-09-30T01:33:04.453 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:33:04.453 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:33:04.453 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:04.453 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:04.454 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:33:04.454 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:33:04.454 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:33:04.454 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:33:04.454 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:33:04.454 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:33:04.454 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:33:04.454 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:33:04.454 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:33:04.454 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:33:04.454 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:33:04.454 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:33:04.455 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.374+0000 7f378cc7e700 10 monclient: _finish_auth 0 2023-09-30T01:33:04.455 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.374+0000 7f378cc7e700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:33:04.455 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.374+0000 7f378ec82700 5 monclient: authenticate success, global_id 4265 2023-09-30T01:33:04.455 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.374+0000 7f378ec82700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:33:04.455 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.374+0000 7f378cc7e700 1 -- 172.21.15.88:0/4094278472 <== mon.8 v2:172.21.15.88:3304/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f377800bc20 con 0x7f3788097a40 2023-09-30T01:33:04.455 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.374+0000 7f378cc7e700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:33:04.455 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.374+0000 7f378ec82700 10 monclient: get_monmap_and_config success 2023-09-30T01:33:04.455 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.374+0000 7f378cc7e700 1 -- 172.21.15.88:0/4094278472 <== mon.8 v2:172.21.15.88:3304/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f377800bd90 con 0x7f3788097a40 2023-09-30T01:33:04.455 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.374+0000 7f378cc7e700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:33:04.455 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.374+0000 7f378ec82700 4 set_mon_vals no callback set 2023-09-30T01:33:04.455 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.374+0000 7f378cc7e700 10 monclient: got monmap 1 from mon.e (according to old e1) 2023-09-30T01:33:04.455 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.374+0000 7f378cc7e700 10 monclient: dump: 2023-09-30T01:33:04.456 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:33:04.456 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:33:04.456 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:04.456 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:04.456 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:33:04.456 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:33:04.456 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:33:04.456 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:33:04.456 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:33:04.456 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:33:04.456 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:33:04.456 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:33:04.457 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:33:04.457 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:33:04.457 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:33:04.457 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:33:04.457 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.374+0000 7f2b66ffd700 1 --2- 172.21.15.88:0/3788500524 >> v2:172.21.15.107:6812/29926 conn(0x7f2b540332d0 0x7f2b54035780 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f2b5c009880 tx=0x7f2b5c017000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:33:04.457 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.374+0000 7f85c759e700 10 monclient: get_monmap_and_config 2023-09-30T01:33:04.458 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.374+0000 7f85c759e700 10 monclient: build_initial_monmap 2023-09-30T01:33:04.458 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.374+0000 7f85c759e700 10 monclient: monmap: 2023-09-30T01:33:04.458 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2023-09-30T01:33:04.458 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:33:04.458 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:33:04.377728+0000 2023-09-30T01:33:04.458 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:33:04.377728+0000 2023-09-30T01:33:04.458 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2023-09-30T01:33:04.458 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:33:04.458 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a-legacy 2023-09-30T01:33:04.458 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.107:6789/0 mon.noname-f-legacy 2023-09-30T01:33:04.458 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.noname-b 2023-09-30T01:33:04.459 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.noname-g 2023-09-30T01:33:04.459 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.noname-c 2023-09-30T01:33:04.459 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.noname-h 2023-09-30T01:33:04.459 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.noname-d 2023-09-30T01:33:04.459 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.noname-i 2023-09-30T01:33:04.459 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.noname-e 2023-09-30T01:33:04.459 INFO:teuthology.orchestra.run.smithi088.stderr:9: v2:172.21.15.88:3300/0 mon.noname-a 2023-09-30T01:33:04.459 INFO:teuthology.orchestra.run.smithi088.stderr:10: v2:172.21.15.107:3300/0 mon.noname-f 2023-09-30T01:33:04.459 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:33:04.459 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.374+0000 7f2b67fff700 1 --2- 172.21.15.88:0/3788500524 >> v2:172.21.15.107:6804/30274 conn(0x7f2b68128ca0 0x7f2b6805dac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:33:04.460 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.374+0000 7f2b67fff700 1 -- 172.21.15.88:0/3788500524 --> v2:172.21.15.107:6804/30274 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f2b6806b850 con 0x7f2b68128ca0 2023-09-30T01:33:04.460 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.374+0000 7f85c759e700 1 Processor -- start 2023-09-30T01:33:04.460 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.374+0000 7f85c759e700 1 -- start start 2023-09-30T01:33:04.460 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.374+0000 7f85c759e700 10 monclient: init 2023-09-30T01:33:04.461 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.374+0000 7f378ec82700 10 monclient: shutdown 2023-09-30T01:33:04.461 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.374+0000 7f378ec82700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:33:04.461 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.374+0000 7f378ec82700 1 -- 172.21.15.88:0/4094278472 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f3788097a40 msgr2=0x7f3788132700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:33:04.461 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.374+0000 7f378ec82700 1 --2- 172.21.15.88:0/4094278472 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f3788097a40 0x7f3788132700 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f37880a3a30 tx=0x7f3778009250 comp rx=0 tx=0).stop 2023-09-30T01:33:04.461 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.374+0000 7f378ec82700 1 -- 172.21.15.88:0/4094278472 shutdown_connections 2023-09-30T01:33:04.461 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.374+0000 7f378ec82700 1 --2- 172.21.15.88:0/4094278472 >> v2:172.21.15.107:3300/0 conn(0x7f37881332e0 0x7f3788133740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:04.461 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.374+0000 7f378ec82700 1 --2- 172.21.15.88:0/4094278472 >> v2:172.21.15.88:3300/0 conn(0x7f37881311c0 0x7f3788132c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:04.461 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.374+0000 7f378ec82700 1 --2- 172.21.15.88:0/4094278472 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f3788097a40 0x7f3788132700 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:04.462 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.374+0000 7f378ec82700 1 -- 172.21.15.88:0/4094278472 >> 172.21.15.88:0/4094278472 conn(0x7f378806e5e0 msgr2=0x7f378806e9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:33:04.462 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.374+0000 7f2b667fc700 1 --2- 172.21.15.88:0/3788500524 >> v2:172.21.15.107:6804/30274 conn(0x7f2b68128ca0 0x7f2b6805dac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:33:04.462 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.374+0000 7f2b667fc700 10 monclient: get_auth_request con 0x7f2b68128ca0 auth_method 0 2023-09-30T01:33:04.462 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.374+0000 7f378ec82700 1 -- 172.21.15.88:0/4094278472 shutdown_connections 2023-09-30T01:33:04.463 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.374+0000 7f378ec82700 1 -- 172.21.15.88:0/4094278472 wait complete. 2023-09-30T01:33:04.463 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.374+0000 7f85c759e700 10 monclient: _reopen_session rank -1 2023-09-30T01:33:04.463 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.374+0000 7f85c759e700 10 monclient: _add_conns ranks=[0,7,9,3,6,5,2,1,8,10,4] 2023-09-30T01:33:04.463 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.374+0000 7f85c759e700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f85c809cf30 addr v1:172.21.15.88:6789/0 2023-09-30T01:33:04.463 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.374+0000 7f85c759e700 1 --2- >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7f85c809da10 0x7f85c8096e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:33:04.463 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.374+0000 7f85c759e700 10 monclient(hunting): picked mon.noname-i con 0x7f85c809da10 addr [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] 2023-09-30T01:33:04.463 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.374+0000 7f85c759e700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f85c81366e0 0x7f85c8138ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:33:04.464 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.374+0000 7f85c759e700 10 monclient(hunting): picked mon.noname-a con 0x7f85c81366e0 addr v2:172.21.15.88:3300/0 2023-09-30T01:33:04.464 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.374+0000 7f85c759e700 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85c804bb20 con 0x7f85c809cf30 2023-09-30T01:33:04.464 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.374+0000 7f85c759e700 10 monclient(hunting): start opening mon connection 2023-09-30T01:33:04.464 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.374+0000 7f85c759e700 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f85c8096b00 con 0x7f85c81366e0 2023-09-30T01:33:04.464 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.374+0000 7f85c759e700 10 monclient(hunting): start opening mon connection 2023-09-30T01:33:04.464 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.374+0000 7f85c759e700 1 -- --> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] -- mon_getmap magic: 0 v1 -- 0x7f85c809c530 con 0x7f85c809da10 2023-09-30T01:33:04.464 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.374+0000 7f85c759e700 10 monclient(hunting): _renew_subs 2023-09-30T01:33:04.464 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.374+0000 7f85c759e700 10 monclient(hunting): authenticate will time out at 856.415683s 2023-09-30T01:33:04.464 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.374+0000 7f378ec82700 10 monclient: build_initial_monmap 2023-09-30T01:33:04.465 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.374+0000 7f85c659c700 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f85c809cf30 0x7f85c809d300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:54098/0 (socket says 172.21.15.88:54098) 2023-09-30T01:33:04.465 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.374+0000 7f85c659c700 1 -- 172.21.15.88:0/3521064764 learned_addr learned my addr 172.21.15.88:0/3521064764 (peer_addr_for_me v1:172.21.15.88:0/0) 2023-09-30T01:33:04.465 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.374+0000 7f378ec82700 10 monclient: monmap: 2023-09-30T01:33:04.465 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2023-09-30T01:33:04.465 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:33:04.465 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 0.000000 2023-09-30T01:33:04.465 INFO:teuthology.orchestra.run.smithi088.stderr:created 0.000000 2023-09-30T01:33:04.466 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2023-09-30T01:33:04.466 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 1 2023-09-30T01:33:04.466 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.noname-a 2023-09-30T01:33:04.466 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.noname-f 2023-09-30T01:33:04.466 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.noname-b 2023-09-30T01:33:04.466 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.noname-g 2023-09-30T01:33:04.466 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.noname-c 2023-09-30T01:33:04.466 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.noname-h 2023-09-30T01:33:04.466 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.noname-d 2023-09-30T01:33:04.466 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.noname-i 2023-09-30T01:33:04.466 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.noname-e 2023-09-30T01:33:04.467 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:33:04.467 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.374+0000 7f2b667fc700 1 --2- 172.21.15.88:0/3788500524 >> v2:172.21.15.107:6804/30274 conn(0x7f2b68128ca0 0x7f2b6805dac0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:33:04.468 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.374+0000 7f85c6d9d700 1 --2- 172.21.15.88:0/3521064764 >> v2:172.21.15.88:3300/0 conn(0x7f85c81366e0 0x7f85c8138ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:33:04.468 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.378+0000 7f85c6d9d700 10 monclient(hunting): get_auth_request con 0x7f85c81366e0 auth_method 0 2023-09-30T01:33:04.468 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.378+0000 7f85c6d9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:33:04.468 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.378+0000 7f85c6d9d700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:33:04.468 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.378+0000 7f85c6d9d700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:33:04.468 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.378+0000 7f85c6d9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:33:04.468 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.378+0000 7f85c6d9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:33:04.468 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.378+0000 7f85c6d9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:33:04.468 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.378+0000 7f85c5d9b700 1 --2- 172.21.15.88:0/3521064764 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7f85c809da10 0x7f85c8096e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:33:04.468 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.378+0000 7f85c6d9d700 10 monclient(hunting): handle_auth_done global_id 4320 payload 274 2023-09-30T01:33:04.468 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.378+0000 7f85c6d9d700 1 -- 172.21.15.88:0/3521064764 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7f85c809da10 msgr2=0x7f85c8096e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:33:04.469 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.378+0000 7f85c6d9d700 1 --2- 172.21.15.88:0/3521064764 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7f85c809da10 0x7f85c8096e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:04.469 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.378+0000 7f85c6d9d700 1 -- 172.21.15.88:0/3521064764 >> v1:172.21.15.88:6789/0 conn(0x7f85c809cf30 legacy=0x7f85c809d300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:33:04.469 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.378+0000 7f85c6d9d700 10 monclient: _finish_hunting 0 2023-09-30T01:33:04.469 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.378+0000 7f85c6d9d700 1 monclient: found mon.noname-a 2023-09-30T01:33:04.469 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.378+0000 7f85c6d9d700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:33:04.469 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.378+0000 7f85c6d9d700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.88:3300/0 2023-09-30T01:33:04.469 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.378+0000 7f85c6d9d700 1 -- 172.21.15.88:0/3521064764 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85c809c290 con 0x7f85c81366e0 2023-09-30T01:33:04.469 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.378+0000 7f85c5d9b700 10 monclient: get_auth_request con 0x7f85c809da10 auth_method 0 2023-09-30T01:33:04.469 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.378+0000 7f85c5d9b700 1 --2- 172.21.15.88:0/3521064764 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7f85c809da10 0x7f85c8096e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-09-30T01:33:04.469 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.378+0000 7f85c6d9d700 1 --2- 172.21.15.88:0/3521064764 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f85c81366e0 0x7f85c8138ac0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f85c0009e60 tx=0x7f85c0009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3d99606dcaaeec48 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:33:04.470 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.378+0000 7f378ec82700 1 Processor -- start 2023-09-30T01:33:04.471 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.378+0000 7f378ec82700 1 -- start start 2023-09-30T01:33:04.471 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.378+0000 7f378ec82700 10 monclient: init 2023-09-30T01:33:04.471 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.378+0000 7f378ec82700 10 monclient: _reopen_session rank -1 2023-09-30T01:33:04.471 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.378+0000 7f378ec82700 10 monclient: _add_conns ranks=[8,0,2,5,7,6,1,4,3] 2023-09-30T01:33:04.471 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.378+0000 7f378ec82700 1 --2- >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f3788097a40 0x7f37880a0a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:33:04.471 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.378+0000 7f378ec82700 10 monclient(hunting): picked mon.noname-e con 0x7f3788097a40 addr [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] 2023-09-30T01:33:04.471 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.378+0000 7f378ec82700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f37881311c0 0x7f37880a0f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:33:04.471 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.378+0000 7f378ec82700 10 monclient(hunting): picked mon.noname-a con 0x7f37881311c0 addr [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] 2023-09-30T01:33:04.471 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.378+0000 7f378ec82700 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f37881332e0 0x7f37880a1480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:33:04.471 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.378+0000 7f378ec82700 10 monclient(hunting): picked mon.noname-b con 0x7f37881332e0 addr [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] 2023-09-30T01:33:04.472 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.378+0000 7f378ec82700 10 monclient(hunting): start opening mon connection 2023-09-30T01:33:04.472 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.378+0000 7f378ec82700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f378812d010 con 0x7f37881311c0 2023-09-30T01:33:04.472 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.378+0000 7f378ec82700 10 monclient(hunting): start opening mon connection 2023-09-30T01:33:04.472 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.378+0000 7f378ec82700 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f378811d9e0 con 0x7f37881332e0 2023-09-30T01:33:04.472 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.378+0000 7f378ec82700 10 monclient(hunting): start opening mon connection 2023-09-30T01:33:04.472 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.378+0000 7f378ec82700 1 -- --> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] -- mon_getmap magic: 0 v1 -- 0x7f3788099c30 con 0x7f3788097a40 2023-09-30T01:33:04.472 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.378+0000 7f378ec82700 10 monclient(hunting): _renew_subs 2023-09-30T01:33:04.472 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.378+0000 7f378ec82700 10 monclient(hunting): authenticate will time out at 856.416884s 2023-09-30T01:33:04.472 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.378+0000 7f378dc80700 1 --2- >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f3788097a40 0x7f37880a0a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:33:04.472 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.378+0000 7f378dc80700 1 --2- >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f3788097a40 0x7f37880a0a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3304/0 says I am v2:172.21.15.88:36680/0 (socket says 172.21.15.88:36680) 2023-09-30T01:33:04.472 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.378+0000 7f378e481700 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f37881332e0 0x7f37880a1480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:33:04.472 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.378+0000 7f378dc80700 1 -- 172.21.15.88:0/2575437705 learned_addr learned my addr 172.21.15.88:0/2575437705 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-09-30T01:33:04.473 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.378+0000 7f378dc80700 10 monclient(hunting): get_auth_request con 0x7f3788097a40 auth_method 0 2023-09-30T01:33:04.473 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.378+0000 7f378dc80700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:33:04.473 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.378+0000 7f378dc80700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:33:04.473 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.378+0000 7f378dc80700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:33:04.473 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.378+0000 7f378e481700 10 monclient(hunting): get_auth_request con 0x7f37881332e0 auth_method 0 2023-09-30T01:33:04.473 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.378+0000 7f378e481700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:33:04.473 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.378+0000 7f378e481700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:33:04.473 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.378+0000 7f378e481700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:33:04.473 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.378+0000 7f378dc80700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:33:04.473 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.378+0000 7f378dc80700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:33:04.473 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.378+0000 7f378dc80700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:33:04.473 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.378+0000 7f378e481700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:33:04.474 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.378+0000 7f378e481700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:33:04.474 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.378+0000 7f378e481700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:33:04.474 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.378+0000 7f378d47f700 1 --2- 172.21.15.88:0/2575437705 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f37881311c0 0x7f37880a0f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:33:04.474 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.378+0000 7f378d47f700 10 monclient(hunting): get_auth_request con 0x7f37881311c0 auth_method 0 2023-09-30T01:33:04.474 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.378+0000 7f378d47f700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:33:04.474 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.378+0000 7f378d47f700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:33:04.474 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.378+0000 7f378d47f700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:33:04.474 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.378+0000 7f378d47f700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:33:04.474 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.378+0000 7f378d47f700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:33:04.474 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.378+0000 7f378d47f700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:33:04.474 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.378+0000 7f378dc80700 10 monclient(hunting): handle_auth_done global_id 4274 payload 824 2023-09-30T01:33:04.474 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.378+0000 7f378dc80700 1 -- 172.21.15.88:0/2575437705 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f37881332e0 msgr2=0x7f37880a1480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:33:04.475 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.378+0000 7f378dc80700 1 --2- 172.21.15.88:0/2575437705 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f37881332e0 0x7f37880a1480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:04.475 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.378+0000 7f378dc80700 1 -- 172.21.15.88:0/2575437705 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f37881311c0 msgr2=0x7f37880a0f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:33:04.475 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.378+0000 7f378dc80700 1 --2- 172.21.15.88:0/2575437705 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f37881311c0 0x7f37880a0f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:04.475 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.378+0000 7f378dc80700 10 monclient: _finish_hunting 0 2023-09-30T01:33:04.475 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.378+0000 7f378dc80700 1 monclient: found mon.noname-e 2023-09-30T01:33:04.475 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.378+0000 7f378dc80700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:33:04.475 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.378+0000 7f378dc80700 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.88:3304/0 2023-09-30T01:33:04.475 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.378+0000 7f378dc80700 1 -- 172.21.15.88:0/2575437705 --> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3784009420 con 0x7f3788097a40 2023-09-30T01:33:04.475 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.378+0000 7f2b4effd700 1 -- 172.21.15.88:0/3788500524 <== osd.3 v2:172.21.15.107:6804/30274 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23999 (crc 0 0 0) 0x7f2b6806b850 con 0x7f2b68128ca0 2023-09-30T01:33:04.478 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.378+0000 7f378dc80700 1 --2- 172.21.15.88:0/2575437705 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f3788097a40 0x7f37880a0a00 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f3788097f50 tx=0x7f3778017000 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:33:04.478 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.382+0000 7f377dffb700 1 -- 172.21.15.88:0/2575437705 <== mon.8 v2:172.21.15.88:3304/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f3778011420 con 0x7f3788097a40 2023-09-30T01:33:04.478 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.382+0000 7f377dffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:33:04.478 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.382+0000 7f377dffb700 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2023-09-30T01:33:04.478 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.382+0000 7f377dffb700 10 monclient: dump: 2023-09-30T01:33:04.478 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:33:04.478 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:33:04.478 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:04.478 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:04.479 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:33:04.479 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:33:04.479 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:33:04.479 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:33:04.479 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:33:04.479 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:33:04.479 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:33:04.479 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:33:04.479 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:33:04.479 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:33:04.479 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:33:04.479 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:33:04.480 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.382+0000 7f377dffb700 10 monclient: _finish_auth 0 2023-09-30T01:33:04.480 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.382+0000 7f377dffb700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:33:04.480 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.382+0000 7f378ec82700 5 monclient: authenticate success, global_id 4274 2023-09-30T01:33:04.480 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.382+0000 7f377dffb700 1 -- 172.21.15.88:0/2575437705 <== mon.8 v2:172.21.15.88:3304/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f37780115c0 con 0x7f3788097a40 2023-09-30T01:33:04.480 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.382+0000 7f378ec82700 10 monclient: _renew_subs 2023-09-30T01:33:04.480 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.382+0000 7f378ec82700 10 monclient: _send_mon_message to mon.e at v2:172.21.15.88:3304/0 2023-09-30T01:33:04.480 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.382+0000 7f378ec82700 1 -- 172.21.15.88:0/2575437705 --> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f37780094f0 con 0x7f3788097a40 2023-09-30T01:33:04.480 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.382+0000 7f377dffb700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:33:04.480 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.382+0000 7f377dffb700 1 -- 172.21.15.88:0/2575437705 <== mon.8 v2:172.21.15.88:3304/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f3778011bb0 con 0x7f3788097a40 2023-09-30T01:33:04.480 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.382+0000 7f377dffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:33:04.480 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.382+0000 7f377dffb700 10 monclient: got monmap 1 from mon.e (according to old e1) 2023-09-30T01:33:04.481 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.382+0000 7f377f7fe700 4 set_mon_vals no callback set 2023-09-30T01:33:04.481 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.382+0000 7f377dffb700 10 monclient: dump: 2023-09-30T01:33:04.481 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:33:04.481 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:33:04.481 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:04.481 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:04.481 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:33:04.481 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:33:04.481 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:33:04.481 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:33:04.481 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:33:04.481 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:33:04.482 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:33:04.482 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:33:04.482 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:33:04.482 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:33:04.482 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:33:04.482 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:33:04.482 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.382+0000 7f378ec82700 10 monclient: _renew_subs 2023-09-30T01:33:04.482 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.382+0000 7f378ec82700 10 monclient: _send_mon_message to mon.e at v2:172.21.15.88:3304/0 2023-09-30T01:33:04.482 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.382+0000 7f378ec82700 1 -- 172.21.15.88:0/2575437705 --> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f37880a2dc0 con 0x7f3788097a40 2023-09-30T01:33:04.482 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.382+0000 7f377dffb700 1 -- 172.21.15.88:0/2575437705 <== mon.8 v2:172.21.15.88:3304/0 4 ==== mgrmap(e 4) v1 ==== 40088+0+0 (secure 0 0 0) 0x7f3778011d10 con 0x7f3788097a40 2023-09-30T01:33:04.482 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.382+0000 7f377dffb700 1 --2- 172.21.15.88:0/2575437705 >> v2:172.21.15.107:6812/29926 conn(0x7f376c0332d0 0x7f376c035780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:33:04.482 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.382+0000 7f377dffb700 1 -- 172.21.15.88:0/2575437705 <== mon.8 v2:172.21.15.88:3304/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 3502+0+0 (secure 0 0 0) 0x7f37780448a0 con 0x7f3788097a40 2023-09-30T01:33:04.482 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.382+0000 7f378d47f700 1 --2- 172.21.15.88:0/2575437705 >> v2:172.21.15.107:6812/29926 conn(0x7f376c0332d0 0x7f376c035780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:33:04.483 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.382+0000 7f378d47f700 10 monclient: get_auth_request con 0x7f376c0332d0 auth_method 0 2023-09-30T01:33:04.483 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.382+0000 7f378d47f700 1 --2- 172.21.15.88:0/2575437705 >> v2:172.21.15.107:6812/29926 conn(0x7f376c0332d0 0x7f376c035780 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f37880a2230 tx=0x7f37840091a0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:33:04.483 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.390+0000 7f378ec82700 1 --2- 172.21.15.88:0/2575437705 >> v2:172.21.15.107:6808/30276 conn(0x7f37880a19c0 0x7f378805dfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:33:04.483 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.390+0000 7f378ec82700 1 -- 172.21.15.88:0/2575437705 --> v2:172.21.15.107:6808/30276 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f378811dc20 con 0x7f37880a19c0 2023-09-30T01:33:04.483 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.390+0000 7f378e481700 1 --2- 172.21.15.88:0/2575437705 >> v2:172.21.15.107:6808/30276 conn(0x7f37880a19c0 0x7f378805dfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:33:04.483 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.390+0000 7f378e481700 10 monclient: get_auth_request con 0x7f37880a19c0 auth_method 0 2023-09-30T01:33:04.483 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.390+0000 7f378e481700 1 --2- 172.21.15.88:0/2575437705 >> v2:172.21.15.107:6808/30276 conn(0x7f37880a19c0 0x7f378805dfd0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:33:04.483 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.390+0000 7f377dffb700 1 -- 172.21.15.88:0/2575437705 <== osd.4 v2:172.21.15.107:6808/30276 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23999 (crc 0 0 0) 0x7f378811dc20 con 0x7f37880a19c0 2023-09-30T01:33:04.485 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.394+0000 7f2b67fff700 1 -- 172.21.15.88:0/3788500524 --> v2:172.21.15.107:6804/30274 -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f2b6806bc40 con 0x7f2b68128ca0 2023-09-30T01:33:04.485 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.394+0000 7f2b4effd700 1 -- 172.21.15.88:0/3788500524 <== osd.3 v2:172.21.15.107:6804/30274 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f2b6806bc40 con 0x7f2b68128ca0 2023-09-30T01:33:04.486 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.394+0000 7f2b67fff700 1 -- 172.21.15.88:0/3788500524 >> v2:172.21.15.107:6804/30274 conn(0x7f2b68128ca0 msgr2=0x7f2b6805dac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:33:04.487 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.394+0000 7f2b67fff700 1 --2- 172.21.15.88:0/3788500524 >> v2:172.21.15.107:6804/30274 conn(0x7f2b68128ca0 0x7f2b6805dac0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:04.487 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.394+0000 7f2b67fff700 1 -- 172.21.15.88:0/3788500524 >> v2:172.21.15.107:6812/29926 conn(0x7f2b540332d0 msgr2=0x7f2b54035780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:33:04.487 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.394+0000 7f2b67fff700 1 --2- 172.21.15.88:0/3788500524 >> v2:172.21.15.107:6812/29926 conn(0x7f2b540332d0 0x7f2b54035780 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f2b5c009880 tx=0x7f2b5c017000 comp rx=0 tx=0).stop 2023-09-30T01:33:04.487 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.398+0000 7f2b67fff700 10 monclient: shutdown 2023-09-30T01:33:04.487 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.398+0000 7f2b67fff700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:33:04.487 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.398+0000 7f2b67fff700 1 -- 172.21.15.88:0/3788500524 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7f2b681356d0 msgr2=0x7f2b68127b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:33:04.487 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.398+0000 7f2b67fff700 1 --2- 172.21.15.88:0/3788500524 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7f2b681356d0 0x7f2b68127b50 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f2b6806a120 tx=0x7f2b6000b040 comp rx=0 tx=0).stop 2023-09-30T01:33:04.487 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.398+0000 7f2b677fe700 1 -- 172.21.15.88:0/3788500524 reap_dead start 2023-09-30T01:33:04.487 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.398+0000 7f2b67fff700 1 -- 172.21.15.88:0/3788500524 shutdown_connections 2023-09-30T01:33:04.487 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.398+0000 7f2b67fff700 1 -- 172.21.15.88:0/3788500524 >> 172.21.15.88:0/3788500524 conn(0x7f2b680a2540 msgr2=0x7f2b68134d50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:33:04.487 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.398+0000 7f2b67fff700 1 -- 172.21.15.88:0/3788500524 shutdown_connections 2023-09-30T01:33:04.488 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.398+0000 7f2b67fff700 1 -- 172.21.15.88:0/3788500524 wait complete. 2023-09-30T01:33:04.488 INFO:teuthology.orchestra.run.smithi088.stdout:47244640259 2023-09-30T01:33:04.488 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2023-09-30T01:33:04.489 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.410+0000 7f378ec82700 1 -- 172.21.15.88:0/2575437705 --> v2:172.21.15.107:6808/30276 -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f3788067b60 con 0x7f37880a19c0 2023-09-30T01:33:04.489 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.410+0000 7f377dffb700 1 -- 172.21.15.88:0/2575437705 <== osd.4 v2:172.21.15.107:6808/30276 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f3788067b60 con 0x7f37880a19c0 2023-09-30T01:33:04.490 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.410+0000 7f378ec82700 1 -- 172.21.15.88:0/2575437705 >> v2:172.21.15.107:6808/30276 conn(0x7f37880a19c0 msgr2=0x7f378805dfd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:33:04.490 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.410+0000 7f378ec82700 1 --2- 172.21.15.88:0/2575437705 >> v2:172.21.15.107:6808/30276 conn(0x7f37880a19c0 0x7f378805dfd0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:04.490 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.410+0000 7f378ec82700 1 -- 172.21.15.88:0/2575437705 >> v2:172.21.15.107:6812/29926 conn(0x7f376c0332d0 msgr2=0x7f376c035780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:33:04.490 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.410+0000 7f378ec82700 1 --2- 172.21.15.88:0/2575437705 >> v2:172.21.15.107:6812/29926 conn(0x7f376c0332d0 0x7f376c035780 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f37880a2230 tx=0x7f37840091a0 comp rx=0 tx=0).stop 2023-09-30T01:33:04.490 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.410+0000 7f378ec82700 10 monclient: shutdown 2023-09-30T01:33:04.490 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.410+0000 7f378ec82700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:33:04.490 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.410+0000 7f378ec82700 1 -- 172.21.15.88:0/2575437705 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f3788097a40 msgr2=0x7f37880a0a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:33:04.490 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.410+0000 7f378ec82700 1 --2- 172.21.15.88:0/2575437705 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f3788097a40 0x7f37880a0a00 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f3788097f50 tx=0x7f3778017000 comp rx=0 tx=0).stop 2023-09-30T01:33:04.491 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.410+0000 7f378e481700 1 -- 172.21.15.88:0/2575437705 reap_dead start 2023-09-30T01:33:04.491 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.410+0000 7f378ec82700 1 -- 172.21.15.88:0/2575437705 shutdown_connections 2023-09-30T01:33:04.491 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.410+0000 7f378ec82700 1 -- 172.21.15.88:0/2575437705 >> 172.21.15.88:0/2575437705 conn(0x7f378806e5e0 msgr2=0x7f3788130770 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:33:04.491 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.414+0000 7f378ec82700 1 -- 172.21.15.88:0/2575437705 shutdown_connections 2023-09-30T01:33:04.491 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.414+0000 7f378ec82700 1 -- 172.21.15.88:0/2575437705 wait complete. 2023-09-30T01:33:04.491 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.414+0000 7f0353fff700 1 -- 172.21.15.88:0/3802989775 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f0364007d80 con 0x7f036c08d5e0 2023-09-30T01:33:04.491 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.414+0000 7f0353fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:33:04.491 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.414+0000 7f0353fff700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T01:33:04.492 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.414+0000 7f0353fff700 10 monclient: dump: 2023-09-30T01:33:04.492 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:33:04.492 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:33:04.492 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:04.492 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:04.492 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:33:04.492 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:33:04.492 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:33:04.492 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:33:04.492 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:33:04.492 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:33:04.492 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:33:04.493 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:33:04.493 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:33:04.493 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:33:04.493 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:33:04.493 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:33:04.493 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.414+0000 7f0353fff700 10 monclient: _finish_auth 0 2023-09-30T01:33:04.493 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.414+0000 7f0353fff700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:33:04.493 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.414+0000 7f0353fff700 1 -- 172.21.15.88:0/3802989775 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0364012430 con 0x7f036c08d5e0 2023-09-30T01:33:04.493 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.414+0000 7f0353fff700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:33:04.493 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.414+0000 7f0353fff700 1 -- 172.21.15.88:0/3802989775 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f0364012a20 con 0x7f036c08d5e0 2023-09-30T01:33:04.493 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.414+0000 7f0353fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:33:04.493 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.414+0000 7f0353fff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:33:04.493 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.414+0000 7f0353fff700 10 monclient: dump: 2023-09-30T01:33:04.494 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:33:04.494 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:33:04.494 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:04.494 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:04.494 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:33:04.494 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:33:04.494 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:33:04.494 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:33:04.494 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:33:04.494 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:33:04.494 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:33:04.494 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:33:04.495 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:33:04.495 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:33:04.495 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:33:04.495 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:33:04.495 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.414+0000 7f037147f700 5 monclient: authenticate success, global_id 4284 2023-09-30T01:33:04.495 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.414+0000 7f037147f700 10 monclient: _renew_subs 2023-09-30T01:33:04.495 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.414+0000 7f037147f700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.88:3300/0 2023-09-30T01:33:04.495 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.414+0000 7f037147f700 1 -- 172.21.15.88:0/3802989775 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f036c08f120 con 0x7f036c08d5e0 2023-09-30T01:33:04.495 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.414+0000 7fca08ff9700 1 -- 172.21.15.88:0/3275358350 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7fca0001a450 con 0x7fca0c0a3270 2023-09-30T01:33:04.496 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.414+0000 7fca08ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:33:04.496 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.414+0000 7f037147f700 10 monclient: _renew_subs 2023-09-30T01:33:04.496 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.414+0000 7f037147f700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.88:3300/0 2023-09-30T01:33:04.496 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.414+0000 7f037147f700 1 -- 172.21.15.88:0/3802989775 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f036c08e470 con 0x7f036c08d5e0 2023-09-30T01:33:04.496 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.414+0000 7f036a7fc700 4 set_mon_vals no callback set 2023-09-30T01:33:04.497 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.414+0000 7fca08ff9700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T01:33:04.497 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.414+0000 7fca08ff9700 10 monclient: dump: 2023-09-30T01:33:04.497 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:33:04.497 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:33:04.497 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:04.497 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:04.498 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:33:04.498 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:33:04.498 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:33:04.498 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:33:04.498 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:33:04.498 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:33:04.498 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:33:04.498 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:33:04.498 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:33:04.498 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:33:04.498 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:33:04.498 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:33:04.499 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.414+0000 7fca08ff9700 10 monclient: _finish_auth 0 2023-09-30T01:33:04.499 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.414+0000 7fca08ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:33:04.499 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.414+0000 7fca12e64700 5 monclient: authenticate success, global_id 4293 2023-09-30T01:33:04.499 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.414+0000 7fca08ff9700 1 -- 172.21.15.88:0/3275358350 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca0001a610 con 0x7fca0c0a3270 2023-09-30T01:33:04.499 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.414+0000 7fca12e64700 10 monclient: _renew_subs 2023-09-30T01:33:04.499 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.414+0000 7fca12e64700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.88:3300/0 2023-09-30T01:33:04.499 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.414+0000 7fca12e64700 1 -- 172.21.15.88:0/3275358350 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca0c1223c0 con 0x7fca0c0a3270 2023-09-30T01:33:04.499 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.414+0000 7fca12e64700 10 monclient: _renew_subs 2023-09-30T01:33:04.499 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.414+0000 7fca12e64700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.88:3300/0 2023-09-30T01:33:04.499 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.414+0000 7fca12e64700 1 -- 172.21.15.88:0/3275358350 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fca0c121070 con 0x7fca0c0a3270 2023-09-30T01:33:04.499 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.414+0000 7fca08ff9700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:33:04.499 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.414+0000 7fca08ff9700 1 -- 172.21.15.88:0/3275358350 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7fca0001ac00 con 0x7fca0c0a3270 2023-09-30T01:33:04.500 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.414+0000 7fca08ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:33:04.500 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.414+0000 7fca0affd700 4 set_mon_vals no callback set 2023-09-30T01:33:04.500 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.414+0000 7fca08ff9700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:33:04.500 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.414+0000 7fca08ff9700 10 monclient: dump: 2023-09-30T01:33:04.500 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:33:04.500 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:33:04.500 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:04.500 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:04.500 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:33:04.500 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:33:04.500 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:33:04.500 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:33:04.501 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:33:04.501 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:33:04.501 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:33:04.501 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:33:04.501 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:33:04.501 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:33:04.501 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:33:04.501 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:33:04.501 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.414+0000 7fca08ff9700 1 -- 172.21.15.88:0/3275358350 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 4) v1 ==== 40088+0+0 (secure 0 0 0) 0x7fca00006c00 con 0x7fca0c0a3270 2023-09-30T01:33:04.501 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.414+0000 7fca12e64700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.88:3300/0 2023-09-30T01:33:04.501 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.414+0000 7fca12e64700 1 -- 172.21.15.88:0/3275358350 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fc9ec000f20 con 0x7fca0c0a3270 2023-09-30T01:33:04.502 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.414+0000 7f66c8ff9700 1 -- 172.21.15.88:0/669031240 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f66c001bc60 con 0x7f66cc097e40 2023-09-30T01:33:04.502 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.414+0000 7f66c8ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:33:04.502 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.414+0000 7f66c8ff9700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T01:33:04.502 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.414+0000 7f66c8ff9700 10 monclient: dump: 2023-09-30T01:33:04.502 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:33:04.502 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:33:04.502 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:04.502 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:04.502 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:33:04.503 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:33:04.503 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:33:04.503 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:33:04.503 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:33:04.503 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:33:04.503 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:33:04.503 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:33:04.503 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:33:04.503 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:33:04.503 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:33:04.503 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:33:04.504 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.414+0000 7f66c8ff9700 10 monclient: _finish_auth 0 2023-09-30T01:33:04.504 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.414+0000 7f66c8ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:33:04.504 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.414+0000 7f66d1c91700 5 monclient: authenticate success, global_id 4302 2023-09-30T01:33:04.504 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.414+0000 7f66d1c91700 10 monclient: _renew_subs 2023-09-30T01:33:04.504 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.414+0000 7f66c8ff9700 1 -- 172.21.15.88:0/669031240 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f66c001bde0 con 0x7f66cc097e40 2023-09-30T01:33:04.504 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.414+0000 7f66d1c91700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.88:3300/0 2023-09-30T01:33:04.504 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.414+0000 7f66d1c91700 1 -- 172.21.15.88:0/669031240 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f66cc0905b0 con 0x7f66cc097e40 2023-09-30T01:33:04.504 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.414+0000 7fca08ff9700 1 --2- 172.21.15.88:0/3275358350 >> v2:172.21.15.107:6812/29926 conn(0x7fc9f0033200 0x7fc9f00356b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:33:04.505 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.414+0000 7fca08ff9700 1 -- 172.21.15.88:0/3275358350 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 3502+0+0 (secure 0 0 0) 0x7fca00044d40 con 0x7fca0c0a3270 2023-09-30T01:33:04.505 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.414+0000 7f66d1c91700 10 monclient: _renew_subs 2023-09-30T01:33:04.505 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.414+0000 7f66d1c91700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.88:3300/0 2023-09-30T01:33:04.505 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.414+0000 7f66d1c91700 1 -- 172.21.15.88:0/669031240 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f66cc08f970 con 0x7f66cc097e40 2023-09-30T01:33:04.506 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.414+0000 7f85c559a700 1 -- 172.21.15.88:0/3521064764 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f85c000e070 con 0x7f85c81366e0 2023-09-30T01:33:04.506 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.414+0000 7f85c559a700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:33:04.506 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.414+0000 7f85c559a700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T01:33:04.506 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.414+0000 7f85c559a700 10 monclient: dump: 2023-09-30T01:33:04.506 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:33:04.506 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:33:04.506 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:04.506 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:04.506 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:33:04.506 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:33:04.506 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:33:04.507 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:33:04.507 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:33:04.507 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:33:04.507 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:33:04.507 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:33:04.507 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:33:04.507 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:33:04.507 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:33:04.507 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:33:04.507 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.414+0000 7f85c559a700 10 monclient: _finish_auth 0 2023-09-30T01:33:04.507 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.414+0000 7f85c559a700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:33:04.507 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.414+0000 7f85c559a700 1 -- 172.21.15.88:0/3521064764 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85c000bc30 con 0x7f85c81366e0 2023-09-30T01:33:04.508 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.414+0000 7f85c559a700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:33:04.508 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.414+0000 7f85c559a700 1 -- 172.21.15.88:0/3521064764 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f85c000bda0 con 0x7f85c81366e0 2023-09-30T01:33:04.508 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.414+0000 7f85c559a700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:33:04.508 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.414+0000 7f85c559a700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:33:04.508 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.414+0000 7f85c559a700 10 monclient: dump: 2023-09-30T01:33:04.508 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:33:04.508 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:33:04.508 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:04.508 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:04.508 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:33:04.508 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:33:04.508 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:33:04.509 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:33:04.509 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:33:04.509 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:33:04.509 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:33:04.509 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:33:04.509 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:33:04.509 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:33:04.509 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:33:04.509 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:33:04.509 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.414+0000 7fca08ff9700 1 --2- 172.21.15.88:0/3275358350 >> v2:172.21.15.88:6808/30177 conn(0x7fc9f0038ca0 0x7fc9f003b0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:33:04.510 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.414+0000 7fca08ff9700 1 -- 172.21.15.88:0/3275358350 --> v2:172.21.15.88:6808/30177 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc9f003b720 con 0x7fc9f0038ca0 2023-09-30T01:33:04.510 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.414+0000 7f85c759e700 5 monclient: authenticate success, global_id 4320 2023-09-30T01:33:04.510 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.414+0000 7f85c759e700 10 monclient: get_monmap_and_config success 2023-09-30T01:33:04.510 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.414+0000 7f85c759e700 4 set_mon_vals no callback set 2023-09-30T01:33:04.510 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.414+0000 7f66c8ff9700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:33:04.511 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.414+0000 7f66c8ff9700 1 -- 172.21.15.88:0/669031240 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f66c0011d20 con 0x7f66cc097e40 2023-09-30T01:33:04.511 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.414+0000 7f66c8ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:33:04.511 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.414+0000 7f66c8ff9700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:33:04.511 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.414+0000 7f66c8ff9700 10 monclient: dump: 2023-09-30T01:33:04.511 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:33:04.511 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:33:04.511 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:04.511 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:04.511 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:33:04.511 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:33:04.511 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:33:04.512 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:33:04.512 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:33:04.512 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:33:04.512 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:33:04.512 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:33:04.512 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:33:04.512 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:33:04.512 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:33:04.512 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:33:04.512 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.414+0000 7f66caffd700 4 set_mon_vals no callback set 2023-09-30T01:33:04.513 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.414+0000 7fca08ff9700 1 -- 172.21.15.88:0/3275358350 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_get_version_reply(handle=1 version=15) v2 ==== 24+0+0 (secure 0 0 0) 0x7fca0001ad60 con 0x7fca0c0a3270 2023-09-30T01:33:04.513 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.414+0000 7fca08ff9700 10 monclient: handle_get_version_reply finishing 1 version 15 2023-09-30T01:33:04.513 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.414+0000 7fca11401700 1 --2- 172.21.15.88:0/3275358350 >> v2:172.21.15.88:6808/30177 conn(0x7fc9f0038ca0 0x7fc9f003b0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:33:04.513 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.414+0000 7fca11401700 10 monclient: get_auth_request con 0x7fc9f0038ca0 auth_method 0 2023-09-30T01:33:04.513 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.414+0000 7fca0bfff700 1 --2- 172.21.15.88:0/3275358350 >> v2:172.21.15.107:6812/29926 conn(0x7fc9f0033200 0x7fc9f00356b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:33:04.514 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.414+0000 7fca0bfff700 10 monclient: get_auth_request con 0x7fc9f0033200 auth_method 0 2023-09-30T01:33:04.514 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.414+0000 7fca11401700 1 --2- 172.21.15.88:0/3275358350 >> v2:172.21.15.88:6808/30177 conn(0x7fc9f0038ca0 0x7fc9f003b0b0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:33:04.514 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.414+0000 7f037147f700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.88:3300/0 2023-09-30T01:33:04.514 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.414+0000 7f037147f700 1 -- 172.21.15.88:0/3802989775 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f036c0710d0 con 0x7f036c08d5e0 2023-09-30T01:33:04.514 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.418+0000 7f66d1c91700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.88:3300/0 2023-09-30T01:33:04.514 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.418+0000 7f66d1c91700 1 -- 172.21.15.88:0/669031240 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f66a8000f20 con 0x7f66cc097e40 2023-09-30T01:33:04.515 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.418+0000 7f85c759e700 10 monclient: shutdown 2023-09-30T01:33:04.515 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.418+0000 7f85c759e700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:33:04.515 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.418+0000 7f85c759e700 1 -- 172.21.15.88:0/3521064764 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f85c81366e0 msgr2=0x7f85c8138ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:33:04.515 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.418+0000 7f85c759e700 1 --2- 172.21.15.88:0/3521064764 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f85c81366e0 0x7f85c8138ac0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f85c0009e60 tx=0x7f85c0009780 comp rx=0 tx=0).stop 2023-09-30T01:33:04.515 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.418+0000 7fca0bfff700 1 --2- 172.21.15.88:0/3275358350 >> v2:172.21.15.107:6812/29926 conn(0x7fc9f0033200 0x7fc9f00356b0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fca0c11d860 tx=0x7fc9f80091a0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:33:04.516 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.418+0000 7f0353fff700 1 -- 172.21.15.88:0/3802989775 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 4) v1 ==== 40088+0+0 (secure 0 0 0) 0x7f03640125a0 con 0x7f036c08d5e0 2023-09-30T01:33:04.516 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.418+0000 7f85c759e700 1 -- 172.21.15.88:0/3521064764 shutdown_connections 2023-09-30T01:33:04.516 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.418+0000 7f85c759e700 1 --2- 172.21.15.88:0/3521064764 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f85c81366e0 0x7f85c8138ac0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:04.516 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.418+0000 7f85c759e700 1 --2- 172.21.15.88:0/3521064764 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7f85c809da10 0x7f85c8096e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:04.516 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.418+0000 7f85c759e700 1 -- 172.21.15.88:0/3521064764 >> 172.21.15.88:0/3521064764 conn(0x7f85c80a2540 msgr2=0x7f85c80a2910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:33:04.517 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.418+0000 7f0353fff700 1 --2- 172.21.15.88:0/3802989775 >> v2:172.21.15.107:6812/29926 conn(0x7f03580332d0 0x7f0358035780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:33:04.517 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.418+0000 7f0353fff700 1 -- 172.21.15.88:0/3802989775 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 3502+0+0 (secure 0 0 0) 0x7f03640449a0 con 0x7f036c08d5e0 2023-09-30T01:33:04.517 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.418+0000 7f85c759e700 1 -- 172.21.15.88:0/3521064764 shutdown_connections 2023-09-30T01:33:04.517 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.418+0000 7f85c759e700 1 -- 172.21.15.88:0/3521064764 wait complete. 2023-09-30T01:33:04.517 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.418+0000 7f0353fff700 1 --2- 172.21.15.88:0/3802989775 >> v2:172.21.15.107:6800/30275 conn(0x7f0358038d70 0x7f035803b180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:33:04.518 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.418+0000 7f0353fff700 1 -- 172.21.15.88:0/3802989775 --> v2:172.21.15.107:6800/30275 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f035803b7f0 con 0x7f0358038d70 2023-09-30T01:33:04.518 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.418+0000 7f0353fff700 1 -- 172.21.15.88:0/3802989775 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_get_version_reply(handle=1 version=15) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0364049220 con 0x7f036c08d5e0 2023-09-30T01:33:04.518 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.418+0000 7f0353fff700 10 monclient: handle_get_version_reply finishing 1 version 15 2023-09-30T01:33:04.518 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.418+0000 7f036bfff700 1 --2- 172.21.15.88:0/3802989775 >> v2:172.21.15.107:6812/29926 conn(0x7f03580332d0 0x7f0358035780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:33:04.518 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.418+0000 7f036bfff700 10 monclient: get_auth_request con 0x7f03580332d0 auth_method 0 2023-09-30T01:33:04.518 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.418+0000 7f85c759e700 10 monclient: build_initial_monmap 2023-09-30T01:33:04.518 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.418+0000 7f85c759e700 10 monclient: monmap: 2023-09-30T01:33:04.519 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2023-09-30T01:33:04.519 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:33:04.519 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 0.000000 2023-09-30T01:33:04.519 INFO:teuthology.orchestra.run.smithi088.stderr:created 0.000000 2023-09-30T01:33:04.519 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2023-09-30T01:33:04.519 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 1 2023-09-30T01:33:04.519 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.noname-a 2023-09-30T01:33:04.519 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.noname-f 2023-09-30T01:33:04.519 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.noname-b 2023-09-30T01:33:04.519 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.noname-g 2023-09-30T01:33:04.519 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.noname-c 2023-09-30T01:33:04.519 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.noname-h 2023-09-30T01:33:04.520 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.noname-d 2023-09-30T01:33:04.520 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.noname-i 2023-09-30T01:33:04.520 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.noname-e 2023-09-30T01:33:04.520 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:33:04.520 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.418+0000 7f85c759e700 1 Processor -- start 2023-09-30T01:33:04.520 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.418+0000 7f036bfff700 1 --2- 172.21.15.88:0/3802989775 >> v2:172.21.15.107:6812/29926 conn(0x7f03580332d0 0x7f0358035780 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f036000ac00 tx=0x7f0360006f90 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:33:04.520 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.418+0000 7f036b7fe700 1 --2- 172.21.15.88:0/3802989775 >> v2:172.21.15.107:6800/30275 conn(0x7f0358038d70 0x7f035803b180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:33:04.521 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.418+0000 7f036b7fe700 10 monclient: get_auth_request con 0x7f0358038d70 auth_method 0 2023-09-30T01:33:04.521 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.418+0000 7fca08ff9700 1 -- 172.21.15.88:0/3275358350 <== osd.0 v2:172.21.15.88:6808/30177 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23999 (crc 0 0 0) 0x7fc9f003b720 con 0x7fc9f0038ca0 2023-09-30T01:33:04.521 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.418+0000 7f85c759e700 1 -- start start 2023-09-30T01:33:04.521 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.418+0000 7f85c759e700 10 monclient: init 2023-09-30T01:33:04.522 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.418+0000 7f66c8ff9700 1 -- 172.21.15.88:0/669031240 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 4) v1 ==== 40088+0+0 (secure 0 0 0) 0x7f66c0011420 con 0x7f66cc097e40 2023-09-30T01:33:04.522 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.418+0000 7f66c8ff9700 1 --2- 172.21.15.88:0/669031240 >> v2:172.21.15.107:6812/29926 conn(0x7f66b00331d0 0x7f66b0035680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:33:04.522 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.418+0000 7f66c8ff9700 1 -- 172.21.15.88:0/669031240 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 3502+0+0 (secure 0 0 0) 0x7f66c00449f0 con 0x7f66cc097e40 2023-09-30T01:33:04.522 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.418+0000 7f66c8ff9700 1 --2- 172.21.15.88:0/669031240 >> v2:172.21.15.88:6800/30176 conn(0x7f66b0038c70 0x7f66b003b080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:33:04.522 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.418+0000 7f66c8ff9700 1 -- 172.21.15.88:0/669031240 --> v2:172.21.15.88:6800/30176 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f66b003b6f0 con 0x7f66b0038c70 2023-09-30T01:33:04.522 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.418+0000 7f66c8ff9700 1 -- 172.21.15.88:0/669031240 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_get_version_reply(handle=1 version=15) v2 ==== 24+0+0 (secure 0 0 0) 0x7f66c00253f0 con 0x7f66cc097e40 2023-09-30T01:33:04.522 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.418+0000 7f66c8ff9700 10 monclient: handle_get_version_reply finishing 1 version 15 2023-09-30T01:33:04.522 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.418+0000 7f66d1490700 1 --2- 172.21.15.88:0/669031240 >> v2:172.21.15.88:6800/30176 conn(0x7f66b0038c70 0x7f66b003b080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:33:04.522 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.418+0000 7f66d1490700 10 monclient: get_auth_request con 0x7f66b0038c70 auth_method 0 2023-09-30T01:33:04.523 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.418+0000 7f036b7fe700 1 --2- 172.21.15.88:0/3802989775 >> v2:172.21.15.107:6800/30275 conn(0x7f0358038d70 0x7f035803b180 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:33:04.523 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.418+0000 7f66d0c8f700 1 --2- 172.21.15.88:0/669031240 >> v2:172.21.15.107:6812/29926 conn(0x7f66b00331d0 0x7f66b0035680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:33:04.523 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.418+0000 7f66d0c8f700 10 monclient: get_auth_request con 0x7f66b00331d0 auth_method 0 2023-09-30T01:33:04.525 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.418+0000 7f85c759e700 10 monclient: _reopen_session rank -1 2023-09-30T01:33:04.525 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.418+0000 7f85c759e700 10 monclient: _add_conns ranks=[8,6,5,7,1,3,2,0,4] 2023-09-30T01:33:04.525 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.418+0000 7f85c759e700 1 --2- >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f85c809cf30 0x7f85c806aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:33:04.525 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.418+0000 7f85c759e700 10 monclient(hunting): picked mon.noname-e con 0x7f85c809cf30 addr [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] 2023-09-30T01:33:04.525 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.418+0000 7f85c759e700 1 --2- >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7f85c809da10 0x7f85c81358b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:33:04.525 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.418+0000 7f85c759e700 10 monclient(hunting): picked mon.noname-d con 0x7f85c809da10 addr [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] 2023-09-30T01:33:04.525 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.418+0000 7f85c759e700 1 --2- >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7f85c81366e0 0x7f85c806ee30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:33:04.525 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.418+0000 7f85c759e700 10 monclient(hunting): picked mon.noname-h con 0x7f85c81366e0 addr [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] 2023-09-30T01:33:04.525 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.418+0000 7f85c759e700 10 monclient(hunting): start opening mon connection 2023-09-30T01:33:04.526 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.418+0000 7f85c759e700 1 -- --> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] -- mon_getmap magic: 0 v1 -- 0x7f85c806d060 con 0x7f85c81366e0 2023-09-30T01:33:04.526 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.418+0000 7f85c759e700 10 monclient(hunting): start opening mon connection 2023-09-30T01:33:04.526 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.418+0000 7f85c759e700 1 -- --> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] -- mon_getmap magic: 0 v1 -- 0x7f85c806d340 con 0x7f85c809da10 2023-09-30T01:33:04.526 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.418+0000 7f85c759e700 10 monclient(hunting): start opening mon connection 2023-09-30T01:33:04.526 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.418+0000 7f85c759e700 1 -- --> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] -- mon_getmap magic: 0 v1 -- 0x7f85c806d620 con 0x7f85c809cf30 2023-09-30T01:33:04.526 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.418+0000 7f85c759e700 10 monclient(hunting): _renew_subs 2023-09-30T01:33:04.526 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.418+0000 7f85c759e700 10 monclient(hunting): authenticate will time out at 856.459718s 2023-09-30T01:33:04.526 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.418+0000 7f85c659c700 1 --2- >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f85c809cf30 0x7f85c806aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:33:04.526 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.418+0000 7f85c659c700 1 --2- >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f85c809cf30 0x7f85c806aba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3304/0 says I am v2:172.21.15.88:36696/0 (socket says 172.21.15.88:36696) 2023-09-30T01:33:04.526 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.418+0000 7f85c659c700 1 -- 172.21.15.88:0/2443232518 learned_addr learned my addr 172.21.15.88:0/2443232518 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-09-30T01:33:04.526 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.418+0000 7f85c659c700 10 monclient(hunting): get_auth_request con 0x7f85c809cf30 auth_method 0 2023-09-30T01:33:04.526 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.418+0000 7f85c659c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:33:04.527 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.418+0000 7f85c659c700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:33:04.527 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.418+0000 7f85c659c700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:33:04.527 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.422+0000 7f85c6d9d700 1 --2- 172.21.15.88:0/2443232518 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7f85c81366e0 0x7f85c806ee30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:33:04.527 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.422+0000 7f85c659c700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:33:04.527 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.422+0000 7f85c659c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:33:04.527 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.422+0000 7f85c659c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:33:04.527 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.422+0000 7f85c5d9b700 1 --2- 172.21.15.88:0/2443232518 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7f85c809da10 0x7f85c81358b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:33:04.527 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.422+0000 7f85c5d9b700 10 monclient(hunting): get_auth_request con 0x7f85c809da10 auth_method 0 2023-09-30T01:33:04.527 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.422+0000 7f85c5d9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:33:04.527 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.422+0000 7f85c5d9b700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:33:04.527 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.422+0000 7f85c5d9b700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:33:04.527 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.422+0000 7f85c6d9d700 10 monclient(hunting): get_auth_request con 0x7f85c81366e0 auth_method 0 2023-09-30T01:33:04.528 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.422+0000 7f85c6d9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:33:04.528 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.422+0000 7f85c6d9d700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:33:04.528 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.422+0000 7f85c6d9d700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:33:04.528 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.422+0000 7f85c5d9b700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:33:04.528 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.422+0000 7f85c5d9b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:33:04.528 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.422+0000 7f85c5d9b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:33:04.528 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.422+0000 7f85c659c700 10 monclient(hunting): handle_auth_done global_id 4283 payload 824 2023-09-30T01:33:04.528 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.422+0000 7f85c659c700 1 -- 172.21.15.88:0/2443232518 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7f85c809da10 msgr2=0x7f85c81358b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:33:04.528 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.422+0000 7f85c659c700 1 --2- 172.21.15.88:0/2443232518 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7f85c809da10 0x7f85c81358b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:04.528 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.422+0000 7f85c659c700 1 -- 172.21.15.88:0/2443232518 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7f85c81366e0 msgr2=0x7f85c806ee30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:33:04.528 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.422+0000 7f85c659c700 1 --2- 172.21.15.88:0/2443232518 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7f85c81366e0 0x7f85c806ee30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:04.528 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.422+0000 7f85c659c700 10 monclient: _finish_hunting 0 2023-09-30T01:33:04.529 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.422+0000 7f85c659c700 1 monclient: found mon.noname-e 2023-09-30T01:33:04.529 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.422+0000 7f85c659c700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:33:04.529 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.422+0000 7f85c659c700 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.88:3304/0 2023-09-30T01:33:04.529 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.422+0000 7f85c659c700 1 -- 172.21.15.88:0/2443232518 --> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85c0009a20 con 0x7f85c809cf30 2023-09-30T01:33:04.529 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.422+0000 7f66d1490700 1 --2- 172.21.15.88:0/669031240 >> v2:172.21.15.88:6800/30176 conn(0x7f66b0038c70 0x7f66b003b080 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:33:04.530 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.422+0000 7f85c659c700 1 --2- 172.21.15.88:0/2443232518 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f85c809cf30 0x7f85c806aba0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f85b800c7c0 tx=0x7f85b8007f30 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:33:04.530 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.422+0000 7f85b67fc700 1 -- 172.21.15.88:0/2443232518 <== mon.8 v2:172.21.15.88:3304/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f85b80070d0 con 0x7f85c809cf30 2023-09-30T01:33:04.530 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.422+0000 7f85b67fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:33:04.530 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.422+0000 7f85b67fc700 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2023-09-30T01:33:04.531 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.422+0000 7f85b67fc700 10 monclient: dump: 2023-09-30T01:33:04.531 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:33:04.531 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:33:04.531 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:04.531 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:04.531 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:33:04.531 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:33:04.531 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:33:04.531 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:33:04.531 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:33:04.531 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:33:04.532 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:33:04.532 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:33:04.532 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:33:04.532 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:33:04.532 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:33:04.532 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:33:04.532 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.422+0000 7f85b67fc700 10 monclient: _finish_auth 0 2023-09-30T01:33:04.532 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.422+0000 7f85b67fc700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:33:04.532 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.422+0000 7f85c759e700 5 monclient: authenticate success, global_id 4283 2023-09-30T01:33:04.532 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.422+0000 7f85b67fc700 1 -- 172.21.15.88:0/2443232518 <== mon.8 v2:172.21.15.88:3304/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85b8007250 con 0x7f85c809cf30 2023-09-30T01:33:04.532 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.422+0000 7f85c759e700 10 monclient: _renew_subs 2023-09-30T01:33:04.532 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.422+0000 7f85c759e700 10 monclient: _send_mon_message to mon.e at v2:172.21.15.88:3304/0 2023-09-30T01:33:04.532 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.422+0000 7f85c759e700 1 -- 172.21.15.88:0/2443232518 --> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f85c806b5b0 con 0x7f85c809cf30 2023-09-30T01:33:04.533 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.422+0000 7f85b67fc700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:33:04.533 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.422+0000 7f85b67fc700 1 -- 172.21.15.88:0/2443232518 <== mon.8 v2:172.21.15.88:3304/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f85b8011360 con 0x7f85c809cf30 2023-09-30T01:33:04.533 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.422+0000 7f85c759e700 10 monclient: _renew_subs 2023-09-30T01:33:04.533 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.422+0000 7f85c759e700 10 monclient: _send_mon_message to mon.e at v2:172.21.15.88:3304/0 2023-09-30T01:33:04.533 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.422+0000 7f85c759e700 1 -- 172.21.15.88:0/2443232518 --> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f85c806c310 con 0x7f85c809cf30 2023-09-30T01:33:04.533 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.422+0000 7f85b67fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:33:04.533 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.422+0000 7f85b67fc700 10 monclient: got monmap 1 from mon.e (according to old e1) 2023-09-30T01:33:04.533 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.422+0000 7f85b67fc700 10 monclient: dump: 2023-09-30T01:33:04.533 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:33:04.533 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:33:04.533 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:04.534 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:04.534 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:33:04.534 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:33:04.534 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:33:04.534 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:33:04.534 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:33:04.534 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:33:04.534 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:33:04.534 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:33:04.534 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:33:04.534 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:33:04.534 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:33:04.534 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:33:04.535 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.422+0000 7f0353fff700 1 -- 172.21.15.88:0/3802989775 <== osd.5 v2:172.21.15.107:6800/30275 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23999 (crc 0 0 0) 0x7f035803b7f0 con 0x7f0358038d70 2023-09-30T01:33:04.535 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.422+0000 7f66c8ff9700 1 -- 172.21.15.88:0/669031240 <== osd.2 v2:172.21.15.88:6800/30176 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23999 (crc 0 0 0) 0x7f66b003b6f0 con 0x7f66b0038c70 2023-09-30T01:33:04.536 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.422+0000 7f85b7fff700 4 set_mon_vals no callback set 2023-09-30T01:33:04.536 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.422+0000 7f85b67fc700 1 -- 172.21.15.88:0/2443232518 <== mon.8 v2:172.21.15.88:3304/0 4 ==== mgrmap(e 4) v1 ==== 40088+0+0 (secure 0 0 0) 0x7f85b8005b30 con 0x7f85c809cf30 2023-09-30T01:33:04.536 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.422+0000 7f85b67fc700 1 --2- 172.21.15.88:0/2443232518 >> v2:172.21.15.107:6812/29926 conn(0x7f85a80332d0 0x7f85a8035780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:33:04.537 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.422+0000 7f85b67fc700 1 -- 172.21.15.88:0/2443232518 <== mon.8 v2:172.21.15.88:3304/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 3502+0+0 (secure 0 0 0) 0x7f85b8042380 con 0x7f85c809cf30 2023-09-30T01:33:04.537 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.422+0000 7f85c5d9b700 1 --2- 172.21.15.88:0/2443232518 >> v2:172.21.15.107:6812/29926 conn(0x7f85a80332d0 0x7f85a8035780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:33:04.537 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.422+0000 7f85c5d9b700 10 monclient: get_auth_request con 0x7f85a80332d0 auth_method 0 2023-09-30T01:33:04.537 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.422+0000 7f85c759e700 1 --2- 172.21.15.88:0/2443232518 >> v2:172.21.15.88:6804/30175 conn(0x7f85ac0015d0 0x7f85ac003a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:33:04.537 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.422+0000 7f85c759e700 1 -- 172.21.15.88:0/2443232518 --> v2:172.21.15.88:6804/30175 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f85ac006a80 con 0x7f85ac0015d0 2023-09-30T01:33:04.537 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.422+0000 7f85c6d9d700 1 --2- 172.21.15.88:0/2443232518 >> v2:172.21.15.88:6804/30175 conn(0x7f85ac0015d0 0x7f85ac003a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:33:04.537 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.422+0000 7f85c6d9d700 10 monclient: get_auth_request con 0x7f85ac0015d0 auth_method 0 2023-09-30T01:33:04.537 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.422+0000 7f85c6d9d700 1 --2- 172.21.15.88:0/2443232518 >> v2:172.21.15.88:6804/30175 conn(0x7f85ac0015d0 0x7f85ac003a80 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:33:04.537 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.422+0000 7f85c5d9b700 1 --2- 172.21.15.88:0/2443232518 >> v2:172.21.15.107:6812/29926 conn(0x7f85a80332d0 0x7f85a8035780 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f85c8136100 tx=0x7f85bc006f90 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:33:04.537 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.426+0000 7f66d0c8f700 1 --2- 172.21.15.88:0/669031240 >> v2:172.21.15.107:6812/29926 conn(0x7f66b00331d0 0x7f66b0035680 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f66bc006aa0 tx=0x7f66bc008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:33:04.538 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.426+0000 7f85b67fc700 1 -- 172.21.15.88:0/2443232518 <== osd.1 v2:172.21.15.88:6804/30175 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23999 (crc 0 0 0) 0x7f85ac006a80 con 0x7f85ac0015d0 2023-09-30T01:33:04.538 INFO:teuthology.orchestra.run.smithi088.stdout:47244640258 2023-09-30T01:33:04.539 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2023-09-30T01:33:04.539 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.434+0000 7f66d1c91700 1 -- 172.21.15.88:0/669031240 --> v2:172.21.15.88:6800/30176 -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f66a8002bd0 con 0x7f66b0038c70 2023-09-30T01:33:04.540 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.438+0000 7f66c8ff9700 1 -- 172.21.15.88:0/669031240 <== osd.2 v2:172.21.15.88:6800/30176 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f66a8002bd0 con 0x7f66b0038c70 2023-09-30T01:33:04.540 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.438+0000 7f66d1c91700 1 -- 172.21.15.88:0/669031240 >> v2:172.21.15.88:6800/30176 conn(0x7f66b0038c70 msgr2=0x7f66b003b080 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:33:04.541 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.438+0000 7f66d1c91700 1 --2- 172.21.15.88:0/669031240 >> v2:172.21.15.88:6800/30176 conn(0x7f66b0038c70 0x7f66b003b080 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:04.541 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.438+0000 7f66d1c91700 1 -- 172.21.15.88:0/669031240 >> v2:172.21.15.107:6812/29926 conn(0x7f66b00331d0 msgr2=0x7f66b0035680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:33:04.541 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.438+0000 7f66d1c91700 1 --2- 172.21.15.88:0/669031240 >> v2:172.21.15.107:6812/29926 conn(0x7f66b00331d0 0x7f66b0035680 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f66bc006aa0 tx=0x7f66bc008040 comp rx=0 tx=0).stop 2023-09-30T01:33:04.541 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.438+0000 7f66d1c91700 10 monclient: shutdown 2023-09-30T01:33:04.541 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.438+0000 7f66d1c91700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:33:04.541 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.438+0000 7f66d1c91700 1 -- 172.21.15.88:0/669031240 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f66cc097e40 msgr2=0x7f66cc142e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:33:04.541 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.438+0000 7f66d1c91700 1 --2- 172.21.15.88:0/669031240 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f66cc097e40 0x7f66cc142e00 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f66cc0987b0 tx=0x7f66c0017000 comp rx=0 tx=0).stop 2023-09-30T01:33:04.541 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.438+0000 7f66d1490700 1 -- 172.21.15.88:0/669031240 reap_dead start 2023-09-30T01:33:04.541 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.438+0000 7f66d1c91700 1 -- 172.21.15.88:0/669031240 shutdown_connections 2023-09-30T01:33:04.541 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.438+0000 7f66d1c91700 1 -- 172.21.15.88:0/669031240 >> 172.21.15.88:0/669031240 conn(0x7f66cc09fe70 msgr2=0x7f66cc0a3b90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:33:04.541 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.438+0000 7f66d1c91700 1 -- 172.21.15.88:0/669031240 shutdown_connections 2023-09-30T01:33:04.541 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.438+0000 7f66d1c91700 1 -- 172.21.15.88:0/669031240 wait complete. 2023-09-30T01:33:04.542 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.438+0000 7f037147f700 1 -- 172.21.15.88:0/3802989775 --> v2:172.21.15.107:6800/30275 -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f036c09be60 con 0x7f0358038d70 2023-09-30T01:33:04.542 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.438+0000 7f0353fff700 1 -- 172.21.15.88:0/3802989775 <== osd.5 v2:172.21.15.107:6800/30275 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f036c09be60 con 0x7f0358038d70 2023-09-30T01:33:04.543 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.438+0000 7f037147f700 1 -- 172.21.15.88:0/3802989775 >> v2:172.21.15.107:6800/30275 conn(0x7f0358038d70 msgr2=0x7f035803b180 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:33:04.543 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.438+0000 7f037147f700 1 --2- 172.21.15.88:0/3802989775 >> v2:172.21.15.107:6800/30275 conn(0x7f0358038d70 0x7f035803b180 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:04.543 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.438+0000 7f037147f700 1 -- 172.21.15.88:0/3802989775 >> v2:172.21.15.107:6812/29926 conn(0x7f03580332d0 msgr2=0x7f0358035780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:33:04.543 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.438+0000 7f037147f700 1 --2- 172.21.15.88:0/3802989775 >> v2:172.21.15.107:6812/29926 conn(0x7f03580332d0 0x7f0358035780 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f036000ac00 tx=0x7f0360006f90 comp rx=0 tx=0).stop 2023-09-30T01:33:04.543 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.438+0000 7f037147f700 10 monclient: shutdown 2023-09-30T01:33:04.544 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.438+0000 7f037147f700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:33:04.544 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.438+0000 7f037147f700 1 -- 172.21.15.88:0/3802989775 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f036c08d5e0 msgr2=0x7f036c1467d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:33:04.544 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.438+0000 7f037147f700 1 --2- 172.21.15.88:0/3802989775 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f036c08d5e0 0x7f036c1467d0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f0364014000 tx=0x7f0364017000 comp rx=0 tx=0).stop 2023-09-30T01:33:04.544 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.438+0000 7f0370c7e700 1 -- 172.21.15.88:0/3802989775 reap_dead start 2023-09-30T01:33:04.544 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.438+0000 7f037147f700 1 -- 172.21.15.88:0/3802989775 shutdown_connections 2023-09-30T01:33:04.544 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.438+0000 7f037147f700 1 -- 172.21.15.88:0/3802989775 >> 172.21.15.88:0/3802989775 conn(0x7f036c0a2540 msgr2=0x7f036c096a30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:33:04.544 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.438+0000 7f037147f700 1 -- 172.21.15.88:0/3802989775 shutdown_connections 2023-09-30T01:33:04.544 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.438+0000 7f037147f700 1 -- 172.21.15.88:0/3802989775 wait complete. 2023-09-30T01:33:04.544 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.442+0000 7fca12e64700 1 -- 172.21.15.88:0/3275358350 --> v2:172.21.15.88:6808/30177 -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fc9ec002bd0 con 0x7fc9f0038ca0 2023-09-30T01:33:04.545 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.442+0000 7fca08ff9700 1 -- 172.21.15.88:0/3275358350 <== osd.0 v2:172.21.15.88:6808/30177 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fc9ec002bd0 con 0x7fc9f0038ca0 2023-09-30T01:33:04.545 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.442+0000 7fca12e64700 1 -- 172.21.15.88:0/3275358350 >> v2:172.21.15.88:6808/30177 conn(0x7fc9f0038ca0 msgr2=0x7fc9f003b0b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:33:04.546 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.442+0000 7fca12e64700 1 --2- 172.21.15.88:0/3275358350 >> v2:172.21.15.88:6808/30177 conn(0x7fc9f0038ca0 0x7fc9f003b0b0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:04.546 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.442+0000 7fca12e64700 1 -- 172.21.15.88:0/3275358350 >> v2:172.21.15.107:6812/29926 conn(0x7fc9f0033200 msgr2=0x7fc9f00356b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:33:04.546 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.442+0000 7fca12e64700 1 --2- 172.21.15.88:0/3275358350 >> v2:172.21.15.107:6812/29926 conn(0x7fc9f0033200 0x7fc9f00356b0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fca0c11d860 tx=0x7fc9f80091a0 comp rx=0 tx=0).stop 2023-09-30T01:33:04.546 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.442+0000 7fca12e64700 10 monclient: shutdown 2023-09-30T01:33:04.546 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.442+0000 7fca12e64700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:33:04.546 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.442+0000 7fca12e64700 1 -- 172.21.15.88:0/3275358350 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fca0c0a3270 msgr2=0x7fca0c142880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:33:04.546 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.442+0000 7fca12e64700 1 --2- 172.21.15.88:0/3275358350 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fca0c0a3270 0x7fca0c142880 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fca00013000 tx=0x7fca00017000 comp rx=0 tx=0).stop 2023-09-30T01:33:04.546 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.442+0000 7fca11401700 1 -- 172.21.15.88:0/3275358350 reap_dead start 2023-09-30T01:33:04.546 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.442+0000 7fca12e64700 1 -- 172.21.15.88:0/3275358350 shutdown_connections 2023-09-30T01:33:04.546 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.442+0000 7fca12e64700 1 -- 172.21.15.88:0/3275358350 >> 172.21.15.88:0/3275358350 conn(0x7fca0c0a67f0 msgr2=0x7fca0c0a20e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:33:04.546 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.442+0000 7fca12e64700 1 -- 172.21.15.88:0/3275358350 shutdown_connections 2023-09-30T01:33:04.547 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.442+0000 7fca12e64700 1 -- 172.21.15.88:0/3275358350 wait complete. 2023-09-30T01:33:04.547 INFO:teuthology.orchestra.run.smithi088.stdout:47244640260 2023-09-30T01:33:04.547 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2023-09-30T01:33:04.548 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.450+0000 7f85c759e700 1 -- 172.21.15.88:0/2443232518 --> v2:172.21.15.88:6804/30175 -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f85ac005b60 con 0x7f85ac0015d0 2023-09-30T01:33:04.548 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.450+0000 7f85b67fc700 1 -- 172.21.15.88:0/2443232518 <== osd.1 v2:172.21.15.88:6804/30175 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f85ac005b60 con 0x7f85ac0015d0 2023-09-30T01:33:04.549 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.450+0000 7f85c759e700 1 -- 172.21.15.88:0/2443232518 >> v2:172.21.15.88:6804/30175 conn(0x7f85ac0015d0 msgr2=0x7f85ac003a80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:33:04.549 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.450+0000 7f85c759e700 1 --2- 172.21.15.88:0/2443232518 >> v2:172.21.15.88:6804/30175 conn(0x7f85ac0015d0 0x7f85ac003a80 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:04.549 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.450+0000 7f85c759e700 1 -- 172.21.15.88:0/2443232518 >> v2:172.21.15.107:6812/29926 conn(0x7f85a80332d0 msgr2=0x7f85a8035780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:33:04.549 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.450+0000 7f85c759e700 1 --2- 172.21.15.88:0/2443232518 >> v2:172.21.15.107:6812/29926 conn(0x7f85a80332d0 0x7f85a8035780 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f85c8136100 tx=0x7f85bc006f90 comp rx=0 tx=0).stop 2023-09-30T01:33:04.549 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.450+0000 7f85c759e700 10 monclient: shutdown 2023-09-30T01:33:04.549 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.450+0000 7f85c759e700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:33:04.549 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.450+0000 7f85c759e700 1 -- 172.21.15.88:0/2443232518 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f85c809cf30 msgr2=0x7f85c806aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:33:04.549 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.450+0000 7f85c759e700 1 --2- 172.21.15.88:0/2443232518 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f85c809cf30 0x7f85c806aba0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f85b800c7c0 tx=0x7f85b8007f30 comp rx=0 tx=0).stop 2023-09-30T01:33:04.550 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.450+0000 7f85c6d9d700 1 -- 172.21.15.88:0/2443232518 reap_dead start 2023-09-30T01:33:04.550 INFO:teuthology.orchestra.run.smithi088.stdout:47244640258 2023-09-30T01:33:04.551 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2023-09-30T01:33:04.551 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.450+0000 7f85c759e700 1 -- 172.21.15.88:0/2443232518 shutdown_connections 2023-09-30T01:33:04.551 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.450+0000 7f85c759e700 1 -- 172.21.15.88:0/2443232518 >> 172.21.15.88:0/2443232518 conn(0x7f85c80a2540 msgr2=0x7f85c8131820 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:33:04.551 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.450+0000 7f85c759e700 1 -- 172.21.15.88:0/2443232518 shutdown_connections 2023-09-30T01:33:04.552 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.450+0000 7f85c759e700 1 -- 172.21.15.88:0/2443232518 wait complete. 2023-09-30T01:33:04.552 INFO:teuthology.orchestra.run.smithi088.stdout:47244640260 2023-09-30T01:33:04.552 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2023-09-30T01:33:04.553 INFO:teuthology.orchestra.run.smithi088.stdout:47244640259 2023-09-30T01:33:04.554 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-09-30T01:33:04.712 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.710+0000 7f00fcb04700 10 monclient: get_monmap_and_config 2023-09-30T01:33:04.713 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.710+0000 7f00fcb04700 10 monclient: build_initial_monmap 2023-09-30T01:33:04.713 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.710+0000 7f00fcb04700 10 monclient: monmap: 2023-09-30T01:33:04.713 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2023-09-30T01:33:04.713 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:33:04.713 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:33:04.713326+0000 2023-09-30T01:33:04.713 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:33:04.713326+0000 2023-09-30T01:33:04.713 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2023-09-30T01:33:04.713 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:33:04.713 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a-legacy 2023-09-30T01:33:04.713 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.107:6789/0 mon.noname-f-legacy 2023-09-30T01:33:04.713 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.noname-b 2023-09-30T01:33:04.714 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.noname-g 2023-09-30T01:33:04.714 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.noname-c 2023-09-30T01:33:04.714 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.noname-h 2023-09-30T01:33:04.714 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.noname-d 2023-09-30T01:33:04.714 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.noname-i 2023-09-30T01:33:04.714 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.noname-e 2023-09-30T01:33:04.714 INFO:teuthology.orchestra.run.smithi088.stderr:9: v2:172.21.15.88:3300/0 mon.noname-a 2023-09-30T01:33:04.714 INFO:teuthology.orchestra.run.smithi088.stderr:10: v2:172.21.15.107:3300/0 mon.noname-f 2023-09-30T01:33:04.714 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:33:04.719 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.710+0000 7f00fcb04700 1 Processor -- start 2023-09-30T01:33:04.719 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.710+0000 7f00fcb04700 1 -- start start 2023-09-30T01:33:04.719 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.710+0000 7f00fcb04700 10 monclient: init 2023-09-30T01:33:04.720 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.710+0000 7f00fcb04700 10 monclient: _reopen_session rank -1 2023-09-30T01:33:04.720 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.710+0000 7f00fcb04700 10 monclient: _add_conns ranks=[5,10,4,0,8,6,3,9,1,7,2] 2023-09-30T01:33:04.720 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.710+0000 7f00fcb04700 1 --2- >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7f00f8062c70 0x7f00f8133850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:33:04.720 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.710+0000 7f00fcb04700 10 monclient(hunting): picked mon.noname-h con 0x7f00f8062c70 addr [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] 2023-09-30T01:33:04.720 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.710+0000 7f00fcb04700 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f00f8133d90 0x7f00f80998b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:33:04.720 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.710+0000 7f00fcb04700 10 monclient(hunting): picked mon.noname-f con 0x7f00f8133d90 addr v2:172.21.15.107:3300/0 2023-09-30T01:33:04.720 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.710+0000 7f00fcb04700 1 --2- >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f00f8099df0 0x7f00f812feb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:33:04.720 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.710+0000 7f00fcb04700 10 monclient(hunting): picked mon.noname-c con 0x7f00f8099df0 addr [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] 2023-09-30T01:33:04.720 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.710+0000 7f00fcb04700 10 monclient(hunting): start opening mon connection 2023-09-30T01:33:04.720 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.710+0000 7f00fcb04700 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f00f809a270 con 0x7f00f8133d90 2023-09-30T01:33:04.720 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.710+0000 7f00fcb04700 10 monclient(hunting): start opening mon connection 2023-09-30T01:33:04.721 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.710+0000 7f00fcb04700 1 -- --> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] -- mon_getmap magic: 0 v1 -- 0x7f00f8063040 con 0x7f00f8099df0 2023-09-30T01:33:04.721 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.710+0000 7f00fcb04700 10 monclient(hunting): start opening mon connection 2023-09-30T01:33:04.721 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.710+0000 7f00fcb04700 1 -- --> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] -- mon_getmap magic: 0 v1 -- 0x7f00f8097dc0 con 0x7f00f8062c70 2023-09-30T01:33:04.721 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.710+0000 7f00fcb04700 10 monclient(hunting): _renew_subs 2023-09-30T01:33:04.721 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.710+0000 7f00fcb04700 10 monclient(hunting): authenticate will time out at 856.750886s 2023-09-30T01:33:04.721 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.710+0000 7f00f7fff700 1 --2- >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f00f8099df0 0x7f00f812feb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:33:04.721 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.710+0000 7f00f7fff700 1 --2- >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f00f8099df0 0x7f00f812feb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3302/0 says I am v2:172.21.15.88:50428/0 (socket says 172.21.15.88:50428) 2023-09-30T01:33:04.721 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.710+0000 7f00f7fff700 1 -- 172.21.15.88:0/1512144847 learned_addr learned my addr 172.21.15.88:0/1512144847 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-09-30T01:33:04.721 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.710+0000 7f00f7fff700 10 monclient(hunting): get_auth_request con 0x7f00f8099df0 auth_method 0 2023-09-30T01:33:04.721 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.710+0000 7f00f7fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:33:04.721 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.710+0000 7f00f7fff700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:33:04.722 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.710+0000 7f00f7fff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:33:04.722 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.710+0000 7f00f77fe700 1 --2- 172.21.15.88:0/1512144847 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7f00f8062c70 0x7f00f8133850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:33:04.722 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.710+0000 7f00f77fe700 10 monclient(hunting): get_auth_request con 0x7f00f8062c70 auth_method 0 2023-09-30T01:33:04.722 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.710+0000 7f00f77fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:33:04.722 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.710+0000 7f00f77fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:33:04.722 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.710+0000 7f00f77fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:33:04.722 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.710+0000 7f00f6ffd700 1 --2- 172.21.15.88:0/1512144847 >> v2:172.21.15.107:3300/0 conn(0x7f00f8133d90 0x7f00f80998b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:33:04.722 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.710+0000 7f00f6ffd700 10 monclient(hunting): get_auth_request con 0x7f00f8133d90 auth_method 0 2023-09-30T01:33:04.722 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.710+0000 7f00f6ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:33:04.722 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.710+0000 7f00f6ffd700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:33:04.722 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.710+0000 7f00f6ffd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:33:04.722 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.710+0000 7f00f7fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:33:04.723 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.710+0000 7f00f7fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:33:04.723 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.710+0000 7f00f7fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:33:04.723 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.710+0000 7f00f77fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:33:04.723 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.710+0000 7f00f77fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:33:04.723 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.710+0000 7f00f77fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:33:04.723 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.710+0000 7f00f6ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:33:04.723 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.710+0000 7f00f6ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:33:04.723 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.710+0000 7f00f6ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:33:04.723 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.710+0000 7f00f7fff700 10 monclient(hunting): handle_auth_done global_id 4288 payload 274 2023-09-30T01:33:04.723 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.710+0000 7f00f7fff700 1 -- 172.21.15.88:0/1512144847 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7f00f8062c70 msgr2=0x7f00f8133850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:33:04.723 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.710+0000 7f00f7fff700 1 --2- 172.21.15.88:0/1512144847 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7f00f8062c70 0x7f00f8133850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:04.723 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.710+0000 7f00f7fff700 1 -- 172.21.15.88:0/1512144847 >> v2:172.21.15.107:3300/0 conn(0x7f00f8133d90 msgr2=0x7f00f80998b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:33:04.724 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.710+0000 7f00f7fff700 1 --2- 172.21.15.88:0/1512144847 >> v2:172.21.15.107:3300/0 conn(0x7f00f8133d90 0x7f00f80998b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:04.724 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.710+0000 7f00f7fff700 10 monclient: _finish_hunting 0 2023-09-30T01:33:04.724 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.710+0000 7f00f7fff700 1 monclient: found mon.noname-c 2023-09-30T01:33:04.724 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.710+0000 7f00f7fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:33:04.724 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.710+0000 7f00f7fff700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.88:3302/0 2023-09-30T01:33:04.724 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.710+0000 7f00f7fff700 1 -- 172.21.15.88:0/1512144847 --> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00f804bb20 con 0x7f00f8099df0 2023-09-30T01:33:04.724 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.714+0000 7f00f7fff700 1 --2- 172.21.15.88:0/1512144847 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f00f8099df0 0x7f00f812feb0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f00f0009f20 tx=0x7f00f0009840 comp rx=0 tx=0).ready entity=mon.4 client_cookie=4eaec231abe4ce1d server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:33:04.724 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.714+0000 7f00f67fc700 1 -- 172.21.15.88:0/1512144847 <== mon.4 v2:172.21.15.88:3302/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f00f000e070 con 0x7f00f8099df0 2023-09-30T01:33:04.724 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.714+0000 7f00f67fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:33:04.724 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.714+0000 7f00f67fc700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T01:33:04.724 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.714+0000 7f00f67fc700 10 monclient: dump: 2023-09-30T01:33:04.724 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:33:04.725 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:33:04.725 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:04.725 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:04.725 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:33:04.725 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:33:04.725 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:33:04.725 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:33:04.725 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:33:04.725 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:33:04.725 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:33:04.725 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:33:04.725 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:33:04.725 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:33:04.726 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:33:04.726 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:33:04.726 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.714+0000 7f00f67fc700 10 monclient: _finish_auth 0 2023-09-30T01:33:04.726 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.714+0000 7f00f67fc700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:33:04.726 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.714+0000 7f00f67fc700 1 -- 172.21.15.88:0/1512144847 <== mon.4 v2:172.21.15.88:3302/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f00f000bdf0 con 0x7f00f8099df0 2023-09-30T01:33:04.726 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.714+0000 7f00f67fc700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:33:04.726 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.714+0000 7f00f67fc700 1 -- 172.21.15.88:0/1512144847 <== mon.4 v2:172.21.15.88:3302/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f00f00092e0 con 0x7f00f8099df0 2023-09-30T01:33:04.726 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.714+0000 7f00f67fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:33:04.726 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.714+0000 7f00f67fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T01:33:04.726 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.714+0000 7f00f67fc700 10 monclient: dump: 2023-09-30T01:33:04.726 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:33:04.726 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:33:04.727 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:04.727 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:04.727 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:33:04.727 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:33:04.727 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:33:04.727 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:33:04.727 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:33:04.727 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:33:04.727 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:33:04.727 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:33:04.727 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:33:04.727 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:33:04.728 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:33:04.728 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:33:04.728 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.714+0000 7f00fcb04700 5 monclient: authenticate success, global_id 4288 2023-09-30T01:33:04.728 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.714+0000 7f00fcb04700 10 monclient: get_monmap_and_config success 2023-09-30T01:33:04.728 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.714+0000 7f00fcb04700 4 set_mon_vals no callback set 2023-09-30T01:33:04.728 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.714+0000 7f00fcb04700 10 monclient: shutdown 2023-09-30T01:33:04.728 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.714+0000 7f00fcb04700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:33:04.728 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.714+0000 7f00fcb04700 1 -- 172.21.15.88:0/1512144847 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f00f8099df0 msgr2=0x7f00f812feb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:33:04.728 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.714+0000 7f00fcb04700 1 --2- 172.21.15.88:0/1512144847 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f00f8099df0 0x7f00f812feb0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f00f0009f20 tx=0x7f00f0009840 comp rx=0 tx=0).stop 2023-09-30T01:33:04.728 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.714+0000 7f00fcb04700 1 -- 172.21.15.88:0/1512144847 shutdown_connections 2023-09-30T01:33:04.728 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.714+0000 7f00fcb04700 1 --2- 172.21.15.88:0/1512144847 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f00f8099df0 0x7f00f812feb0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:04.728 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.714+0000 7f00fcb04700 1 --2- 172.21.15.88:0/1512144847 >> v2:172.21.15.107:3300/0 conn(0x7f00f8133d90 0x7f00f80998b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:04.729 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.714+0000 7f00fcb04700 1 --2- 172.21.15.88:0/1512144847 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7f00f8062c70 0x7f00f8133850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:04.729 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.714+0000 7f00fcb04700 1 -- 172.21.15.88:0/1512144847 >> 172.21.15.88:0/1512144847 conn(0x7f00f809fe10 msgr2=0x7f00f80a0210 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:33:04.729 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.714+0000 7f00fcb04700 1 -- 172.21.15.88:0/1512144847 shutdown_connections 2023-09-30T01:33:04.729 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.714+0000 7f00fcb04700 1 -- 172.21.15.88:0/1512144847 wait complete. 2023-09-30T01:33:04.729 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.714+0000 7f00fcb04700 10 monclient: build_initial_monmap 2023-09-30T01:33:04.729 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.718+0000 7f00fcb04700 10 monclient: monmap: 2023-09-30T01:33:04.729 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2023-09-30T01:33:04.729 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:33:04.729 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 0.000000 2023-09-30T01:33:04.729 INFO:teuthology.orchestra.run.smithi088.stderr:created 0.000000 2023-09-30T01:33:04.729 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2023-09-30T01:33:04.729 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 1 2023-09-30T01:33:04.729 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.noname-a 2023-09-30T01:33:04.730 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.noname-f 2023-09-30T01:33:04.730 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.noname-b 2023-09-30T01:33:04.730 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.noname-g 2023-09-30T01:33:04.730 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.noname-c 2023-09-30T01:33:04.730 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.noname-h 2023-09-30T01:33:04.730 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.noname-d 2023-09-30T01:33:04.730 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.noname-i 2023-09-30T01:33:04.730 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.noname-e 2023-09-30T01:33:04.730 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:33:04.730 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.718+0000 7f00fcb04700 1 Processor -- start 2023-09-30T01:33:04.730 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.718+0000 7f00fcb04700 1 -- start start 2023-09-30T01:33:04.731 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.718+0000 7f00fcb04700 10 monclient: init 2023-09-30T01:33:04.734 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.718+0000 7f00fcb04700 10 monclient: _reopen_session rank -1 2023-09-30T01:33:04.734 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.718+0000 7f00fcb04700 10 monclient: _add_conns ranks=[1,2,0,3,5,7,8,6,4] 2023-09-30T01:33:04.735 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.718+0000 7f00fcb04700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f00f8062c70 0x7f00f8119a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:33:04.735 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.718+0000 7f00fcb04700 10 monclient(hunting): picked mon.noname-f con 0x7f00f8062c70 addr [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] 2023-09-30T01:33:04.735 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.718+0000 7f00fcb04700 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f00f8099df0 0x7f00f8119fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:33:04.735 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.718+0000 7f00fcb04700 10 monclient(hunting): picked mon.noname-b con 0x7f00f8099df0 addr [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] 2023-09-30T01:33:04.735 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.718+0000 7f00fcb04700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f00f8133d90 0x7f00f811a4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:33:04.735 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.718+0000 7f00fcb04700 10 monclient(hunting): picked mon.noname-a con 0x7f00f8133d90 addr [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] 2023-09-30T01:33:04.735 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.718+0000 7f00fcb04700 10 monclient(hunting): start opening mon connection 2023-09-30T01:33:04.735 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.718+0000 7f00fcb04700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00f812dc40 con 0x7f00f8133d90 2023-09-30T01:33:04.735 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.718+0000 7f00fcb04700 10 monclient(hunting): start opening mon connection 2023-09-30T01:33:04.735 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.718+0000 7f00fcb04700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00f811b760 con 0x7f00f8062c70 2023-09-30T01:33:04.735 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.718+0000 7f00fcb04700 10 monclient(hunting): start opening mon connection 2023-09-30T01:33:04.736 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.718+0000 7f00fcb04700 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00f811ca60 con 0x7f00f8099df0 2023-09-30T01:33:04.736 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.718+0000 7f00fcb04700 10 monclient(hunting): _renew_subs 2023-09-30T01:33:04.736 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.718+0000 7f00fcb04700 10 monclient(hunting): authenticate will time out at 856.757393s 2023-09-30T01:33:04.736 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.718+0000 7f00f7fff700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f00f8133d90 0x7f00f811a4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:33:04.736 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.718+0000 7f00f6ffd700 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f00f8099df0 0x7f00f8119fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:33:04.736 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.718+0000 7f00f6ffd700 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f00f8099df0 0x7f00f8119fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3301/0 says I am v2:172.21.15.88:52442/0 (socket says 172.21.15.88:52442) 2023-09-30T01:33:04.736 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.718+0000 7f00f6ffd700 1 -- 172.21.15.88:0/3746327561 learned_addr learned my addr 172.21.15.88:0/3746327561 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-09-30T01:33:04.736 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.718+0000 7f00f6ffd700 10 monclient(hunting): get_auth_request con 0x7f00f8099df0 auth_method 0 2023-09-30T01:33:04.736 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.718+0000 7f00f6ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:33:04.736 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.718+0000 7f00f6ffd700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:33:04.736 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.718+0000 7f00f6ffd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:33:04.737 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.718+0000 7f00f77fe700 1 --2- 172.21.15.88:0/3746327561 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f00f8062c70 0x7f00f8119a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:33:04.737 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.718+0000 7f00f77fe700 10 monclient(hunting): get_auth_request con 0x7f00f8062c70 auth_method 0 2023-09-30T01:33:04.737 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.718+0000 7f00f77fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:33:04.737 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.718+0000 7f00f77fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:33:04.737 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.718+0000 7f00f77fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:33:04.737 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.718+0000 7f00f6ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:33:04.737 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.718+0000 7f00f6ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:33:04.737 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.718+0000 7f00f6ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:33:04.737 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.718+0000 7f00f7fff700 10 monclient(hunting): get_auth_request con 0x7f00f8133d90 auth_method 0 2023-09-30T01:33:04.737 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.718+0000 7f00f7fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:33:04.737 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.718+0000 7f00f7fff700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:33:04.737 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.718+0000 7f00f7fff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:33:04.738 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.718+0000 7f00f77fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:33:04.738 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.718+0000 7f00f77fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:33:04.738 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.718+0000 7f00f77fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:33:04.738 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.718+0000 7f00f6ffd700 10 monclient(hunting): handle_auth_done global_id 4286 payload 824 2023-09-30T01:33:04.738 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.718+0000 7f00f6ffd700 1 -- 172.21.15.88:0/3746327561 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f00f8062c70 msgr2=0x7f00f8119a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:33:04.738 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.718+0000 7f00f6ffd700 1 --2- 172.21.15.88:0/3746327561 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f00f8062c70 0x7f00f8119a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:04.738 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.718+0000 7f00f6ffd700 1 -- 172.21.15.88:0/3746327561 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f00f8133d90 msgr2=0x7f00f811a4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:33:04.738 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.718+0000 7f00f6ffd700 1 --2- 172.21.15.88:0/3746327561 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f00f8133d90 0x7f00f811a4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:04.738 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.718+0000 7f00f6ffd700 10 monclient: _finish_hunting 0 2023-09-30T01:33:04.738 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.718+0000 7f00f6ffd700 1 monclient: found mon.noname-b 2023-09-30T01:33:04.738 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.718+0000 7f00f6ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:33:04.739 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.718+0000 7f00f6ffd700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.88:3301/0 2023-09-30T01:33:04.739 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.718+0000 7f00f6ffd700 1 -- 172.21.15.88:0/3746327561 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00f0009ae0 con 0x7f00f8099df0 2023-09-30T01:33:04.739 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.718+0000 7f00f7fff700 1 --2- 172.21.15.88:0/3746327561 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f00f8133d90 0x7f00f811a4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-30T01:33:04.739 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.718+0000 7f00f6ffd700 1 --2- 172.21.15.88:0/3746327561 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f00f8099df0 0x7f00f8119fa0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f00ec009fd0 tx=0x7f00ec0091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:33:04.739 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.718+0000 7f00df7fe700 1 -- 172.21.15.88:0/3746327561 <== mon.2 v2:172.21.15.88:3301/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f00ec013b00 con 0x7f00f8099df0 2023-09-30T01:33:04.739 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.718+0000 7f00df7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:33:04.739 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.718+0000 7f00df7fe700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T01:33:04.739 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.718+0000 7f00df7fe700 10 monclient: dump: 2023-09-30T01:33:04.739 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:33:04.739 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:33:04.739 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:04.739 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:04.739 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:33:04.740 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:33:04.740 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:33:04.740 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:33:04.740 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:33:04.740 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:33:04.740 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:33:04.740 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:33:04.740 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:33:04.740 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:33:04.740 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:33:04.740 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:33:04.740 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.718+0000 7f00df7fe700 10 monclient: _finish_auth 0 2023-09-30T01:33:04.741 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.718+0000 7f00df7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:33:04.741 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.718+0000 7f00fcb04700 5 monclient: authenticate success, global_id 4286 2023-09-30T01:33:04.741 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.718+0000 7f00df7fe700 1 -- 172.21.15.88:0/3746327561 <== mon.2 v2:172.21.15.88:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f00ec013c80 con 0x7f00f8099df0 2023-09-30T01:33:04.741 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.718+0000 7f00fcb04700 10 monclient: _renew_subs 2023-09-30T01:33:04.741 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.718+0000 7f00fcb04700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.88:3301/0 2023-09-30T01:33:04.741 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.718+0000 7f00fcb04700 1 -- 172.21.15.88:0/3746327561 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f00f809f630 con 0x7f00f8099df0 2023-09-30T01:33:04.741 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.718+0000 7f00fcb04700 10 monclient: _renew_subs 2023-09-30T01:33:04.741 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.718+0000 7f00fcb04700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.88:3301/0 2023-09-30T01:33:04.741 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.718+0000 7f00fcb04700 1 -- 172.21.15.88:0/3746327561 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f00f811cd40 con 0x7f00f8099df0 2023-09-30T01:33:04.741 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.718+0000 7f00df7fe700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:33:04.741 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.718+0000 7f00df7fe700 1 -- 172.21.15.88:0/3746327561 <== mon.2 v2:172.21.15.88:3301/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f00ec013df0 con 0x7f00f8099df0 2023-09-30T01:33:04.741 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.718+0000 7f00df7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:33:04.742 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.718+0000 7f00df7fe700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T01:33:04.742 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.718+0000 7f00f5ffb700 4 set_mon_vals no callback set 2023-09-30T01:33:04.742 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.718+0000 7f00df7fe700 10 monclient: dump: 2023-09-30T01:33:04.742 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:33:04.742 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:33:04.742 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:04.742 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:04.742 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:33:04.742 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:33:04.742 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:33:04.742 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:33:04.742 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:33:04.743 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:33:04.743 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:33:04.743 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:33:04.743 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:33:04.743 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:33:04.743 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:33:04.743 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:33:04.743 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.722+0000 7f00df7fe700 1 -- 172.21.15.88:0/3746327561 <== mon.2 v2:172.21.15.88:3301/0 4 ==== mgrmap(e 5) v1 ==== 40144+0+0 (secure 0 0 0) 0x7f00ec013680 con 0x7f00f8099df0 2023-09-30T01:33:04.743 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.722+0000 7f00df7fe700 1 --2- 172.21.15.88:0/3746327561 >> v2:172.21.15.107:6812/29926 conn(0x7f00e4033360 0x7f00e4035810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:33:04.743 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.722+0000 7f00df7fe700 1 -- 172.21.15.88:0/3746327561 <== mon.2 v2:172.21.15.88:3301/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 3502+0+0 (secure 0 0 0) 0x7f00ec043800 con 0x7f00f8099df0 2023-09-30T01:33:04.743 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.722+0000 7f00f77fe700 1 --2- 172.21.15.88:0/3746327561 >> v2:172.21.15.107:6812/29926 conn(0x7f00e4033360 0x7f00e4035810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:33:04.743 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.722+0000 7f00f77fe700 10 monclient: get_auth_request con 0x7f00e4033360 auth_method 0 2023-09-30T01:33:04.744 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.722+0000 7f00f77fe700 1 --2- 172.21.15.88:0/3746327561 >> v2:172.21.15.107:6812/29926 conn(0x7f00e4033360 0x7f00e4035810 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f00e8009db0 tx=0x7f00e80091a0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:33:04.744 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.722+0000 7f00dd7fa700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:33:04.744 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.722+0000 7f00dd7fa700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:33:04.744 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.722+0000 7f00dd7fa700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.88:3301/0 2023-09-30T01:33:04.744 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.722+0000 7f00dd7fa700 1 -- 172.21.15.88:0/3746327561 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f00f811d120 con 0x7f00f8099df0 2023-09-30T01:33:04.747 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.718+0000 7f1c6875d700 10 monclient: get_monmap_and_config 2023-09-30T01:33:04.747 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.718+0000 7f1c6875d700 10 monclient: build_initial_monmap 2023-09-30T01:33:04.747 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.718+0000 7f1c6875d700 10 monclient: monmap: 2023-09-30T01:33:04.747 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2023-09-30T01:33:04.747 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:33:04.747 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:33:04.723154+0000 2023-09-30T01:33:04.747 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:33:04.723154+0000 2023-09-30T01:33:04.747 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2023-09-30T01:33:04.748 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:33:04.748 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a-legacy 2023-09-30T01:33:04.748 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.107:6789/0 mon.noname-f-legacy 2023-09-30T01:33:04.748 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.noname-b 2023-09-30T01:33:04.748 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.noname-g 2023-09-30T01:33:04.748 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.noname-c 2023-09-30T01:33:04.748 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.noname-h 2023-09-30T01:33:04.748 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.noname-d 2023-09-30T01:33:04.748 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.noname-i 2023-09-30T01:33:04.748 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.noname-e 2023-09-30T01:33:04.748 INFO:teuthology.orchestra.run.smithi088.stderr:9: v2:172.21.15.88:3300/0 mon.noname-a 2023-09-30T01:33:04.748 INFO:teuthology.orchestra.run.smithi088.stderr:10: v2:172.21.15.107:3300/0 mon.noname-f 2023-09-30T01:33:04.749 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:33:04.749 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.726+0000 7f1c6875d700 1 Processor -- start 2023-09-30T01:33:04.749 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.726+0000 7f1c6875d700 1 -- start start 2023-09-30T01:33:04.749 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.726+0000 7f1c6875d700 10 monclient: init 2023-09-30T01:33:04.749 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.726+0000 7f1c6875d700 10 monclient: _reopen_session rank -1 2023-09-30T01:33:04.749 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.726+0000 7f1c6875d700 10 monclient: _add_conns ranks=[7,2,8,5,6,0,10,4,3,1,9] 2023-09-30T01:33:04.749 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.726+0000 7f1c6875d700 1 --2- >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7f1c6009d300 0x7f1c6009bc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:33:04.749 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.726+0000 7f1c6875d700 10 monclient(hunting): picked mon.noname-i con 0x7f1c6009d300 addr [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] 2023-09-30T01:33:04.749 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.726+0000 7f1c6875d700 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1c6009c160 0x7f1c6009d6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:33:04.749 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.726+0000 7f1c6875d700 10 monclient(hunting): picked mon.noname-b con 0x7f1c6009c160 addr [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] 2023-09-30T01:33:04.749 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.726+0000 7f1c6875d700 1 --2- >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f1c6009dc10 0x7f1c60095090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:33:04.749 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.726+0000 7f1c6875d700 10 monclient(hunting): picked mon.noname-e con 0x7f1c6009dc10 addr [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] 2023-09-30T01:33:04.750 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.726+0000 7f1c6875d700 10 monclient(hunting): start opening mon connection 2023-09-30T01:33:04.750 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.726+0000 7f1c6875d700 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c6009b4e0 con 0x7f1c6009c160 2023-09-30T01:33:04.750 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.726+0000 7f1c6875d700 10 monclient(hunting): start opening mon connection 2023-09-30T01:33:04.750 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.726+0000 7f1c6875d700 1 -- --> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] -- mon_getmap magic: 0 v1 -- 0x7f1c60099a10 con 0x7f1c6009d300 2023-09-30T01:33:04.750 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.726+0000 7f1c6875d700 10 monclient(hunting): start opening mon connection 2023-09-30T01:33:04.750 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.726+0000 7f1c6875d700 1 -- --> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] -- mon_getmap magic: 0 v1 -- 0x7f1c6009aeb0 con 0x7f1c6009dc10 2023-09-30T01:33:04.750 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.726+0000 7f1c6875d700 10 monclient(hunting): _renew_subs 2023-09-30T01:33:04.750 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.726+0000 7f1c6875d700 10 monclient(hunting): authenticate will time out at 856.765357s 2023-09-30T01:33:04.750 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.726+0000 7f1c65cf8700 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1c6009c160 0x7f1c6009d6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:33:04.750 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.726+0000 7f1c66cfa700 1 --2- >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f1c6009dc10 0x7f1c60095090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:33:04.750 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.726+0000 7f1c65cf8700 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1c6009c160 0x7f1c6009d6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3301/0 says I am v2:172.21.15.88:52458/0 (socket says 172.21.15.88:52458) 2023-09-30T01:33:04.751 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.726+0000 7f1c65cf8700 1 -- 172.21.15.88:0/1091546605 learned_addr learned my addr 172.21.15.88:0/1091546605 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-09-30T01:33:04.751 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.726+0000 7f1c65cf8700 10 monclient(hunting): get_auth_request con 0x7f1c6009c160 auth_method 0 2023-09-30T01:33:04.751 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.726+0000 7f1c65cf8700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:33:04.751 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.726+0000 7f1c65cf8700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:33:04.751 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.726+0000 7f1c65cf8700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:33:04.751 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.726+0000 7f1c66cfa700 10 monclient(hunting): get_auth_request con 0x7f1c6009dc10 auth_method 0 2023-09-30T01:33:04.751 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.726+0000 7f1c66cfa700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:33:04.751 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.726+0000 7f1c66cfa700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:33:04.751 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.726+0000 7f1c66cfa700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:33:04.751 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.726+0000 7f1c664f9700 1 --2- 172.21.15.88:0/1091546605 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7f1c6009d300 0x7f1c6009bc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:33:04.751 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.726+0000 7f1c65cf8700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:33:04.751 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.726+0000 7f1c65cf8700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:33:04.752 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.726+0000 7f1c65cf8700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:33:04.752 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.726+0000 7f1c66cfa700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:33:04.752 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.726+0000 7f1c66cfa700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:33:04.752 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.726+0000 7f1c66cfa700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:33:04.752 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.726+0000 7f1c664f9700 10 monclient(hunting): get_auth_request con 0x7f1c6009d300 auth_method 0 2023-09-30T01:33:04.752 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.726+0000 7f1c664f9700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:33:04.752 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.726+0000 7f1c664f9700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:33:04.752 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.726+0000 7f1c664f9700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:33:04.752 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.726+0000 7f1c664f9700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:33:04.752 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.726+0000 7f1c664f9700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:33:04.752 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.726+0000 7f1c664f9700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:33:04.752 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.726+0000 7f1c66cfa700 10 monclient(hunting): handle_auth_done global_id 4292 payload 274 2023-09-30T01:33:04.753 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.726+0000 7f1c66cfa700 1 -- 172.21.15.88:0/1091546605 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7f1c6009d300 msgr2=0x7f1c6009bc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:33:04.753 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.726+0000 7f1c66cfa700 1 --2- 172.21.15.88:0/1091546605 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7f1c6009d300 0x7f1c6009bc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:04.753 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.726+0000 7f1c66cfa700 1 -- 172.21.15.88:0/1091546605 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1c6009c160 msgr2=0x7f1c6009d6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:33:04.753 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.726+0000 7f1c66cfa700 1 --2- 172.21.15.88:0/1091546605 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1c6009c160 0x7f1c6009d6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:04.753 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.726+0000 7f1c66cfa700 10 monclient: _finish_hunting 0 2023-09-30T01:33:04.753 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.726+0000 7f1c66cfa700 1 monclient: found mon.noname-e 2023-09-30T01:33:04.753 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.726+0000 7f1c66cfa700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:33:04.753 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.726+0000 7f1c66cfa700 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.88:3304/0 2023-09-30T01:33:04.753 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.726+0000 7f1c66cfa700 1 -- 172.21.15.88:0/1091546605 --> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c6005ecb0 con 0x7f1c6009dc10 2023-09-30T01:33:04.753 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.726+0000 7f1c65cf8700 1 --2- 172.21.15.88:0/1091546605 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1c6009c160 0x7f1c6009d6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:33:04.753 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.726+0000 7f1c66cfa700 1 --2- 172.21.15.88:0/1091546605 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f1c6009dc10 0x7f1c60095090 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f1c5000dd40 tx=0x7f1c5000d630 comp rx=0 tx=0).ready entity=mon.8 client_cookie=ea141a66f3561904 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:33:04.754 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.726+0000 7f1c654f7700 1 -- 172.21.15.88:0/1091546605 <== mon.8 v2:172.21.15.88:3304/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f1c50012070 con 0x7f1c6009dc10 2023-09-30T01:33:04.754 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.726+0000 7f1c654f7700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:33:04.754 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.726+0000 7f1c654f7700 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2023-09-30T01:33:04.754 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.726+0000 7f1c654f7700 10 monclient: dump: 2023-09-30T01:33:04.754 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:33:04.754 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:33:04.754 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:04.754 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:04.754 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:33:04.754 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:33:04.754 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:33:04.754 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:33:04.755 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:33:04.755 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:33:04.755 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:33:04.755 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:33:04.755 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:33:04.755 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:33:04.755 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:33:04.755 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:33:04.755 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.726+0000 7f1c654f7700 10 monclient: _finish_auth 0 2023-09-30T01:33:04.755 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.726+0000 7f1c654f7700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:33:04.755 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.726+0000 7f1c654f7700 1 -- 172.21.15.88:0/1091546605 <== mon.8 v2:172.21.15.88:3304/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c5000d2e0 con 0x7f1c6009dc10 2023-09-30T01:33:04.755 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.726+0000 7f1c654f7700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:33:04.756 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.726+0000 7f1c654f7700 1 -- 172.21.15.88:0/1091546605 <== mon.8 v2:172.21.15.88:3304/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f1c5000dea0 con 0x7f1c6009dc10 2023-09-30T01:33:04.756 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.726+0000 7f1c654f7700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:33:04.756 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.726+0000 7f1c654f7700 10 monclient: got monmap 1 from mon.e (according to old e1) 2023-09-30T01:33:04.756 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.726+0000 7f1c654f7700 10 monclient: dump: 2023-09-30T01:33:04.756 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:33:04.756 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:33:04.756 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:04.756 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:04.756 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:33:04.756 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:33:04.756 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:33:04.756 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:33:04.757 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:33:04.757 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:33:04.757 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:33:04.757 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:33:04.757 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:33:04.757 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:33:04.757 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:33:04.757 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:33:04.757 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.726+0000 7f1c6875d700 5 monclient: authenticate success, global_id 4292 2023-09-30T01:33:04.757 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.726+0000 7f1c6875d700 10 monclient: get_monmap_and_config success 2023-09-30T01:33:04.757 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.726+0000 7f1c6875d700 4 set_mon_vals no callback set 2023-09-30T01:33:04.757 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.726+0000 7f1c6875d700 10 monclient: shutdown 2023-09-30T01:33:04.758 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.726+0000 7f1c6875d700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:33:04.758 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.726+0000 7f1c6875d700 1 -- 172.21.15.88:0/1091546605 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f1c6009dc10 msgr2=0x7f1c60095090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:33:04.758 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.726+0000 7f1c6875d700 1 --2- 172.21.15.88:0/1091546605 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f1c6009dc10 0x7f1c60095090 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f1c5000dd40 tx=0x7f1c5000d630 comp rx=0 tx=0).stop 2023-09-30T01:33:04.758 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.726+0000 7f1c6875d700 1 -- 172.21.15.88:0/1091546605 shutdown_connections 2023-09-30T01:33:04.758 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.726+0000 7f1c6875d700 1 --2- 172.21.15.88:0/1091546605 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f1c6009dc10 0x7f1c60095090 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:04.758 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.726+0000 7f1c6875d700 1 --2- 172.21.15.88:0/1091546605 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1c6009c160 0x7f1c6009d6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:04.758 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.726+0000 7f1c6875d700 1 --2- 172.21.15.88:0/1091546605 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7f1c6009d300 0x7f1c6009bc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:04.758 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.730+0000 7f1c6875d700 1 -- 172.21.15.88:0/1091546605 >> 172.21.15.88:0/1091546605 conn(0x7f1c600a24c0 msgr2=0x7f1c600a2890 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:33:04.758 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.726+0000 7f1fd1bbe700 10 monclient: get_monmap_and_config 2023-09-30T01:33:04.759 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.726+0000 7f1fd1bbe700 10 monclient: build_initial_monmap 2023-09-30T01:33:04.759 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.726+0000 7f1fd1bbe700 10 monclient: monmap: 2023-09-30T01:33:04.759 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2023-09-30T01:33:04.759 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:33:04.759 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:33:04.730467+0000 2023-09-30T01:33:04.759 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:33:04.730467+0000 2023-09-30T01:33:04.759 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2023-09-30T01:33:04.759 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:33:04.759 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a-legacy 2023-09-30T01:33:04.759 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.107:6789/0 mon.noname-f-legacy 2023-09-30T01:33:04.759 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.noname-b 2023-09-30T01:33:04.760 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.noname-g 2023-09-30T01:33:04.760 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.noname-c 2023-09-30T01:33:04.760 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.noname-h 2023-09-30T01:33:04.760 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.noname-d 2023-09-30T01:33:04.760 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.noname-i 2023-09-30T01:33:04.760 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.noname-e 2023-09-30T01:33:04.760 INFO:teuthology.orchestra.run.smithi088.stderr:9: v2:172.21.15.88:3300/0 mon.noname-a 2023-09-30T01:33:04.760 INFO:teuthology.orchestra.run.smithi088.stderr:10: v2:172.21.15.107:3300/0 mon.noname-f 2023-09-30T01:33:04.760 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:33:04.761 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.730+0000 7f1c6875d700 1 -- 172.21.15.88:0/1091546605 shutdown_connections 2023-09-30T01:33:04.761 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.730+0000 7f1c6875d700 1 -- 172.21.15.88:0/1091546605 wait complete. 2023-09-30T01:33:04.761 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.730+0000 7f1c6875d700 10 monclient: build_initial_monmap 2023-09-30T01:33:04.761 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.730+0000 7f1c6875d700 10 monclient: monmap: 2023-09-30T01:33:04.761 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2023-09-30T01:33:04.761 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:33:04.761 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 0.000000 2023-09-30T01:33:04.762 INFO:teuthology.orchestra.run.smithi088.stderr:created 0.000000 2023-09-30T01:33:04.762 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2023-09-30T01:33:04.762 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 1 2023-09-30T01:33:04.762 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.noname-a 2023-09-30T01:33:04.762 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.noname-f 2023-09-30T01:33:04.762 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.noname-b 2023-09-30T01:33:04.762 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.noname-g 2023-09-30T01:33:04.762 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.noname-c 2023-09-30T01:33:04.762 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.noname-h 2023-09-30T01:33:04.762 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.noname-d 2023-09-30T01:33:04.762 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.noname-i 2023-09-30T01:33:04.762 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.noname-e 2023-09-30T01:33:04.763 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:33:04.763 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.730+0000 7f1fd1bbe700 1 Processor -- start 2023-09-30T01:33:04.763 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.730+0000 7f1c6875d700 1 Processor -- start 2023-09-30T01:33:04.763 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.730+0000 7f1fd1bbe700 1 -- start start 2023-09-30T01:33:04.764 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.730+0000 7f1fd1bbe700 10 monclient: init 2023-09-30T01:33:04.764 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.730+0000 7f1c6875d700 1 -- start start 2023-09-30T01:33:04.764 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.730+0000 7f1c6875d700 10 monclient: init 2023-09-30T01:33:04.764 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.730+0000 7f1fd1bbe700 10 monclient: _reopen_session rank -1 2023-09-30T01:33:04.764 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.730+0000 7f1fd1bbe700 10 monclient: _add_conns ranks=[0,4,8,2,1,5,3,6,7,9,10] 2023-09-30T01:33:04.765 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.730+0000 7f0d7d6e5700 10 monclient: get_monmap_and_config 2023-09-30T01:33:04.765 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.730+0000 7f0d7d6e5700 10 monclient: build_initial_monmap 2023-09-30T01:33:04.765 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.730+0000 7f0d7d6e5700 10 monclient: monmap: 2023-09-30T01:33:04.765 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2023-09-30T01:33:04.765 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:33:04.765 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:33:04.733006+0000 2023-09-30T01:33:04.765 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:33:04.733006+0000 2023-09-30T01:33:04.765 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2023-09-30T01:33:04.765 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:33:04.766 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a-legacy 2023-09-30T01:33:04.766 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.107:6789/0 mon.noname-f-legacy 2023-09-30T01:33:04.766 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.noname-b 2023-09-30T01:33:04.766 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.noname-g 2023-09-30T01:33:04.766 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.noname-c 2023-09-30T01:33:04.766 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.noname-h 2023-09-30T01:33:04.766 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.noname-d 2023-09-30T01:33:04.766 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.noname-i 2023-09-30T01:33:04.766 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.noname-e 2023-09-30T01:33:04.766 INFO:teuthology.orchestra.run.smithi088.stderr:9: v2:172.21.15.88:3300/0 mon.noname-a 2023-09-30T01:33:04.766 INFO:teuthology.orchestra.run.smithi088.stderr:10: v2:172.21.15.107:3300/0 mon.noname-f 2023-09-30T01:33:04.766 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:33:04.767 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.730+0000 7f0d7d6e5700 1 Processor -- start 2023-09-30T01:33:04.767 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.730+0000 7f1c6875d700 10 monclient: _reopen_session rank -1 2023-09-30T01:33:04.767 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.730+0000 7f1fd1bbe700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f1fcc09d300 addr v1:172.21.15.88:6789/0 2023-09-30T01:33:04.767 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.730+0000 7f1c6875d700 10 monclient: _add_conns ranks=[7,5,1,0,6,4,2,3,8] 2023-09-30T01:33:04.767 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.730+0000 7f1c6875d700 1 --2- >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7f1c6009c160 0x7f1c6012e8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:33:04.768 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.730+0000 7f1c6875d700 10 monclient(hunting): picked mon.noname-i con 0x7f1c6009c160 addr [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] 2023-09-30T01:33:04.768 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.730+0000 7f1fd1bbe700 1 --2- >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f1fcc09d6d0 0x7f1fcc09daa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:33:04.768 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.730+0000 7f1fd1bbe700 10 monclient(hunting): picked mon.noname-c con 0x7f1fcc09d6d0 addr [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] 2023-09-30T01:33:04.768 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.730+0000 7f1c6875d700 1 --2- >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7f1c6009d300 0x7f1c6013ddc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:33:04.769 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.730+0000 7f1fd1bbe700 1 --2- >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f1fcc134a60 0x7f1fcc136e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:33:04.769 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.730+0000 7f1fd1bbe700 10 monclient(hunting): picked mon.noname-e con 0x7f1fcc134a60 addr [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] 2023-09-30T01:33:04.769 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.730+0000 7f1fd1bbe700 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1fcc05ecb0 con 0x7f1fcc09d300 2023-09-30T01:33:04.769 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.730+0000 7f1fd1bbe700 10 monclient(hunting): start opening mon connection 2023-09-30T01:33:04.769 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.730+0000 7f1fd1bbe700 1 -- --> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] -- mon_getmap magic: 0 v1 -- 0x7f1fcc09b4e0 con 0x7f1fcc09d6d0 2023-09-30T01:33:04.769 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.730+0000 7f1fd1bbe700 10 monclient(hunting): start opening mon connection 2023-09-30T01:33:04.769 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.730+0000 7f1fd1bbe700 1 -- --> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] -- mon_getmap magic: 0 v1 -- 0x7f1fcc099a10 con 0x7f1fcc134a60 2023-09-30T01:33:04.770 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.730+0000 7f1fd1bbe700 10 monclient(hunting): _renew_subs 2023-09-30T01:33:04.770 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.730+0000 7f1fd1bbe700 10 monclient(hunting): authenticate will time out at 856.770017s 2023-09-30T01:33:04.770 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.730+0000 7f1fcb7fe700 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f1fcc09d300 0x7f1fcc09bc20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:54106/0 (socket says 172.21.15.88:54106) 2023-09-30T01:33:04.770 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.730+0000 7f1fcb7fe700 1 -- 172.21.15.88:0/1692040348 learned_addr learned my addr 172.21.15.88:0/1692040348 (peer_addr_for_me v1:172.21.15.88:0/0) 2023-09-30T01:33:04.770 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.730+0000 7f1c6875d700 10 monclient(hunting): picked mon.noname-h con 0x7f1c6009d300 addr [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] 2023-09-30T01:33:04.770 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.730+0000 7f1c6875d700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1c6009dc10 0x7f1c6013e300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:33:04.770 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.730+0000 7f1c6875d700 10 monclient(hunting): picked mon.noname-f con 0x7f1c6009dc10 addr [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] 2023-09-30T01:33:04.770 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.730+0000 7f1c6875d700 10 monclient(hunting): start opening mon connection 2023-09-30T01:33:04.771 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.730+0000 7f1c6875d700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c60129430 con 0x7f1c6009dc10 2023-09-30T01:33:04.771 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.730+0000 7f1c6875d700 10 monclient(hunting): start opening mon connection 2023-09-30T01:33:04.771 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.730+0000 7f1c6875d700 1 -- --> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] -- mon_getmap magic: 0 v1 -- 0x7f1c6013e840 con 0x7f1c6009d300 2023-09-30T01:33:04.771 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.730+0000 7f1c6875d700 10 monclient(hunting): start opening mon connection 2023-09-30T01:33:04.771 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.730+0000 7f1c6875d700 1 -- --> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] -- mon_getmap magic: 0 v1 -- 0x7f1c60128a90 con 0x7f1c6009c160 2023-09-30T01:33:04.771 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.730+0000 7f1c6875d700 10 monclient(hunting): _renew_subs 2023-09-30T01:33:04.771 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.730+0000 7f1c6875d700 10 monclient(hunting): authenticate will time out at 856.770215s 2023-09-30T01:33:04.771 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.730+0000 7f1c65cf8700 1 --2- >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7f1c6009d300 0x7f1c6013ddc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:33:04.771 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.730+0000 7f1c664f9700 1 --2- >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7f1c6009c160 0x7f1c6012e8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:33:04.771 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.730+0000 7f1c664f9700 1 --2- >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7f1c6009c160 0x7f1c6012e8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3303/0 says I am v2:172.21.15.88:47086/0 (socket says 172.21.15.88:47086) 2023-09-30T01:33:04.771 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.730+0000 7f1c664f9700 1 -- 172.21.15.88:0/728105367 learned_addr learned my addr 172.21.15.88:0/728105367 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-09-30T01:33:04.771 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.730+0000 7f1c664f9700 10 monclient(hunting): get_auth_request con 0x7f1c6009c160 auth_method 0 2023-09-30T01:33:04.772 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.730+0000 7f1c664f9700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:33:04.772 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.730+0000 7f1c664f9700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:33:04.772 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.730+0000 7f1c664f9700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:33:04.772 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.730+0000 7f1c65cf8700 10 monclient(hunting): get_auth_request con 0x7f1c6009d300 auth_method 0 2023-09-30T01:33:04.772 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.730+0000 7f1c65cf8700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:33:04.772 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.730+0000 7f1c65cf8700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:33:04.772 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.730+0000 7f1c65cf8700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:33:04.772 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.730+0000 7f1fcbfff700 1 --2- 172.21.15.88:0/1692040348 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f1fcc134a60 0x7f1fcc136e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:33:04.773 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.730+0000 7f1fcbfff700 10 monclient(hunting): get_auth_request con 0x7f1fcc134a60 auth_method 0 2023-09-30T01:33:04.773 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.730+0000 7f1fcbfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:33:04.773 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.730+0000 7f1fcbfff700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:33:04.773 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.730+0000 7f1fcbfff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:33:04.773 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.730+0000 7f1fcaffd700 1 --2- 172.21.15.88:0/1692040348 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f1fcc09d6d0 0x7f1fcc09daa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:33:04.773 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.730+0000 7f1fcaffd700 10 monclient(hunting): get_auth_request con 0x7f1fcc09d6d0 auth_method 0 2023-09-30T01:33:04.773 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.730+0000 7f1fcaffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:33:04.773 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.730+0000 7f1fcaffd700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:33:04.773 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.730+0000 7f1fcaffd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:33:04.774 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.730+0000 7f1c664f9700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:33:04.774 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.730+0000 7f1c664f9700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:33:04.774 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.730+0000 7f1c664f9700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:33:04.774 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.730+0000 7f1c65cf8700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:33:04.774 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.730+0000 7f1c65cf8700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:33:04.774 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.730+0000 7f1fcbfff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:33:04.774 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.730+0000 7f1fcbfff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:33:04.774 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.730+0000 7f1fcbfff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:33:04.775 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.730+0000 7f1fcaffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:33:04.775 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.730+0000 7f1fcaffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:33:04.775 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.730+0000 7f1fcaffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:33:04.775 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.730+0000 7f1c65cf8700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:33:04.775 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.730+0000 7f1c66cfa700 1 --2- 172.21.15.88:0/728105367 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1c6009dc10 0x7f1c6013e300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:33:04.775 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.730+0000 7f1c66cfa700 10 monclient(hunting): get_auth_request con 0x7f1c6009dc10 auth_method 0 2023-09-30T01:33:04.775 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.730+0000 7f1c66cfa700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:33:04.776 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.730+0000 7f1c66cfa700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:33:04.776 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.730+0000 7f1c66cfa700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:33:04.776 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.730+0000 7f1c66cfa700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:33:04.776 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.730+0000 7f1c66cfa700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:33:04.776 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.730+0000 7f1c66cfa700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:33:04.776 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.730+0000 7f1fcbfff700 10 monclient(hunting): handle_auth_done global_id 4301 payload 274 2023-09-30T01:33:04.776 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.730+0000 7f1fcbfff700 1 -- 172.21.15.88:0/1692040348 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f1fcc09d6d0 msgr2=0x7f1fcc09daa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:33:04.777 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.730+0000 7f1fcbfff700 1 --2- 172.21.15.88:0/1692040348 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f1fcc09d6d0 0x7f1fcc09daa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:04.777 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.730+0000 7f1fcbfff700 1 -- 172.21.15.88:0/1692040348 >> v1:172.21.15.88:6789/0 conn(0x7f1fcc09d300 legacy=0x7f1fcc09bc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:33:04.777 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.730+0000 7f1fcbfff700 10 monclient: _finish_hunting 0 2023-09-30T01:33:04.777 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.730+0000 7f1fcbfff700 1 monclient: found mon.noname-e 2023-09-30T01:33:04.777 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.730+0000 7f1fcbfff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:33:04.777 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.730+0000 7f1fcbfff700 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.88:3304/0 2023-09-30T01:33:04.777 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.730+0000 7f1fcbfff700 1 -- 172.21.15.88:0/1692040348 --> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fcc09aeb0 con 0x7f1fcc134a60 2023-09-30T01:33:04.777 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.730+0000 7f1c664f9700 10 monclient(hunting): handle_auth_done global_id 4282 payload 824 2023-09-30T01:33:04.777 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.730+0000 7f1c664f9700 1 -- 172.21.15.88:0/728105367 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7f1c6009d300 msgr2=0x7f1c6013ddc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:33:04.778 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.730+0000 7f1c664f9700 1 --2- 172.21.15.88:0/728105367 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7f1c6009d300 0x7f1c6013ddc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:04.778 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.730+0000 7f1c664f9700 1 -- 172.21.15.88:0/728105367 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1c6009dc10 msgr2=0x7f1c6013e300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:33:04.778 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.730+0000 7f1c664f9700 1 --2- 172.21.15.88:0/728105367 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1c6009dc10 0x7f1c6013e300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:04.778 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.730+0000 7f1c664f9700 10 monclient: _finish_hunting 0 2023-09-30T01:33:04.778 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.730+0000 7f1c664f9700 1 monclient: found mon.noname-i 2023-09-30T01:33:04.778 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.730+0000 7f1c664f9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:33:04.778 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.730+0000 7f1c664f9700 10 monclient: _send_mon_message to mon.noname-i at v2:172.21.15.107:3303/0 2023-09-30T01:33:04.778 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.730+0000 7f1c664f9700 1 -- 172.21.15.88:0/728105367 --> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c5800bb00 con 0x7f1c6009c160 2023-09-30T01:33:04.778 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.730+0000 7f0d7d6e5700 1 -- start start 2023-09-30T01:33:04.779 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.730+0000 7f1fcbfff700 1 --2- 172.21.15.88:0/1692040348 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f1fcc134a60 0x7f1fcc136e40 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f1fb4009910 tx=0x7f1fb40091a0 comp rx=0 tx=0).ready entity=mon.8 client_cookie=dc6bc5fadd7a1e3 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:33:04.779 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.730+0000 7f0d7d6e5700 10 monclient: init 2023-09-30T01:33:04.779 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.730+0000 7f1c664f9700 1 --2- 172.21.15.88:0/728105367 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7f1c6009c160 0x7f1c6012e8a0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f1c5c00ecb0 tx=0x7f1c5c009120 comp rx=0 tx=0).ready entity=mon.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:33:04.780 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.730+0000 7f0d7d6e5700 10 monclient: _reopen_session rank -1 2023-09-30T01:33:04.780 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.730+0000 7f0d7d6e5700 10 monclient: _add_conns ranks=[3,5,8,10,7,4,6,1,2,9,0] 2023-09-30T01:33:04.780 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.730+0000 7f0d7d6e5700 1 --2- >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7f0d7809c250 0x7f0d7809b430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:33:04.780 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.730+0000 7f0d7d6e5700 10 monclient(hunting): picked mon.noname-g con 0x7f0d7809c250 addr [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] 2023-09-30T01:33:04.780 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.730+0000 7f0d7d6e5700 1 --2- >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7f0d7809c620 0x7f0d7809ceb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:33:04.781 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.730+0000 7f0d7d6e5700 10 monclient(hunting): picked mon.noname-h con 0x7f0d7809c620 addr [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] 2023-09-30T01:33:04.781 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.730+0000 7f0d7d6e5700 1 --2- >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f0d7809d3f0 0x7f0d7809d840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:33:04.781 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.730+0000 7f0d7d6e5700 10 monclient(hunting): picked mon.noname-e con 0x7f0d7809d3f0 addr [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] 2023-09-30T01:33:04.781 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.730+0000 7f0d7d6e5700 10 monclient(hunting): start opening mon connection 2023-09-30T01:33:04.781 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.730+0000 7f0d7d6e5700 1 -- --> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d7809dd80 con 0x7f0d7809c250 2023-09-30T01:33:04.781 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.730+0000 7f0d7d6e5700 10 monclient(hunting): start opening mon connection 2023-09-30T01:33:04.781 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.730+0000 7f0d7d6e5700 1 -- --> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] -- mon_getmap magic: 0 v1 -- 0x7f0d7809ec30 con 0x7f0d7809c620 2023-09-30T01:33:04.781 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.730+0000 7f0d7d6e5700 10 monclient(hunting): start opening mon connection 2023-09-30T01:33:04.781 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.730+0000 7f0d7d6e5700 1 -- --> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] -- mon_getmap magic: 0 v1 -- 0x7f0d7809a770 con 0x7f0d7809d3f0 2023-09-30T01:33:04.781 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.730+0000 7f0d7d6e5700 10 monclient(hunting): _renew_subs 2023-09-30T01:33:04.781 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.730+0000 7f0d7d6e5700 10 monclient(hunting): authenticate will time out at 856.771009s 2023-09-30T01:33:04.782 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.730+0000 7f0d767fc700 1 --2- >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7f0d7809c620 0x7f0d7809ceb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:33:04.782 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.730+0000 7f0d777fe700 1 --2- >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f0d7809d3f0 0x7f0d7809d840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:33:04.782 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.730+0000 7f0d777fe700 1 --2- >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f0d7809d3f0 0x7f0d7809d840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3304/0 says I am v2:172.21.15.88:36720/0 (socket says 172.21.15.88:36720) 2023-09-30T01:33:04.782 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.730+0000 7f0d767fc700 1 --2- >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7f0d7809c620 0x7f0d7809ceb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3302/0 says I am v2:172.21.15.88:49734/0 (socket says 172.21.15.88:49734) 2023-09-30T01:33:04.782 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.730+0000 7f0d777fe700 1 -- 172.21.15.88:0/2259177135 learned_addr learned my addr 172.21.15.88:0/2259177135 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-09-30T01:33:04.782 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.730+0000 7f0d777fe700 10 monclient(hunting): get_auth_request con 0x7f0d7809d3f0 auth_method 0 2023-09-30T01:33:04.782 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.730+0000 7f0d777fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:33:04.782 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.730+0000 7f0d777fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:33:04.782 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.730+0000 7f0d777fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:33:04.782 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.730+0000 7f0d767fc700 10 monclient(hunting): get_auth_request con 0x7f0d7809c620 auth_method 0 2023-09-30T01:33:04.782 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.730+0000 7f0d767fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:33:04.783 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.730+0000 7f0d767fc700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:33:04.783 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.730+0000 7f0d767fc700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:33:04.783 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.730+0000 7f0d76ffd700 1 --2- 172.21.15.88:0/2259177135 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7f0d7809c250 0x7f0d7809b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:33:04.783 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.730+0000 7f0d777fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:33:04.783 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.730+0000 7f0d777fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:33:04.783 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.730+0000 7f0d777fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:33:04.783 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.730+0000 7f0d76ffd700 10 monclient(hunting): get_auth_request con 0x7f0d7809c250 auth_method 0 2023-09-30T01:33:04.783 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.730+0000 7f1fca7fc700 1 -- 172.21.15.88:0/1692040348 <== mon.8 v2:172.21.15.88:3304/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f1fb4011ab0 con 0x7f1fcc134a60 2023-09-30T01:33:04.783 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.730+0000 7f1fca7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:33:04.784 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.730+0000 7f0d76ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:33:04.785 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.730+0000 7f1fca7fc700 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2023-09-30T01:33:04.785 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.730+0000 7f1fca7fc700 10 monclient: dump: 2023-09-30T01:33:04.785 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:33:04.785 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:33:04.785 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:04.785 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:04.785 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:33:04.785 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:33:04.785 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:33:04.785 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:33:04.785 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:33:04.786 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:33:04.786 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:33:04.786 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:33:04.786 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:33:04.786 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:33:04.786 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:33:04.786 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:33:04.786 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.730+0000 7f1fca7fc700 10 monclient: _finish_auth 0 2023-09-30T01:33:04.786 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.730+0000 7f1fca7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:33:04.786 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.730+0000 7f1fd1bbe700 5 monclient: authenticate success, global_id 4301 2023-09-30T01:33:04.786 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.730+0000 7f1fd1bbe700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:33:04.787 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.730+0000 7f1fca7fc700 1 -- 172.21.15.88:0/1692040348 <== mon.8 v2:172.21.15.88:3304/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1fb4011c30 con 0x7f1fcc134a60 2023-09-30T01:33:04.787 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.730+0000 7f1fca7fc700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:33:04.787 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.730+0000 7f1fd1bbe700 10 monclient: get_monmap_and_config success 2023-09-30T01:33:04.787 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.730+0000 7f1fca7fc700 1 -- 172.21.15.88:0/1692040348 <== mon.8 v2:172.21.15.88:3304/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f1fb4011da0 con 0x7f1fcc134a60 2023-09-30T01:33:04.787 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.730+0000 7f1fd1bbe700 4 set_mon_vals no callback set 2023-09-30T01:33:04.787 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.730+0000 7f1fca7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:33:04.787 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.730+0000 7f1fca7fc700 10 monclient: got monmap 1 from mon.e (according to old e1) 2023-09-30T01:33:04.787 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.730+0000 7f1fca7fc700 10 monclient: dump: 2023-09-30T01:33:04.787 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:33:04.787 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:33:04.787 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:04.787 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:04.788 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:33:04.788 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:33:04.788 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:33:04.788 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:33:04.788 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:33:04.788 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:33:04.788 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:33:04.788 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:33:04.788 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:33:04.788 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:33:04.788 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:33:04.788 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:33:04.789 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.730+0000 7f1c567fc700 1 -- 172.21.15.88:0/728105367 <== mon.7 v2:172.21.15.107:3303/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f1c5c00ccd0 con 0x7f1c6009c160 2023-09-30T01:33:04.789 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.730+0000 7f1c567fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:33:04.789 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.730+0000 7f1c567fc700 10 monclient: got monmap 1 from mon.noname-i (according to old e1) 2023-09-30T01:33:04.789 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.730+0000 7f1c567fc700 10 monclient: dump: 2023-09-30T01:33:04.789 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:33:04.789 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:33:04.789 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:04.790 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:04.790 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:33:04.790 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:33:04.790 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:33:04.790 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:33:04.790 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:33:04.790 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:33:04.790 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:33:04.790 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:33:04.790 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:33:04.790 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:33:04.791 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:33:04.791 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:33:04.791 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.730+0000 7f1c567fc700 10 monclient: _finish_auth 0 2023-09-30T01:33:04.791 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.730+0000 7f1c567fc700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:33:04.791 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.730+0000 7f1c6875d700 5 monclient: authenticate success, global_id 4282 2023-09-30T01:33:04.791 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.730+0000 7f1c6875d700 10 monclient: _renew_subs 2023-09-30T01:33:04.791 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.730+0000 7f1c6875d700 10 monclient: _send_mon_message to mon.i at v2:172.21.15.107:3303/0 2023-09-30T01:33:04.791 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.730+0000 7f1c6875d700 1 -- 172.21.15.88:0/728105367 --> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c5000d8d0 con 0x7f1c6009c160 2023-09-30T01:33:04.791 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.730+0000 7f0d76ffd700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:33:04.792 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.730+0000 7f1c6875d700 10 monclient: _renew_subs 2023-09-30T01:33:04.792 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.730+0000 7f1c6875d700 10 monclient: _send_mon_message to mon.i at v2:172.21.15.107:3303/0 2023-09-30T01:33:04.792 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.730+0000 7f1c6875d700 1 -- 172.21.15.88:0/728105367 --> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c60129050 con 0x7f1c6009c160 2023-09-30T01:33:04.793 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.734+0000 7f1c567fc700 1 -- 172.21.15.88:0/728105367 <== mon.7 v2:172.21.15.107:3303/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c5c00ce30 con 0x7f1c6009c160 2023-09-30T01:33:04.793 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.734+0000 7f1c567fc700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:33:04.793 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.734+0000 7f1c567fc700 1 -- 172.21.15.88:0/728105367 <== mon.7 v2:172.21.15.107:3303/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f1c5c013470 con 0x7f1c6009c160 2023-09-30T01:33:04.793 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.734+0000 7f1c567fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:33:04.793 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.734+0000 7f1c567fc700 10 monclient: got monmap 1 from mon.i (according to old e1) 2023-09-30T01:33:04.793 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.734+0000 7f1c567fc700 10 monclient: dump: 2023-09-30T01:33:04.793 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:33:04.793 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:33:04.793 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:04.793 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:04.793 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:33:04.794 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:33:04.794 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:33:04.794 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:33:04.794 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:33:04.794 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:33:04.794 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:33:04.794 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:33:04.794 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:33:04.794 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:33:04.794 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:33:04.794 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:33:04.795 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.734+0000 7f1c57fff700 4 set_mon_vals no callback set 2023-09-30T01:33:04.795 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.734+0000 7f00df7fe700 1 -- 172.21.15.88:0/3746327561 <== mon.2 v2:172.21.15.88:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f00ec00c3e0 con 0x7f00f8099df0 2023-09-30T01:33:04.795 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.734+0000 7f00df7fe700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:33:04.795 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.734+0000 7f00df7fe700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:33:04.795 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.734+0000 7f1fd1bbe700 10 monclient: shutdown 2023-09-30T01:33:04.796 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.734+0000 7f1fd1bbe700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:33:04.796 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.734+0000 7f1fd1bbe700 1 -- 172.21.15.88:0/1692040348 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f1fcc134a60 msgr2=0x7f1fcc136e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:33:04.796 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.734+0000 7f1fd1bbe700 1 --2- 172.21.15.88:0/1692040348 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f1fcc134a60 0x7f1fcc136e40 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f1fb4009910 tx=0x7f1fb40091a0 comp rx=0 tx=0).stop 2023-09-30T01:33:04.796 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.734+0000 7f1fd1bbe700 1 -- 172.21.15.88:0/1692040348 shutdown_connections 2023-09-30T01:33:04.796 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.734+0000 7f1fd1bbe700 1 --2- 172.21.15.88:0/1692040348 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f1fcc134a60 0x7f1fcc136e40 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:04.796 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.734+0000 7f1fd1bbe700 1 --2- 172.21.15.88:0/1692040348 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f1fcc09d6d0 0x7f1fcc09daa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:04.796 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.734+0000 7f1fd1bbe700 1 -- 172.21.15.88:0/1692040348 >> 172.21.15.88:0/1692040348 conn(0x7f1fcc0a24c0 msgr2=0x7f1fcc0a2890 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:33:04.797 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.734+0000 7f1c567fc700 1 -- 172.21.15.88:0/728105367 <== mon.7 v2:172.21.15.107:3303/0 4 ==== mgrmap(e 5) v1 ==== 40144+0+0 (secure 0 0 0) 0x7f1c5c00c850 con 0x7f1c6009c160 2023-09-30T01:33:04.797 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.734+0000 7f1c567fc700 1 --2- 172.21.15.88:0/728105367 >> v2:172.21.15.107:6812/29926 conn(0x7f1c44033360 0x7f1c44035810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:33:04.797 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.734+0000 7f1c567fc700 1 -- 172.21.15.88:0/728105367 <== mon.7 v2:172.21.15.107:3303/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 3502+0+0 (secure 0 0 0) 0x7f1c5c043ac0 con 0x7f1c6009c160 2023-09-30T01:33:04.797 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.734+0000 7f1fd1bbe700 1 -- 172.21.15.88:0/1692040348 shutdown_connections 2023-09-30T01:33:04.797 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.734+0000 7f1fd1bbe700 1 -- 172.21.15.88:0/1692040348 wait complete. 2023-09-30T01:33:04.797 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.734+0000 7f1fd1bbe700 10 monclient: build_initial_monmap 2023-09-30T01:33:04.798 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.734+0000 7f1c65cf8700 1 --2- 172.21.15.88:0/728105367 >> v2:172.21.15.107:6812/29926 conn(0x7f1c44033360 0x7f1c44035810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:33:04.798 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.734+0000 7f1fd1bbe700 10 monclient: monmap: 2023-09-30T01:33:04.798 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2023-09-30T01:33:04.798 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:33:04.798 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 0.000000 2023-09-30T01:33:04.798 INFO:teuthology.orchestra.run.smithi088.stderr:created 0.000000 2023-09-30T01:33:04.798 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2023-09-30T01:33:04.799 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 1 2023-09-30T01:33:04.799 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.noname-a 2023-09-30T01:33:04.799 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.noname-f 2023-09-30T01:33:04.799 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.noname-b 2023-09-30T01:33:04.799 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.noname-g 2023-09-30T01:33:04.799 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.noname-c 2023-09-30T01:33:04.799 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.noname-h 2023-09-30T01:33:04.799 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.noname-d 2023-09-30T01:33:04.799 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.noname-i 2023-09-30T01:33:04.799 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.noname-e 2023-09-30T01:33:04.799 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:33:04.800 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.734+0000 7f1c65cf8700 10 monclient: get_auth_request con 0x7f1c44033360 auth_method 0 2023-09-30T01:33:04.800 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.734+0000 7f1fd1bbe700 1 Processor -- start 2023-09-30T01:33:04.800 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.734+0000 7f1fd1bbe700 1 -- start start 2023-09-30T01:33:04.800 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.734+0000 7f1fd1bbe700 10 monclient: init 2023-09-30T01:33:04.801 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.734+0000 7f1c65cf8700 1 --2- 172.21.15.88:0/728105367 >> v2:172.21.15.107:6812/29926 conn(0x7f1c44033360 0x7f1c44035810 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f1c6006ed70 tx=0x7f1c5800d040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:33:04.801 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.734+0000 7f1fd1bbe700 10 monclient: _reopen_session rank -1 2023-09-30T01:33:04.801 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.734+0000 7f1fd1bbe700 10 monclient: _add_conns ranks=[1,6,3,7,5,0,4,2,8] 2023-09-30T01:33:04.802 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.734+0000 7f1fd1bbe700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1fcc09d300 0x7f1fcc094860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:33:04.802 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.734+0000 7f1fd1bbe700 10 monclient(hunting): picked mon.noname-f con 0x7f1fcc09d300 addr [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] 2023-09-30T01:33:04.802 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.734+0000 7f1fd1bbe700 1 --2- >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7f1fcc09d6d0 0x7f1fcc06bbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:33:04.802 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.734+0000 7f1fd1bbe700 10 monclient(hunting): picked mon.noname-d con 0x7f1fcc09d6d0 addr [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] 2023-09-30T01:33:04.802 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.734+0000 7f1fd1bbe700 1 --2- >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7f1fcc134a60 0x7f1fcc06c0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:33:04.802 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.734+0000 7ff728995700 10 monclient: get_monmap_and_config 2023-09-30T01:33:04.802 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.734+0000 7ff728995700 10 monclient: build_initial_monmap 2023-09-30T01:33:04.803 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.734+0000 7f1fd1bbe700 10 monclient(hunting): picked mon.noname-g con 0x7f1fcc134a60 addr [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] 2023-09-30T01:33:04.803 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.734+0000 7f1fd1bbe700 10 monclient(hunting): start opening mon connection 2023-09-30T01:33:04.803 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.734+0000 7f1fd1bbe700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fcc06f300 con 0x7f1fcc09d300 2023-09-30T01:33:04.803 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.734+0000 7f1fd1bbe700 10 monclient(hunting): start opening mon connection 2023-09-30T01:33:04.803 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.734+0000 7f1fd1bbe700 1 -- --> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1fcc088fd0 con 0x7f1fcc134a60 2023-09-30T01:33:04.803 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.734+0000 7f1fd1bbe700 10 monclient(hunting): start opening mon connection 2023-09-30T01:33:04.803 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.734+0000 7f1fd1bbe700 1 -- --> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] -- mon_getmap magic: 0 v1 -- 0x7f1fcc06eb80 con 0x7f1fcc09d6d0 2023-09-30T01:33:04.804 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.734+0000 7f1fd1bbe700 10 monclient(hunting): _renew_subs 2023-09-30T01:33:04.804 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.734+0000 7f1fd1bbe700 10 monclient(hunting): authenticate will time out at 856.774785s 2023-09-30T01:33:04.804 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.734+0000 7f4a25309700 10 monclient: get_monmap_and_config 2023-09-30T01:33:04.804 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.734+0000 7f4a25309700 10 monclient: build_initial_monmap 2023-09-30T01:33:04.804 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.734+0000 7ff728995700 10 monclient: monmap: 2023-09-30T01:33:04.805 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2023-09-30T01:33:04.805 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:33:04.805 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:33:04.738583+0000 2023-09-30T01:33:04.805 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:33:04.738583+0000 2023-09-30T01:33:04.805 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2023-09-30T01:33:04.805 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:33:04.805 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a-legacy 2023-09-30T01:33:04.805 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.107:6789/0 mon.noname-f-legacy 2023-09-30T01:33:04.805 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.noname-b 2023-09-30T01:33:04.805 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.noname-g 2023-09-30T01:33:04.805 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.noname-c 2023-09-30T01:33:04.806 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.noname-h 2023-09-30T01:33:04.806 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.noname-d 2023-09-30T01:33:04.806 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.noname-i 2023-09-30T01:33:04.806 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.noname-e 2023-09-30T01:33:04.806 INFO:teuthology.orchestra.run.smithi088.stderr:9: v2:172.21.15.88:3300/0 mon.noname-a 2023-09-30T01:33:04.806 INFO:teuthology.orchestra.run.smithi088.stderr:10: v2:172.21.15.107:3300/0 mon.noname-f 2023-09-30T01:33:04.806 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:33:04.806 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.734+0000 7f4a25309700 10 monclient: monmap: 2023-09-30T01:33:04.807 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2023-09-30T01:33:04.807 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:33:04.807 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:33:04.738635+0000 2023-09-30T01:33:04.807 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:33:04.738635+0000 2023-09-30T01:33:04.807 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2023-09-30T01:33:04.807 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:33:04.807 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a-legacy 2023-09-30T01:33:04.807 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.107:6789/0 mon.noname-f-legacy 2023-09-30T01:33:04.807 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.noname-b 2023-09-30T01:33:04.807 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.noname-g 2023-09-30T01:33:04.807 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.noname-c 2023-09-30T01:33:04.807 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.noname-h 2023-09-30T01:33:04.808 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.noname-d 2023-09-30T01:33:04.808 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.noname-i 2023-09-30T01:33:04.808 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.noname-e 2023-09-30T01:33:04.808 INFO:teuthology.orchestra.run.smithi088.stderr:9: v2:172.21.15.88:3300/0 mon.noname-a 2023-09-30T01:33:04.808 INFO:teuthology.orchestra.run.smithi088.stderr:10: v2:172.21.15.107:3300/0 mon.noname-f 2023-09-30T01:33:04.808 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:33:04.809 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.734+0000 7f1fcaffd700 1 --2- >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7f1fcc09d6d0 0x7f1fcc06bbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:33:04.809 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.734+0000 7f1fcaffd700 1 --2- >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7f1fcc09d6d0 0x7f1fcc06bbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3303/0 says I am v2:172.21.15.88:53570/0 (socket says 172.21.15.88:53570) 2023-09-30T01:33:04.810 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.734+0000 7f1fcaffd700 1 -- 172.21.15.88:0/2208255062 learned_addr learned my addr 172.21.15.88:0/2208255062 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-09-30T01:33:04.810 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.734+0000 7f1fcaffd700 10 monclient(hunting): get_auth_request con 0x7f1fcc09d6d0 auth_method 0 2023-09-30T01:33:04.810 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.734+0000 7f1fcaffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:33:04.810 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.734+0000 7f1fcaffd700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:33:04.810 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.734+0000 7f1fcaffd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:33:04.810 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.734+0000 7f1fcbfff700 1 --2- 172.21.15.88:0/2208255062 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7f1fcc134a60 0x7f1fcc06c0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:33:04.810 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.734+0000 7f1fcbfff700 10 monclient(hunting): get_auth_request con 0x7f1fcc134a60 auth_method 0 2023-09-30T01:33:04.810 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.734+0000 7f1fcbfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:33:04.810 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.734+0000 7f1fcbfff700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:33:04.810 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.734+0000 7f1fcbfff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:33:04.811 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.734+0000 7f1fcaffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:33:04.811 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.734+0000 7f1fcaffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:33:04.811 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.734+0000 7f1fcaffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:33:04.811 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.734+0000 7f1fcbfff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:33:04.811 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.734+0000 7f1fcbfff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:33:04.811 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.734+0000 7f1fcbfff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:33:04.811 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.734+0000 7f1fcaffd700 10 monclient(hunting): handle_auth_done global_id 4245 payload 824 2023-09-30T01:33:04.811 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.734+0000 7f1fcaffd700 1 -- 172.21.15.88:0/2208255062 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7f1fcc134a60 msgr2=0x7f1fcc06c0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:33:04.811 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.734+0000 7f1fcaffd700 1 --2- 172.21.15.88:0/2208255062 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7f1fcc134a60 0x7f1fcc06c0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:04.811 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.734+0000 7f1fcaffd700 1 -- 172.21.15.88:0/2208255062 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1fcc09d300 msgr2=0x7f1fcc094860 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-09-30T01:33:04.811 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.734+0000 7f1fcaffd700 1 --2- 172.21.15.88:0/2208255062 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1fcc09d300 0x7f1fcc094860 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:04.812 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.734+0000 7f1fcaffd700 10 monclient: _finish_hunting 0 2023-09-30T01:33:04.812 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.734+0000 7f1fcaffd700 1 monclient: found mon.noname-d 2023-09-30T01:33:04.812 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.734+0000 7f1fcaffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:33:04.812 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.734+0000 7f1fcaffd700 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.88:3303/0 2023-09-30T01:33:04.812 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.734+0000 7f1fcaffd700 1 -- 172.21.15.88:0/2208255062 --> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fb4009420 con 0x7f1fcc09d6d0 2023-09-30T01:33:04.812 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.734+0000 7ff728995700 1 Processor -- start 2023-09-30T01:33:04.813 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.734+0000 7f1c6875d700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:33:04.813 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.734+0000 7f1c6875d700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:33:04.813 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.734+0000 7f1c6875d700 10 monclient: _send_mon_message to mon.i at v2:172.21.15.107:3303/0 2023-09-30T01:33:04.813 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.734+0000 7f1c6875d700 1 -- 172.21.15.88:0/728105367 --> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1c480050b0 con 0x7f1c6009c160 2023-09-30T01:33:04.813 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.730+0000 7f0d76ffd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:33:04.813 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.730+0000 7f0d767fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:33:04.814 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.730+0000 7f0d767fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:33:04.814 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.730+0000 7f0d767fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:33:04.814 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.730+0000 7f0d777fe700 10 monclient(hunting): handle_auth_done global_id 4310 payload 274 2023-09-30T01:33:04.814 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.730+0000 7f0d777fe700 1 -- 172.21.15.88:0/2259177135 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7f0d7809c620 msgr2=0x7f0d7809ceb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:33:04.814 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.730+0000 7f0d777fe700 1 --2- 172.21.15.88:0/2259177135 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7f0d7809c620 0x7f0d7809ceb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:04.814 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.730+0000 7f0d777fe700 1 -- 172.21.15.88:0/2259177135 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7f0d7809c250 msgr2=0x7f0d7809b430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:33:04.814 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.730+0000 7f0d777fe700 1 --2- 172.21.15.88:0/2259177135 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7f0d7809c250 0x7f0d7809b430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:04.814 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.730+0000 7f0d777fe700 10 monclient: _finish_hunting 0 2023-09-30T01:33:04.814 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.730+0000 7f0d777fe700 1 monclient: found mon.noname-e 2023-09-30T01:33:04.814 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.730+0000 7f0d777fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:33:04.815 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.730+0000 7f0d777fe700 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.88:3304/0 2023-09-30T01:33:04.815 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.730+0000 7f0d777fe700 1 -- 172.21.15.88:0/2259177135 --> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d7805ecb0 con 0x7f0d7809d3f0 2023-09-30T01:33:04.815 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.734+0000 7f1fcaffd700 1 --2- 172.21.15.88:0/2208255062 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7f1fcc09d6d0 0x7f1fcc06bbb0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f1fbc00ec00 tx=0x7f1fbc0091a0 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:33:04.815 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.734+0000 7f0d777fe700 1 --2- 172.21.15.88:0/2259177135 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f0d7809d3f0 0x7f0d7809d840 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f0d6c0099e0 tx=0x7f0d6c009250 comp rx=0 tx=0).ready entity=mon.8 client_cookie=172c8a677db351d7 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:33:04.815 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.734+0000 7f0d75ffb700 1 -- 172.21.15.88:0/2259177135 <== mon.8 v2:172.21.15.88:3304/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f0d6c00bce0 con 0x7f0d7809d3f0 2023-09-30T01:33:04.816 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.734+0000 7f0d75ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:33:04.816 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.734+0000 7f0d75ffb700 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2023-09-30T01:33:04.816 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.734+0000 7f0d75ffb700 10 monclient: dump: 2023-09-30T01:33:04.816 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:33:04.816 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:33:04.816 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:04.816 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:04.816 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:33:04.816 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:33:04.816 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:33:04.816 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:33:04.817 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:33:04.817 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:33:04.817 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:33:04.817 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:33:04.817 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:33:04.817 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:33:04.817 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:33:04.817 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:33:04.817 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.734+0000 7f0d75ffb700 10 monclient: _finish_auth 0 2023-09-30T01:33:04.817 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.734+0000 7f0d75ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:33:04.817 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.734+0000 7f0d7d6e5700 5 monclient: authenticate success, global_id 4310 2023-09-30T01:33:04.818 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.734+0000 7f0d75ffb700 1 -- 172.21.15.88:0/2259177135 <== mon.8 v2:172.21.15.88:3304/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d6c00be60 con 0x7f0d7809d3f0 2023-09-30T01:33:04.818 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.734+0000 7f0d75ffb700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:33:04.818 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.734+0000 7f0d7d6e5700 10 monclient: get_monmap_and_config success 2023-09-30T01:33:04.818 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.734+0000 7f0d7d6e5700 4 set_mon_vals no callback set 2023-09-30T01:33:04.818 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.734+0000 7f0d75ffb700 1 -- 172.21.15.88:0/2259177135 <== mon.8 v2:172.21.15.88:3304/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f0d6c0118a0 con 0x7f0d7809d3f0 2023-09-30T01:33:04.818 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.734+0000 7f0d75ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:33:04.818 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.734+0000 7f0d75ffb700 10 monclient: got monmap 1 from mon.e (according to old e1) 2023-09-30T01:33:04.818 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.734+0000 7f0d75ffb700 10 monclient: dump: 2023-09-30T01:33:04.818 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:33:04.818 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:33:04.818 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:04.818 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:04.819 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:33:04.819 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:33:04.819 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:33:04.819 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:33:04.819 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:33:04.819 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:33:04.819 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:33:04.819 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:33:04.819 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:33:04.819 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:33:04.819 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:33:04.820 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:33:04.820 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.734+0000 7f0d7d6e5700 10 monclient: shutdown 2023-09-30T01:33:04.820 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.734+0000 7f0d7d6e5700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:33:04.820 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.734+0000 7f0d7d6e5700 1 -- 172.21.15.88:0/2259177135 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f0d7809d3f0 msgr2=0x7f0d7809d840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:33:04.820 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.734+0000 7f0d7d6e5700 1 --2- 172.21.15.88:0/2259177135 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f0d7809d3f0 0x7f0d7809d840 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f0d6c0099e0 tx=0x7f0d6c009250 comp rx=0 tx=0).stop 2023-09-30T01:33:04.820 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.734+0000 7f0d7d6e5700 1 -- 172.21.15.88:0/2259177135 shutdown_connections 2023-09-30T01:33:04.820 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.734+0000 7f0d7d6e5700 1 --2- 172.21.15.88:0/2259177135 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f0d7809d3f0 0x7f0d7809d840 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:04.820 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.734+0000 7f0d7d6e5700 1 --2- 172.21.15.88:0/2259177135 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7f0d7809c620 0x7f0d7809ceb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:04.820 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.734+0000 7f0d7d6e5700 1 --2- 172.21.15.88:0/2259177135 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7f0d7809c250 0x7f0d7809b430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:04.820 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.734+0000 7f0d7d6e5700 1 -- 172.21.15.88:0/2259177135 >> 172.21.15.88:0/2259177135 conn(0x7f0d780a24c0 msgr2=0x7f0d780a2890 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:33:04.821 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.734+0000 7ff728995700 1 -- start start 2023-09-30T01:33:04.821 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.734+0000 7ff728995700 10 monclient: init 2023-09-30T01:33:04.821 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.734+0000 7f4a25309700 1 Processor -- start 2023-09-30T01:33:04.821 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.734+0000 7f4a25309700 1 -- start start 2023-09-30T01:33:04.821 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.734+0000 7f4a25309700 10 monclient: init 2023-09-30T01:33:04.822 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.734+0000 7f0d7d6e5700 1 -- 172.21.15.88:0/2259177135 shutdown_connections 2023-09-30T01:33:04.822 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.734+0000 7f0d7d6e5700 1 -- 172.21.15.88:0/2259177135 wait complete. 2023-09-30T01:33:04.822 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.738+0000 7f4a25309700 10 monclient: _reopen_session rank -1 2023-09-30T01:33:04.822 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.738+0000 7f4a25309700 10 monclient: _add_conns ranks=[3,4,1,7,6,8,9,10,2,0,5] 2023-09-30T01:33:04.823 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.738+0000 7f4a25309700 1 --2- >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7f4a200a3270 0x7f4a200a0160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:33:04.823 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.738+0000 7f4a25309700 10 monclient(hunting): picked mon.noname-g con 0x7f4a200a3270 addr [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] 2023-09-30T01:33:04.823 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.738+0000 7f4a25309700 1 --2- >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f4a200a4b00 0x7f4a200a06a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:33:04.823 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.738+0000 7f4a25309700 10 monclient(hunting): picked mon.noname-c con 0x7f4a200a4b00 addr [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] 2023-09-30T01:33:04.823 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.738+0000 7ff728995700 10 monclient: _reopen_session rank -1 2023-09-30T01:33:04.823 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.738+0000 7ff728995700 10 monclient: _add_conns ranks=[2,9,7,6,1,4,10,8,0,3,5] 2023-09-30T01:33:04.824 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.738+0000 7f4a25309700 10 monclient(hunting): picked mon.noname-f-legacy con 0x7f4a201333c0 addr v1:172.21.15.107:6789/0 2023-09-30T01:33:04.824 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.738+0000 7f4a25309700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a2005ecb0 con 0x7f4a201333c0 2023-09-30T01:33:04.824 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.738+0000 7f4a25309700 10 monclient(hunting): start opening mon connection 2023-09-30T01:33:04.824 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.738+0000 7f4a25309700 1 -- --> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a200a4ed0 con 0x7f4a200a3270 2023-09-30T01:33:04.824 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.738+0000 7f4a25309700 10 monclient(hunting): start opening mon connection 2023-09-30T01:33:04.824 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.738+0000 7f4a25309700 1 -- --> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] -- mon_getmap magic: 0 v1 -- 0x7f4a200a6330 con 0x7f4a200a4b00 2023-09-30T01:33:04.824 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.738+0000 7f4a25309700 10 monclient(hunting): _renew_subs 2023-09-30T01:33:04.824 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.738+0000 7f4a25309700 10 monclient(hunting): authenticate will time out at 856.776530s 2023-09-30T01:33:04.825 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.738+0000 7ff728995700 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff7240a1340 0x7ff724131820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:33:04.825 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.738+0000 7ff728995700 10 monclient(hunting): picked mon.noname-b con 0x7ff7240a1340 addr [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] 2023-09-30T01:33:04.825 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.738+0000 7f1fc37fe700 1 -- 172.21.15.88:0/2208255062 <== mon.6 v2:172.21.15.88:3303/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f1fbc00ccd0 con 0x7f1fcc09d6d0 2023-09-30T01:33:04.825 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.738+0000 7f1fc37fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:33:04.825 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.738+0000 7f1fc37fe700 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2023-09-30T01:33:04.826 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.738+0000 7ff728995700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7ff72406ede0 0x7ff72406f1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:33:04.826 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.738+0000 7ff728995700 10 monclient(hunting): picked mon.noname-a con 0x7ff72406ede0 addr v2:172.21.15.88:3300/0 2023-09-30T01:33:04.826 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.738+0000 7ff728995700 1 --2- >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7ff72409eef0 0x7ff72409f390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:33:04.826 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.738+0000 7f1fc37fe700 10 monclient: dump: 2023-09-30T01:33:04.826 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:33:04.826 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:33:04.826 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:04.826 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:04.827 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:33:04.827 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:33:04.827 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:33:04.827 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:33:04.827 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:33:04.827 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:33:04.827 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:33:04.827 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:33:04.827 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:33:04.827 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:33:04.827 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:33:04.827 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:33:04.828 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.738+0000 7f1fc37fe700 10 monclient: _finish_auth 0 2023-09-30T01:33:04.828 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.738+0000 7f1fc37fe700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:33:04.828 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.738+0000 7f1fc37fe700 1 -- 172.21.15.88:0/2208255062 <== mon.6 v2:172.21.15.88:3303/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1fbc00ce70 con 0x7f1fcc09d6d0 2023-09-30T01:33:04.828 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.738+0000 7f1fd1bbe700 5 monclient: authenticate success, global_id 4245 2023-09-30T01:33:04.828 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.738+0000 7f1fd1bbe700 10 monclient: _renew_subs 2023-09-30T01:33:04.828 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.738+0000 7f1fd1bbe700 10 monclient: _send_mon_message to mon.d at v2:172.21.15.88:3303/0 2023-09-30T01:33:04.828 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.738+0000 7f1fd1bbe700 1 -- 172.21.15.88:0/2208255062 --> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1fcc06d7d0 con 0x7f1fcc09d6d0 2023-09-30T01:33:04.828 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.738+0000 7f1fc37fe700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:33:04.828 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.738+0000 7f1fc37fe700 1 -- 172.21.15.88:0/2208255062 <== mon.6 v2:172.21.15.88:3303/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f1fbc0138f0 con 0x7f1fcc09d6d0 2023-09-30T01:33:04.828 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.738+0000 7f1fc37fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:33:04.828 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.738+0000 7f1fc37fe700 10 monclient: got monmap 1 from mon.d (according to old e1) 2023-09-30T01:33:04.829 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.738+0000 7f0d7d6e5700 10 monclient: build_initial_monmap 2023-09-30T01:33:04.829 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.738+0000 7f0d7d6e5700 10 monclient: monmap: 2023-09-30T01:33:04.829 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2023-09-30T01:33:04.829 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:33:04.829 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 0.000000 2023-09-30T01:33:04.829 INFO:teuthology.orchestra.run.smithi088.stderr:created 0.000000 2023-09-30T01:33:04.829 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2023-09-30T01:33:04.829 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 1 2023-09-30T01:33:04.829 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.noname-a 2023-09-30T01:33:04.829 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.noname-f 2023-09-30T01:33:04.830 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.noname-b 2023-09-30T01:33:04.830 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.noname-g 2023-09-30T01:33:04.830 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.noname-c 2023-09-30T01:33:04.830 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.noname-h 2023-09-30T01:33:04.830 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.noname-d 2023-09-30T01:33:04.830 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.noname-i 2023-09-30T01:33:04.830 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.noname-e 2023-09-30T01:33:04.830 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:33:04.830 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.738+0000 7f1fc37fe700 10 monclient: dump: 2023-09-30T01:33:04.831 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:33:04.831 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:33:04.831 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:04.831 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:04.831 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:33:04.831 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:33:04.831 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:33:04.831 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:33:04.831 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:33:04.831 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:33:04.831 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:33:04.832 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:33:04.832 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:33:04.832 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:33:04.832 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:33:04.832 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:33:04.832 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.738+0000 7f1fd1bbe700 10 monclient: _renew_subs 2023-09-30T01:33:04.832 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.738+0000 7f1fd1bbe700 10 monclient: _send_mon_message to mon.d at v2:172.21.15.88:3303/0 2023-09-30T01:33:04.832 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.738+0000 7f1fd1bbe700 1 -- 172.21.15.88:0/2208255062 --> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1fcc094f40 con 0x7f1fcc09d6d0 2023-09-30T01:33:04.832 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.738+0000 7f4a1e7fc700 1 --2- >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f4a200a4b00 0x7f4a200a06a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:33:04.832 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.738+0000 7f4a1e7fc700 1 --2- >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f4a200a4b00 0x7f4a200a06a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3302/0 says I am v2:172.21.15.88:50440/0 (socket says 172.21.15.88:50440) 2023-09-30T01:33:04.833 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.738+0000 7ff72259c700 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff7240a1340 0x7ff724131820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:33:04.833 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.738+0000 7ff722d9d700 1 --2- >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7ff72409eef0 0x7ff72409f390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:33:04.833 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.738+0000 7ff722d9d700 1 --2- >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7ff72409eef0 0x7ff72409f390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3303/0 says I am v2:172.21.15.88:47100/0 (socket says 172.21.15.88:47100) 2023-09-30T01:33:04.834 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.738+0000 7ff722d9d700 1 -- 172.21.15.88:0/1007842781 learned_addr learned my addr 172.21.15.88:0/1007842781 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-09-30T01:33:04.834 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.738+0000 7ff72259c700 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff7240a1340 0x7ff724131820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3301/0 says I am v2:172.21.15.88:52470/0 (socket says 172.21.15.88:52470) 2023-09-30T01:33:04.834 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.738+0000 7ff728995700 10 monclient(hunting): picked mon.noname-i con 0x7ff72409eef0 addr [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] 2023-09-30T01:33:04.834 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.738+0000 7ff728995700 10 monclient(hunting): start opening mon connection 2023-09-30T01:33:04.834 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.738+0000 7ff728995700 1 -- 172.21.15.88:0/1007842781 --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff7240a0090 con 0x7ff72406ede0 2023-09-30T01:33:04.834 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.738+0000 7ff728995700 10 monclient(hunting): start opening mon connection 2023-09-30T01:33:04.834 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.738+0000 7ff728995700 1 -- 172.21.15.88:0/1007842781 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff7240a1710 con 0x7ff7240a1340 2023-09-30T01:33:04.834 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.738+0000 7ff728995700 10 monclient(hunting): start opening mon connection 2023-09-30T01:33:04.834 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.738+0000 7ff728995700 1 -- 172.21.15.88:0/1007842781 --> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] -- mon_getmap magic: 0 v1 -- 0x7ff72406c7a0 con 0x7ff72409eef0 2023-09-30T01:33:04.834 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.738+0000 7ff728995700 10 monclient(hunting): _renew_subs 2023-09-30T01:33:04.834 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.738+0000 7ff728995700 10 monclient(hunting): authenticate will time out at 856.777022s 2023-09-30T01:33:04.835 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.738+0000 7ff722d9d700 10 monclient(hunting): get_auth_request con 0x7ff72409eef0 auth_method 0 2023-09-30T01:33:04.835 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.738+0000 7ff722d9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:33:04.835 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.738+0000 7ff722d9d700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:33:04.835 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.738+0000 7ff722d9d700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:33:04.835 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.738+0000 7ff72259c700 10 monclient(hunting): get_auth_request con 0x7ff7240a1340 auth_method 0 2023-09-30T01:33:04.835 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.738+0000 7ff72259c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:33:04.835 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.738+0000 7ff72259c700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:33:04.835 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.738+0000 7ff72259c700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:33:04.835 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.738+0000 7f1fc97fa700 4 set_mon_vals no callback set 2023-09-30T01:33:04.836 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.738+0000 7ff722d9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:33:04.836 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.738+0000 7ff722d9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:33:04.836 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.738+0000 7ff722d9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:33:04.836 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.738+0000 7ff72259c700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:33:04.836 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.738+0000 7ff72259c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:33:04.836 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.738+0000 7ff72259c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:33:04.836 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.738+0000 7f0d7d6e5700 1 Processor -- start 2023-09-30T01:33:04.837 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.738+0000 7ff72259c700 10 monclient(hunting): handle_auth_done global_id 4304 payload 274 2023-09-30T01:33:04.837 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.738+0000 7ff71bfff700 1 --2- 172.21.15.88:0/1007842781 >> v2:172.21.15.88:3300/0 conn(0x7ff72406ede0 0x7ff72406f1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:33:04.837 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.738+0000 7ff72259c700 1 -- 172.21.15.88:0/1007842781 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7ff72409eef0 msgr2=0x7ff72409f390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:33:04.837 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.738+0000 7ff72259c700 1 --2- 172.21.15.88:0/1007842781 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7ff72409eef0 0x7ff72409f390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:04.837 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.738+0000 7ff72259c700 1 -- 172.21.15.88:0/1007842781 >> v2:172.21.15.88:3300/0 conn(0x7ff72406ede0 msgr2=0x7ff72406f1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:33:04.838 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.738+0000 7f0d7d6e5700 1 -- start start 2023-09-30T01:33:04.838 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.738+0000 7f0d7d6e5700 10 monclient: init 2023-09-30T01:33:04.838 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.738+0000 7f4a1e7fc700 1 -- 172.21.15.88:0/1425017912 learned_addr learned my addr 172.21.15.88:0/1425017912 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-09-30T01:33:04.838 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.738+0000 7f4a1e7fc700 10 monclient(hunting): get_auth_request con 0x7f4a200a4b00 auth_method 0 2023-09-30T01:33:04.839 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.738+0000 7ff72259c700 1 --2- 172.21.15.88:0/1007842781 >> v2:172.21.15.88:3300/0 conn(0x7ff72406ede0 0x7ff72406f1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:04.839 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.738+0000 7ff72259c700 10 monclient: _finish_hunting 0 2023-09-30T01:33:04.839 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.738+0000 7f4a1e7fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:33:04.839 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.738+0000 7f4a1e7fc700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:33:04.839 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.738+0000 7f4a1e7fc700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:33:04.839 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.738+0000 7ff72259c700 1 monclient: found mon.noname-b 2023-09-30T01:33:04.840 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.738+0000 7ff72259c700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:33:04.840 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.738+0000 7ff72259c700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.88:3301/0 2023-09-30T01:33:04.840 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.738+0000 7ff72259c700 1 -- 172.21.15.88:0/1007842781 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff72405ecb0 con 0x7ff7240a1340 2023-09-30T01:33:04.840 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.738+0000 7ff71bfff700 10 monclient: get_auth_request con 0x7ff72406ede0 auth_method 0 2023-09-30T01:33:04.840 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.738+0000 7ff71bfff700 1 --2- 172.21.15.88:0/1007842781 >> v2:172.21.15.88:3300/0 conn(0x7ff72406ede0 0x7ff72406f1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-09-30T01:33:04.840 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.738+0000 7f4a1e7fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:33:04.840 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.738+0000 7f4a1e7fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:33:04.840 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.738+0000 7f4a1e7fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:33:04.841 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.738+0000 7f4a1e7fc700 10 monclient(hunting): handle_auth_done global_id 4306 payload 274 2023-09-30T01:33:04.841 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.738+0000 7f4a1e7fc700 1 -- 172.21.15.88:0/1425017912 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7f4a200a3270 msgr2=0x7f4a200a0160 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2023-09-30T01:33:04.841 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.738+0000 7f4a1e7fc700 1 --2- 172.21.15.88:0/1425017912 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7f4a200a3270 0x7f4a200a0160 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:04.841 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.738+0000 7f4a1e7fc700 1 -- 172.21.15.88:0/1425017912 >> v1:172.21.15.107:6789/0 conn(0x7f4a201333c0 legacy=0x7f4a201337b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:33:04.841 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.738+0000 7f4a1e7fc700 10 monclient: _finish_hunting 0 2023-09-30T01:33:04.841 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.738+0000 7f4a1e7fc700 1 monclient: found mon.noname-c 2023-09-30T01:33:04.841 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.738+0000 7f4a1e7fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:33:04.841 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.738+0000 7f4a1e7fc700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.88:3302/0 2023-09-30T01:33:04.841 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.738+0000 7f4a1e7fc700 1 -- 172.21.15.88:0/1425017912 --> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a20098610 con 0x7f4a200a4b00 2023-09-30T01:33:04.842 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.738+0000 7f0d7d6e5700 10 monclient: _reopen_session rank -1 2023-09-30T01:33:04.842 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.738+0000 7f0d7d6e5700 10 monclient: _add_conns ranks=[0,8,3,2,7,5,4,6,1] 2023-09-30T01:33:04.842 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.738+0000 7f0d7d6e5700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0d7809c250 0x7f0d7806c630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:33:04.842 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.738+0000 7f0d7d6e5700 10 monclient(hunting): picked mon.noname-a con 0x7f0d7809c250 addr [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] 2023-09-30T01:33:04.842 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.738+0000 7f0d7d6e5700 1 --2- >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f0d7809c620 0x7f0d7806cb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:33:04.842 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.738+0000 7f0d7d6e5700 10 monclient(hunting): picked mon.noname-e con 0x7f0d7809c620 addr [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] 2023-09-30T01:33:04.842 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.738+0000 7f0d7d6e5700 1 --2- >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7f0d7809d3f0 0x7f0d7806d0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:33:04.842 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.738+0000 7f0d7d6e5700 10 monclient(hunting): picked mon.noname-g con 0x7f0d7809d3f0 addr [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] 2023-09-30T01:33:04.842 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.738+0000 7f0d7d6e5700 10 monclient(hunting): start opening mon connection 2023-09-30T01:33:04.843 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.738+0000 7f0d7d6e5700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d78095820 con 0x7f0d7809c250 2023-09-30T01:33:04.843 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.738+0000 7f0d7d6e5700 10 monclient(hunting): start opening mon connection 2023-09-30T01:33:04.843 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.738+0000 7f0d7d6e5700 1 -- --> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d7809a360 con 0x7f0d7809d3f0 2023-09-30T01:33:04.843 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.738+0000 7f0d7d6e5700 10 monclient(hunting): start opening mon connection 2023-09-30T01:33:04.843 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.738+0000 7f4a1e7fc700 1 --2- 172.21.15.88:0/1425017912 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f4a200a4b00 0x7f4a200a06a0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f4a140099e0 tx=0x7f4a14009250 comp rx=0 tx=0).ready entity=mon.4 client_cookie=fb090608aeb62d3d server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:33:04.843 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.738+0000 7ff72259c700 1 --2- 172.21.15.88:0/1007842781 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff7240a1340 0x7ff724131820 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7ff714009900 tx=0x7ff714009120 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6b50a0bcc790a24f server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:33:04.844 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.738+0000 7f0d7d6e5700 1 -- --> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] -- mon_getmap magic: 0 v1 -- 0x7f0d7806f600 con 0x7f0d7809c620 2023-09-30T01:33:04.844 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.738+0000 7f0d7d6e5700 10 monclient(hunting): _renew_subs 2023-09-30T01:33:04.844 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.738+0000 7f0d7d6e5700 10 monclient(hunting): authenticate will time out at 856.778106s 2023-09-30T01:33:04.844 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.738+0000 7f0d767fc700 1 --2- >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f0d7809c620 0x7f0d7806cb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:33:04.844 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.738+0000 7f0d777fe700 1 --2- >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7f0d7809d3f0 0x7f0d7806d0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:33:04.845 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.738+0000 7ff721d9b700 1 -- 172.21.15.88:0/1007842781 <== mon.2 v2:172.21.15.88:3301/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7ff714005c10 con 0x7ff7240a1340 2023-09-30T01:33:04.845 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.738+0000 7ff721d9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:33:04.845 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.738+0000 7f0d777fe700 1 --2- >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7f0d7809d3f0 0x7f0d7806d0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3301/0 says I am v2:172.21.15.88:48344/0 (socket says 172.21.15.88:48344) 2023-09-30T01:33:04.845 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.738+0000 7f0d777fe700 1 -- 172.21.15.88:0/3073637209 learned_addr learned my addr 172.21.15.88:0/3073637209 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-09-30T01:33:04.845 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.738+0000 7f0d777fe700 10 monclient(hunting): get_auth_request con 0x7f0d7809d3f0 auth_method 0 2023-09-30T01:33:04.845 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.738+0000 7f0d777fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:33:04.845 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.738+0000 7f1fc37fe700 1 -- 172.21.15.88:0/2208255062 <== mon.6 v2:172.21.15.88:3303/0 4 ==== mgrmap(e 5) v1 ==== 40144+0+0 (secure 0 0 0) 0x7f1fbc013470 con 0x7f1fcc09d6d0 2023-09-30T01:33:04.846 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.738+0000 7f0d777fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:33:04.846 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.738+0000 7f0d777fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:33:04.846 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.738+0000 7f0d777fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:33:04.846 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.738+0000 7f1fd1bbe700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:33:04.847 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.738+0000 7f1fd1bbe700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:33:04.847 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.738+0000 7f1fd1bbe700 10 monclient: _send_mon_message to mon.d at v2:172.21.15.88:3303/0 2023-09-30T01:33:04.847 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.738+0000 7ff721d9b700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T01:33:04.847 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.738+0000 7ff721d9b700 10 monclient: dump: 2023-09-30T01:33:04.847 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:33:04.847 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:33:04.847 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:04.848 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:04.848 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:33:04.848 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:33:04.848 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:33:04.848 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:33:04.848 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:33:04.848 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:33:04.848 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:33:04.848 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:33:04.848 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:33:04.848 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:33:04.848 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:33:04.849 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:33:04.849 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.738+0000 7ff721d9b700 10 monclient: _finish_auth 0 2023-09-30T01:33:04.849 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.738+0000 7ff721d9b700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:33:04.849 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.738+0000 7ff721d9b700 1 -- 172.21.15.88:0/1007842781 <== mon.2 v2:172.21.15.88:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff714005d90 con 0x7ff7240a1340 2023-09-30T01:33:04.849 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.738+0000 7ff721d9b700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:33:04.849 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.738+0000 7ff721d9b700 1 -- 172.21.15.88:0/1007842781 <== mon.2 v2:172.21.15.88:3301/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7ff71400b430 con 0x7ff7240a1340 2023-09-30T01:33:04.849 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.738+0000 7ff728995700 5 monclient: authenticate success, global_id 4304 2023-09-30T01:33:04.849 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.738+0000 7ff728995700 10 monclient: get_monmap_and_config success 2023-09-30T01:33:04.849 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.738+0000 7ff728995700 4 set_mon_vals no callback set 2023-09-30T01:33:04.849 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.738+0000 7ff721d9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:33:04.849 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.738+0000 7ff721d9b700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T01:33:04.849 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.738+0000 7ff721d9b700 10 monclient: dump: 2023-09-30T01:33:04.850 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:33:04.850 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:33:04.850 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:04.850 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:04.850 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:33:04.850 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:33:04.850 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:33:04.850 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:33:04.850 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:33:04.850 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:33:04.850 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:33:04.850 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:33:04.851 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:33:04.851 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:33:04.851 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:33:04.851 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:33:04.851 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.738+0000 7f1fc37fe700 1 --2- 172.21.15.88:0/2208255062 >> v2:172.21.15.107:6812/29926 conn(0x7f1fac033360 0x7f1fac035810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:33:04.851 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.738+0000 7f1fd1bbe700 1 -- 172.21.15.88:0/2208255062 --> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1fb00050b0 con 0x7f1fcc09d6d0 2023-09-30T01:33:04.851 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.738+0000 7f0d777fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:33:04.852 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.738+0000 7f4a1dffb700 1 -- 172.21.15.88:0/1425017912 <== mon.4 v2:172.21.15.88:3302/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f4a1400bca0 con 0x7f4a200a4b00 2023-09-30T01:33:04.852 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.738+0000 7f4a1dffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:33:04.852 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.738+0000 7f4a1dffb700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T01:33:04.852 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.738+0000 7f4a1dffb700 10 monclient: dump: 2023-09-30T01:33:04.852 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:33:04.852 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:33:04.853 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:04.853 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:04.853 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:33:04.853 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:33:04.853 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:33:04.853 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:33:04.853 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:33:04.853 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:33:04.853 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:33:04.853 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:33:04.853 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:33:04.853 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:33:04.854 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:33:04.854 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:33:04.854 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.738+0000 7f4a1dffb700 10 monclient: _finish_auth 0 2023-09-30T01:33:04.854 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.738+0000 7f4a1dffb700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:33:04.854 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.738+0000 7f4a1dffb700 1 -- 172.21.15.88:0/1425017912 <== mon.4 v2:172.21.15.88:3302/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a1400be20 con 0x7f4a200a4b00 2023-09-30T01:33:04.854 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.738+0000 7f4a1dffb700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:33:04.854 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.738+0000 7f4a1dffb700 1 -- 172.21.15.88:0/1425017912 <== mon.4 v2:172.21.15.88:3302/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f4a140118a0 con 0x7f4a200a4b00 2023-09-30T01:33:04.854 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.738+0000 7f4a1dffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:33:04.854 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.738+0000 7f4a1dffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T01:33:04.854 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.738+0000 7f4a1dffb700 10 monclient: dump: 2023-09-30T01:33:04.854 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:33:04.855 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:33:04.855 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:04.855 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:04.855 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:33:04.855 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:33:04.855 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:33:04.855 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:33:04.855 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:33:04.855 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:33:04.855 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:33:04.855 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:33:04.855 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:33:04.855 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:33:04.856 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:33:04.856 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:33:04.856 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.738+0000 7f4a25309700 5 monclient: authenticate success, global_id 4306 2023-09-30T01:33:04.856 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.738+0000 7f4a25309700 10 monclient: get_monmap_and_config success 2023-09-30T01:33:04.856 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.738+0000 7f4a25309700 4 set_mon_vals no callback set 2023-09-30T01:33:04.856 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.738+0000 7f0d76ffd700 1 --2- 172.21.15.88:0/3073637209 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0d7809c250 0x7f0d7806c630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:33:04.857 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.738+0000 7f0d777fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:33:04.857 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.738+0000 7f0d76ffd700 10 monclient(hunting): get_auth_request con 0x7f0d7809c250 auth_method 0 2023-09-30T01:33:04.857 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.738+0000 7f0d76ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:33:04.857 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.738+0000 7f0d76ffd700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:33:04.857 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.738+0000 7f0d76ffd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:33:04.857 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.738+0000 7f0d76ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:33:04.857 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.738+0000 7f0d76ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:33:04.857 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.738+0000 7f0d76ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:33:04.857 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.738+0000 7f0d767fc700 10 monclient(hunting): get_auth_request con 0x7f0d7809c620 auth_method 0 2023-09-30T01:33:04.857 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.738+0000 7f0d767fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:33:04.857 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.738+0000 7f0d767fc700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:33:04.858 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.738+0000 7f0d767fc700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:33:04.858 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.738+0000 7f0d767fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:33:04.858 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.738+0000 7f0d767fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:33:04.858 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.738+0000 7f0d767fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:33:04.858 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.738+0000 7f0d777fe700 10 monclient(hunting): handle_auth_done global_id 4251 payload 824 2023-09-30T01:33:04.858 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.738+0000 7f0d777fe700 1 -- 172.21.15.88:0/3073637209 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f0d7809c620 msgr2=0x7f0d7806cb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:33:04.858 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.738+0000 7f1fcb7fe700 1 --2- 172.21.15.88:0/2208255062 >> v2:172.21.15.107:6812/29926 conn(0x7f1fac033360 0x7f1fac035810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:33:04.858 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.738+0000 7f1fc37fe700 1 -- 172.21.15.88:0/2208255062 <== mon.6 v2:172.21.15.88:3303/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 3502+0+0 (secure 0 0 0) 0x7f1fbc043a60 con 0x7f1fcc09d6d0 2023-09-30T01:33:04.859 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.738+0000 7f0d777fe700 1 --2- 172.21.15.88:0/3073637209 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f0d7809c620 0x7f0d7806cb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:04.859 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.738+0000 7f0d777fe700 1 -- 172.21.15.88:0/3073637209 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0d7809c250 msgr2=0x7f0d7806c630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:33:04.859 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.738+0000 7f0d777fe700 1 --2- 172.21.15.88:0/3073637209 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0d7809c250 0x7f0d7806c630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:04.859 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.738+0000 7f0d777fe700 10 monclient: _finish_hunting 0 2023-09-30T01:33:04.859 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.738+0000 7f0d777fe700 1 monclient: found mon.noname-g 2023-09-30T01:33:04.859 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.738+0000 7f0d777fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:33:04.860 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.738+0000 7f0d777fe700 10 monclient: _send_mon_message to mon.noname-g at v2:172.21.15.107:3301/0 2023-09-30T01:33:04.860 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.738+0000 7f0d777fe700 1 -- 172.21.15.88:0/3073637209 --> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d6c0094f0 con 0x7f0d7809d3f0 2023-09-30T01:33:04.860 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.738+0000 7f0d76ffd700 1 --2- 172.21.15.88:0/3073637209 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0d7809c250 0x7f0d7806c630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:33:04.860 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.738+0000 7f0d767fc700 1 --2- 172.21.15.88:0/3073637209 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f0d7809c620 0x7f0d7806cb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:33:04.860 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.738+0000 7f1fcb7fe700 10 monclient: get_auth_request con 0x7f1fac033360 auth_method 0 2023-09-30T01:33:04.860 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.738+0000 7f0d777fe700 1 --2- 172.21.15.88:0/3073637209 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7f0d7809d3f0 0x7f0d7806d0b0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f0d7809a8b0 tx=0x7f0d6c017000 comp rx=0 tx=0).ready entity=mon.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:33:04.861 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.738+0000 7f4a25309700 10 monclient: shutdown 2023-09-30T01:33:04.861 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.738+0000 7f4a25309700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:33:04.861 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.738+0000 7f4a25309700 1 -- 172.21.15.88:0/1425017912 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f4a200a4b00 msgr2=0x7f4a200a06a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:33:04.861 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.738+0000 7f4a25309700 1 --2- 172.21.15.88:0/1425017912 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f4a200a4b00 0x7f4a200a06a0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f4a140099e0 tx=0x7f4a14009250 comp rx=0 tx=0).stop 2023-09-30T01:33:04.861 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.738+0000 7ff728995700 10 monclient: shutdown 2023-09-30T01:33:04.861 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.738+0000 7ff728995700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:33:04.862 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.738+0000 7ff728995700 1 -- 172.21.15.88:0/1007842781 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff7240a1340 msgr2=0x7ff724131820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:33:04.862 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.738+0000 7ff728995700 1 --2- 172.21.15.88:0/1007842781 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff7240a1340 0x7ff724131820 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7ff714009900 tx=0x7ff714009120 comp rx=0 tx=0).stop 2023-09-30T01:33:04.862 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.738+0000 7f4a25309700 1 -- 172.21.15.88:0/1425017912 shutdown_connections 2023-09-30T01:33:04.862 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.738+0000 7f4a25309700 1 --2- 172.21.15.88:0/1425017912 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f4a200a4b00 0x7f4a200a06a0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:04.862 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.738+0000 7f4a25309700 1 --2- 172.21.15.88:0/1425017912 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7f4a200a3270 0x7f4a200a0160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:04.862 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.738+0000 7f4a25309700 1 -- 172.21.15.88:0/1425017912 >> 172.21.15.88:0/1425017912 conn(0x7f4a200a67f0 msgr2=0x7f4a200a6bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:33:04.863 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.738+0000 7ff728995700 1 -- 172.21.15.88:0/1007842781 shutdown_connections 2023-09-30T01:33:04.863 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.738+0000 7ff728995700 1 --2- 172.21.15.88:0/1007842781 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7ff72409eef0 0x7ff72409f390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:04.863 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.738+0000 7ff728995700 1 --2- 172.21.15.88:0/1007842781 >> v2:172.21.15.88:3300/0 conn(0x7ff72406ede0 0x7ff72406f1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:04.863 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.738+0000 7ff728995700 1 --2- 172.21.15.88:0/1007842781 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff7240a1340 0x7ff724131820 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:04.863 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.738+0000 7ff728995700 1 -- 172.21.15.88:0/1007842781 >> 172.21.15.88:0/1007842781 conn(0x7ff724097a40 msgr2=0x7ff724063450 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:33:04.863 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.738+0000 7f1fcb7fe700 1 --2- 172.21.15.88:0/2208255062 >> v2:172.21.15.107:6812/29926 conn(0x7f1fac033360 0x7f1fac035810 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f1fc4001460 tx=0x7f1fc4007c40 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:33:04.867 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.742+0000 7f4a25309700 1 -- 172.21.15.88:0/1425017912 shutdown_connections 2023-09-30T01:33:04.867 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.742+0000 7f4a25309700 1 -- 172.21.15.88:0/1425017912 wait complete. 2023-09-30T01:33:04.868 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.742+0000 7f4a25309700 10 monclient: build_initial_monmap 2023-09-30T01:33:04.868 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.742+0000 7f4a25309700 10 monclient: monmap: 2023-09-30T01:33:04.868 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2023-09-30T01:33:04.868 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:33:04.868 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 0.000000 2023-09-30T01:33:04.868 INFO:teuthology.orchestra.run.smithi088.stderr:created 0.000000 2023-09-30T01:33:04.868 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2023-09-30T01:33:04.868 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 1 2023-09-30T01:33:04.868 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.noname-a 2023-09-30T01:33:04.868 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.noname-f 2023-09-30T01:33:04.868 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.noname-b 2023-09-30T01:33:04.869 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.noname-g 2023-09-30T01:33:04.869 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.noname-c 2023-09-30T01:33:04.869 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.noname-h 2023-09-30T01:33:04.869 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.noname-d 2023-09-30T01:33:04.869 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.noname-i 2023-09-30T01:33:04.869 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.noname-e 2023-09-30T01:33:04.869 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:33:04.869 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.742+0000 7f4a25309700 1 Processor -- start 2023-09-30T01:33:04.869 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.742+0000 7f4a25309700 1 -- start start 2023-09-30T01:33:04.869 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.742+0000 7f4a25309700 10 monclient: init 2023-09-30T01:33:04.869 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.742+0000 7f4a25309700 10 monclient: _reopen_session rank -1 2023-09-30T01:33:04.869 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.742+0000 7f4a25309700 10 monclient: _add_conns ranks=[5,0,7,8,2,4,3,6,1] 2023-09-30T01:33:04.870 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.742+0000 7f4a25309700 1 --2- >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7f4a200a3270 0x7f4a20142880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:33:04.870 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.742+0000 7f4a25309700 10 monclient(hunting): picked mon.noname-h con 0x7f4a200a3270 addr [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] 2023-09-30T01:33:04.870 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.742+0000 7f4a25309700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4a200a4b00 0x7f4a20142dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:33:04.870 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.742+0000 7f4a25309700 10 monclient(hunting): picked mon.noname-a con 0x7f4a200a4b00 addr [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] 2023-09-30T01:33:04.870 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.742+0000 7f4a25309700 1 --2- >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7f4a201333c0 0x7f4a20143300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:33:04.870 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.742+0000 7f4a25309700 10 monclient(hunting): picked mon.noname-i con 0x7f4a201333c0 addr [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] 2023-09-30T01:33:04.870 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.742+0000 7f4a25309700 10 monclient(hunting): start opening mon connection 2023-09-30T01:33:04.870 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.742+0000 7f4a25309700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a201213f0 con 0x7f4a200a4b00 2023-09-30T01:33:04.870 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.742+0000 7f4a25309700 10 monclient(hunting): start opening mon connection 2023-09-30T01:33:04.870 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.742+0000 7f4a25309700 1 -- --> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] -- mon_getmap magic: 0 v1 -- 0x7f4a20097c70 con 0x7f4a200a3270 2023-09-30T01:33:04.870 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.742+0000 7f4a25309700 10 monclient(hunting): start opening mon connection 2023-09-30T01:33:04.871 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.742+0000 7f4a25309700 1 -- --> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] -- mon_getmap magic: 0 v1 -- 0x7f4a200a0e80 con 0x7f4a201333c0 2023-09-30T01:33:04.871 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.742+0000 7f4a25309700 10 monclient(hunting): _renew_subs 2023-09-30T01:33:04.871 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.742+0000 7f4a25309700 10 monclient(hunting): authenticate will time out at 856.781424s 2023-09-30T01:33:04.871 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.742+0000 7f4a1e7fc700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4a200a4b00 0x7f4a20142dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:33:04.871 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.742+0000 7f4a1e7fc700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4a200a4b00 0x7f4a20142dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:57878/0 (socket says 172.21.15.88:57878) 2023-09-30T01:33:04.871 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.742+0000 7f4a1e7fc700 1 -- 172.21.15.88:0/2569192909 learned_addr learned my addr 172.21.15.88:0/2569192909 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-09-30T01:33:04.871 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.742+0000 7f4a1e7fc700 10 monclient(hunting): get_auth_request con 0x7f4a200a4b00 auth_method 0 2023-09-30T01:33:04.871 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.742+0000 7f4a1e7fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:33:04.871 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.742+0000 7f4a1e7fc700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:33:04.871 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.742+0000 7f4a1e7fc700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:33:04.871 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.742+0000 7f4a1e7fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:33:04.871 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.742+0000 7f4a1e7fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:33:04.872 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.742+0000 7f4a1e7fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:33:04.872 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.742+0000 7f4a1effd700 1 --2- 172.21.15.88:0/2569192909 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7f4a200a3270 0x7f4a20142880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:33:04.872 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.742+0000 7f4a1f7fe700 1 --2- 172.21.15.88:0/2569192909 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7f4a201333c0 0x7f4a20143300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:33:04.872 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.742+0000 7f4a1effd700 10 monclient(hunting): get_auth_request con 0x7f4a200a3270 auth_method 0 2023-09-30T01:33:04.872 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.742+0000 7f4a1effd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:33:04.872 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.742+0000 7f4a1effd700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:33:04.872 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.742+0000 7f4a1effd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:33:04.872 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.742+0000 7f4a1f7fe700 10 monclient(hunting): get_auth_request con 0x7f4a201333c0 auth_method 0 2023-09-30T01:33:04.872 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.742+0000 7f4a1f7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:33:04.872 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.742+0000 7f4a1f7fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:33:04.872 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.742+0000 7f4a1f7fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:33:04.872 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.742+0000 7f4a1e7fc700 10 monclient(hunting): handle_auth_done global_id 4365 payload 824 2023-09-30T01:33:04.873 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.742+0000 7f4a1e7fc700 1 -- 172.21.15.88:0/2569192909 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7f4a201333c0 msgr2=0x7f4a20143300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:33:04.873 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.742+0000 7f4a1e7fc700 1 --2- 172.21.15.88:0/2569192909 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7f4a201333c0 0x7f4a20143300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:04.873 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.742+0000 7f4a1e7fc700 1 -- 172.21.15.88:0/2569192909 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7f4a200a3270 msgr2=0x7f4a20142880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:33:04.873 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.742+0000 7f4a1e7fc700 1 --2- 172.21.15.88:0/2569192909 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7f4a200a3270 0x7f4a20142880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:04.873 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.742+0000 7f4a1e7fc700 10 monclient: _finish_hunting 0 2023-09-30T01:33:04.873 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.742+0000 7f4a1e7fc700 1 monclient: found mon.noname-a 2023-09-30T01:33:04.873 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.742+0000 7f4a1e7fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:33:04.873 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.742+0000 7f4a1e7fc700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.88:3300/0 2023-09-30T01:33:04.873 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.742+0000 7f4a1e7fc700 1 -- 172.21.15.88:0/2569192909 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a140094f0 con 0x7f4a200a4b00 2023-09-30T01:33:04.873 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.742+0000 7f4a1effd700 1 --2- 172.21.15.88:0/2569192909 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7f4a200a3270 0x7f4a20142880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-30T01:33:04.873 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.742+0000 7f4a1e7fc700 1 --2- 172.21.15.88:0/2569192909 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4a200a4b00 0x7f4a20142dc0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f4a14011ad0 tx=0x7f4a14016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:33:04.874 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.742+0000 7f4a06ffd700 1 -- 172.21.15.88:0/2569192909 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f4a1400bca0 con 0x7f4a200a4b00 2023-09-30T01:33:04.874 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.742+0000 7f4a06ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:33:04.874 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.742+0000 7f4a06ffd700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T01:33:04.874 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.742+0000 7f4a06ffd700 10 monclient: dump: 2023-09-30T01:33:04.874 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:33:04.874 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:33:04.874 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:04.874 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:04.874 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:33:04.874 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:33:04.874 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:33:04.874 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:33:04.875 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:33:04.875 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:33:04.875 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:33:04.875 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:33:04.875 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:33:04.875 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:33:04.875 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:33:04.875 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:33:04.875 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.742+0000 7f4a06ffd700 10 monclient: _finish_auth 0 2023-09-30T01:33:04.875 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.742+0000 7f4a06ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:33:04.875 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.742+0000 7f4a25309700 5 monclient: authenticate success, global_id 4365 2023-09-30T01:33:04.875 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.742+0000 7f4a06ffd700 1 -- 172.21.15.88:0/2569192909 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a14020070 con 0x7f4a200a4b00 2023-09-30T01:33:04.876 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.742+0000 7f4a25309700 10 monclient: _renew_subs 2023-09-30T01:33:04.876 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.742+0000 7f4a25309700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.88:3300/0 2023-09-30T01:33:04.876 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.742+0000 7f4a25309700 1 -- 172.21.15.88:0/2569192909 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a2011f700 con 0x7f4a200a4b00 2023-09-30T01:33:04.876 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.742+0000 7f4a06ffd700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:33:04.876 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.742+0000 7f4a06ffd700 1 -- 172.21.15.88:0/2569192909 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f4a14006a90 con 0x7f4a200a4b00 2023-09-30T01:33:04.876 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.742+0000 7f4a06ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:33:04.876 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.742+0000 7f4a06ffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:33:04.876 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.742+0000 7f4a06ffd700 10 monclient: dump: 2023-09-30T01:33:04.876 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:33:04.876 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:33:04.876 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:04.877 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:04.877 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:33:04.877 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:33:04.877 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:33:04.877 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:33:04.877 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:33:04.877 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:33:04.877 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:33:04.877 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:33:04.877 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:33:04.877 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:33:04.877 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:33:04.878 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:33:04.878 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.742+0000 7f4a25309700 10 monclient: _renew_subs 2023-09-30T01:33:04.878 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.742+0000 7f4a25309700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.88:3300/0 2023-09-30T01:33:04.878 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.742+0000 7f4a25309700 1 -- 172.21.15.88:0/2569192909 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a20122460 con 0x7f4a200a4b00 2023-09-30T01:33:04.878 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.742+0000 7f4a06ffd700 1 -- 172.21.15.88:0/2569192909 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 40144+0+0 (secure 0 0 0) 0x7f4a14022030 con 0x7f4a200a4b00 2023-09-30T01:33:04.878 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.742+0000 7f4a06ffd700 1 --2- 172.21.15.88:0/2569192909 >> v2:172.21.15.107:6812/29926 conn(0x7f4a0c0332c0 0x7f4a0c035770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:33:04.878 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.742+0000 7f4a06ffd700 1 -- 172.21.15.88:0/2569192909 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 3502+0+0 (secure 0 0 0) 0x7f4a140450e0 con 0x7f4a200a4b00 2023-09-30T01:33:04.878 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.742+0000 7f4a25309700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:33:04.878 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.742+0000 7f4a25309700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:33:04.878 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.742+0000 7f4a25309700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.88:3300/0 2023-09-30T01:33:04.878 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.742+0000 7f4a25309700 1 -- 172.21.15.88:0/2569192909 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4a20071960 con 0x7f4a200a4b00 2023-09-30T01:33:04.878 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.742+0000 7f4a1effd700 1 --2- 172.21.15.88:0/2569192909 >> v2:172.21.15.107:6812/29926 conn(0x7f4a0c0332c0 0x7f4a0c035770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:33:04.879 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.742+0000 7f4a1effd700 10 monclient: get_auth_request con 0x7f4a0c0332c0 auth_method 0 2023-09-30T01:33:04.879 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.746+0000 7ff728995700 1 -- 172.21.15.88:0/1007842781 shutdown_connections 2023-09-30T01:33:04.879 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.746+0000 7f4a1cff9700 4 set_mon_vals no callback set 2023-09-30T01:33:04.880 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.746+0000 7ff728995700 1 -- 172.21.15.88:0/1007842781 wait complete. 2023-09-30T01:33:04.880 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.746+0000 7ff728995700 10 monclient: build_initial_monmap 2023-09-30T01:33:04.880 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.746+0000 7ff728995700 10 monclient: monmap: 2023-09-30T01:33:04.880 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2023-09-30T01:33:04.880 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:33:04.880 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 0.000000 2023-09-30T01:33:04.880 INFO:teuthology.orchestra.run.smithi088.stderr:created 0.000000 2023-09-30T01:33:04.880 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2023-09-30T01:33:04.880 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 1 2023-09-30T01:33:04.880 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.noname-a 2023-09-30T01:33:04.881 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.noname-f 2023-09-30T01:33:04.881 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.noname-b 2023-09-30T01:33:04.881 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.noname-g 2023-09-30T01:33:04.881 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.noname-c 2023-09-30T01:33:04.881 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.noname-h 2023-09-30T01:33:04.881 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.noname-d 2023-09-30T01:33:04.881 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.noname-i 2023-09-30T01:33:04.881 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.noname-e 2023-09-30T01:33:04.881 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:33:04.881 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.746+0000 7f4a1effd700 1 --2- 172.21.15.88:0/2569192909 >> v2:172.21.15.107:6812/29926 conn(0x7f4a0c0332c0 0x7f4a0c035770 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f4a10001410 tx=0x7f4a100096a0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:33:04.882 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.746+0000 7ff728995700 1 Processor -- start 2023-09-30T01:33:04.882 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.746+0000 7ff728995700 1 -- start start 2023-09-30T01:33:04.882 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.746+0000 7ff728995700 10 monclient: init 2023-09-30T01:33:04.882 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.746+0000 7f1fc37fe700 1 -- 172.21.15.88:0/2208255062 <== mon.6 v2:172.21.15.88:3303/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f1fbc013720 con 0x7f1fcc09d6d0 2023-09-30T01:33:04.883 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.746+0000 7f1fc37fe700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:33:04.883 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.746+0000 7f1fc37fe700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:33:04.884 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.746+0000 7ff728995700 10 monclient: _reopen_session rank -1 2023-09-30T01:33:04.884 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.746+0000 7ff728995700 10 monclient: _add_conns ranks=[5,3,8,7,2,6,0,1,4] 2023-09-30T01:33:04.884 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.746+0000 7ff728995700 1 --2- >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7ff72406ede0 0x7ff72409d370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:33:04.884 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.746+0000 7ff728995700 10 monclient(hunting): picked mon.noname-h con 0x7ff72406ede0 addr [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] 2023-09-30T01:33:04.884 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.746+0000 7ff728995700 1 --2- >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7ff72409eef0 0x7ff724131f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:33:04.884 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.746+0000 7ff728995700 10 monclient(hunting): picked mon.noname-g con 0x7ff72409eef0 addr [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] 2023-09-30T01:33:04.884 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.746+0000 7ff728995700 1 --2- >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7ff724132450 0x7ff7241328a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:33:04.884 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.746+0000 7ff728995700 10 monclient(hunting): picked mon.noname-e con 0x7ff724132450 addr [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] 2023-09-30T01:33:04.884 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.746+0000 7ff728995700 10 monclient(hunting): start opening mon connection 2023-09-30T01:33:04.884 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.746+0000 7ff728995700 1 -- --> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff724132de0 con 0x7ff72409eef0 2023-09-30T01:33:04.885 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.746+0000 7ff728995700 10 monclient(hunting): start opening mon connection 2023-09-30T01:33:04.885 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.746+0000 7ff728995700 1 -- --> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] -- mon_getmap magic: 0 v1 -- 0x7ff7241347c0 con 0x7ff72406ede0 2023-09-30T01:33:04.885 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.746+0000 7ff728995700 10 monclient(hunting): start opening mon connection 2023-09-30T01:33:04.885 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.746+0000 7ff728995700 1 -- --> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] -- mon_getmap magic: 0 v1 -- 0x7ff724133410 con 0x7ff724132450 2023-09-30T01:33:04.885 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.746+0000 7ff728995700 10 monclient(hunting): _renew_subs 2023-09-30T01:33:04.885 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.746+0000 7ff728995700 10 monclient(hunting): authenticate will time out at 856.785718s 2023-09-30T01:33:04.885 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.746+0000 7f1c567fc700 1 -- 172.21.15.88:0/728105367 <== mon.7 v2:172.21.15.107:3303/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f1c5c013970 con 0x7f1c6009c160 2023-09-30T01:33:04.885 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.746+0000 7f1c567fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:33:04.885 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.746+0000 7f1c567fc700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:33:04.888 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.746+0000 7ff722d9d700 1 --2- >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7ff724132450 0x7ff7241328a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:33:04.888 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.746+0000 7ff72259c700 1 --2- >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7ff72406ede0 0x7ff72409d370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:33:04.888 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.746+0000 7ff722d9d700 1 --2- >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7ff724132450 0x7ff7241328a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3304/0 says I am v2:172.21.15.88:36744/0 (socket says 172.21.15.88:36744) 2023-09-30T01:33:04.888 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.746+0000 7ff72259c700 10 monclient(hunting): get_auth_request con 0x7ff72406ede0 auth_method 0 2023-09-30T01:33:04.889 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.746+0000 7ff722d9d700 1 -- 172.21.15.88:0/3609853028 learned_addr learned my addr 172.21.15.88:0/3609853028 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-09-30T01:33:04.889 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.746+0000 7ff72259c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:33:04.889 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.746+0000 7ff72259c700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:33:04.889 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.746+0000 7ff72259c700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:33:04.889 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.746+0000 7ff722d9d700 10 monclient(hunting): get_auth_request con 0x7ff724132450 auth_method 0 2023-09-30T01:33:04.889 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.746+0000 7ff722d9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:33:04.889 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.746+0000 7ff722d9d700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:33:04.889 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.746+0000 7ff722d9d700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:33:04.889 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.746+0000 7ff722d9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:33:04.889 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.746+0000 7ff722d9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:33:04.889 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.746+0000 7ff722d9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:33:04.890 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.746+0000 7ff72259c700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:33:04.890 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.746+0000 7ff72259c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:33:04.890 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.746+0000 7ff72259c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:33:04.890 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.746+0000 7ff722d9d700 10 monclient(hunting): handle_auth_done global_id 4328 payload 824 2023-09-30T01:33:04.890 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.746+0000 7ff722d9d700 1 -- 172.21.15.88:0/3609853028 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7ff72406ede0 msgr2=0x7ff72409d370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:33:04.890 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.746+0000 7ff722d9d700 1 --2- 172.21.15.88:0/3609853028 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7ff72406ede0 0x7ff72409d370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:04.890 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.746+0000 7ff722d9d700 1 -- 172.21.15.88:0/3609853028 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7ff72409eef0 msgr2=0x7ff724131f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:33:04.890 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.746+0000 7ff722d9d700 1 --2- 172.21.15.88:0/3609853028 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7ff72409eef0 0x7ff724131f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:04.890 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.746+0000 7ff722d9d700 10 monclient: _finish_hunting 0 2023-09-30T01:33:04.890 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.746+0000 7ff722d9d700 1 monclient: found mon.noname-e 2023-09-30T01:33:04.890 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.746+0000 7ff722d9d700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:33:04.891 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.746+0000 7ff722d9d700 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.88:3304/0 2023-09-30T01:33:04.891 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.746+0000 7ff722d9d700 1 -- 172.21.15.88:0/3609853028 --> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7140093c0 con 0x7ff724132450 2023-09-30T01:33:04.891 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.746+0000 7ff722d9d700 1 --2- 172.21.15.88:0/3609853028 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7ff724132450 0x7ff7241328a0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7ff71c009db0 tx=0x7ff71c00b040 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:33:04.891 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.750+0000 7ff71a7fc700 1 -- 172.21.15.88:0/3609853028 <== mon.8 v2:172.21.15.88:3304/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7ff71c009b10 con 0x7ff724132450 2023-09-30T01:33:04.891 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.750+0000 7ff71a7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:33:04.891 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.750+0000 7ff71a7fc700 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2023-09-30T01:33:04.891 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.750+0000 7ff71a7fc700 10 monclient: dump: 2023-09-30T01:33:04.891 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:33:04.891 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:33:04.891 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:04.891 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:04.892 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:33:04.892 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:33:04.892 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:33:04.892 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:33:04.892 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:33:04.892 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:33:04.892 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:33:04.892 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:33:04.892 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:33:04.892 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:33:04.892 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:33:04.893 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:33:04.893 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.750+0000 7ff71a7fc700 10 monclient: _finish_auth 0 2023-09-30T01:33:04.893 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.750+0000 7ff71a7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:33:04.893 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.750+0000 7ff71a7fc700 1 -- 172.21.15.88:0/3609853028 <== mon.8 v2:172.21.15.88:3304/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff71c00f040 con 0x7ff724132450 2023-09-30T01:33:04.893 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.750+0000 7ff71a7fc700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:33:04.893 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.750+0000 7ff71a7fc700 1 -- 172.21.15.88:0/3609853028 <== mon.8 v2:172.21.15.88:3304/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7ff71c00cdc0 con 0x7ff724132450 2023-09-30T01:33:04.893 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.750+0000 7ff71a7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:33:04.893 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.750+0000 7ff71a7fc700 10 monclient: got monmap 1 from mon.e (according to old e1) 2023-09-30T01:33:04.893 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.750+0000 7ff71a7fc700 10 monclient: dump: 2023-09-30T01:33:04.893 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:33:04.893 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:33:04.893 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:04.894 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:04.894 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:33:04.894 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:33:04.894 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:33:04.894 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:33:04.894 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:33:04.894 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:33:04.894 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:33:04.894 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:33:04.894 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:33:04.894 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:33:04.894 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:33:04.895 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:33:04.895 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.750+0000 7ff728995700 5 monclient: authenticate success, global_id 4328 2023-09-30T01:33:04.895 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.750+0000 7ff728995700 10 monclient: _renew_subs 2023-09-30T01:33:04.895 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.750+0000 7ff728995700 10 monclient: _send_mon_message to mon.e at v2:172.21.15.88:3304/0 2023-09-30T01:33:04.895 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.750+0000 7ff728995700 1 -- 172.21.15.88:0/3609853028 --> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff7241354d0 con 0x7ff724132450 2023-09-30T01:33:04.895 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.750+0000 7ff72159a700 4 set_mon_vals no callback set 2023-09-30T01:33:04.895 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.750+0000 7ff728995700 10 monclient: _renew_subs 2023-09-30T01:33:04.895 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.750+0000 7ff728995700 10 monclient: _send_mon_message to mon.e at v2:172.21.15.88:3304/0 2023-09-30T01:33:04.895 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.750+0000 7ff728995700 1 -- 172.21.15.88:0/3609853028 --> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff724134200 con 0x7ff724132450 2023-09-30T01:33:04.895 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.750+0000 7ff728995700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:33:04.895 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.750+0000 7ff728995700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:33:04.896 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.750+0000 7ff728995700 10 monclient: _send_mon_message to mon.e at v2:172.21.15.88:3304/0 2023-09-30T01:33:04.896 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.750+0000 7ff71a7fc700 1 -- 172.21.15.88:0/3609853028 <== mon.8 v2:172.21.15.88:3304/0 4 ==== mgrmap(e 5) v1 ==== 40144+0+0 (secure 0 0 0) 0x7ff71c00c940 con 0x7ff724132450 2023-09-30T01:33:04.896 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.750+0000 7ff728995700 1 -- 172.21.15.88:0/3609853028 --> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff7240a3be0 con 0x7ff724132450 2023-09-30T01:33:04.896 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.750+0000 7ff71a7fc700 1 --2- 172.21.15.88:0/3609853028 >> v2:172.21.15.107:6812/29926 conn(0x7ff704033360 0x7ff704035810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:33:04.896 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.750+0000 7ff71a7fc700 1 -- 172.21.15.88:0/3609853028 <== mon.8 v2:172.21.15.88:3304/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 3502+0+0 (secure 0 0 0) 0x7ff71c043a70 con 0x7ff724132450 2023-09-30T01:33:04.896 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.750+0000 7ff72259c700 1 --2- 172.21.15.88:0/3609853028 >> v2:172.21.15.107:6812/29926 conn(0x7ff704033360 0x7ff704035810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:33:04.896 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.750+0000 7ff72259c700 10 monclient: get_auth_request con 0x7ff704033360 auth_method 0 2023-09-30T01:33:04.896 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.750+0000 7ff72259c700 1 --2- 172.21.15.88:0/3609853028 >> v2:172.21.15.107:6812/29926 conn(0x7ff704033360 0x7ff704035810 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7ff714009900 tx=0x7ff714016000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:33:04.896 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.754+0000 7f4a06ffd700 1 -- 172.21.15.88:0/2569192909 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f4a1401cb10 con 0x7f4a200a4b00 2023-09-30T01:33:04.897 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.754+0000 7f4a06ffd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:33:04.897 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.754+0000 7f4a06ffd700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:33:04.897 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.758+0000 7ff71a7fc700 1 -- 172.21.15.88:0/3609853028 <== mon.8 v2:172.21.15.88:3304/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7ff71c00cbf0 con 0x7ff724132450 2023-09-30T01:33:04.897 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.758+0000 7ff71a7fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:33:04.897 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.758+0000 7ff71a7fc700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:33:04.899 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.798+0000 7f0d5effd700 1 -- 172.21.15.88:0/3073637209 <== mon.3 v2:172.21.15.107:3301/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f0d6c01ccb0 con 0x7f0d7809d3f0 2023-09-30T01:33:04.899 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.798+0000 7f0d5effd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:33:04.899 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.798+0000 7f0d5effd700 10 monclient: got monmap 1 from mon.noname-g (according to old e1) 2023-09-30T01:33:04.899 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.798+0000 7f0d5effd700 10 monclient: dump: 2023-09-30T01:33:04.899 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:33:04.899 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:33:04.899 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:04.899 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:04.900 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:33:04.900 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:33:04.900 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:33:04.900 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:33:04.900 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:33:04.900 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:33:04.900 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:33:04.900 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:33:04.900 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:33:04.900 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:33:04.900 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:33:04.900 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:33:04.901 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.798+0000 7f0d5effd700 10 monclient: _finish_auth 0 2023-09-30T01:33:04.901 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.798+0000 7f0d5effd700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:33:04.901 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.798+0000 7f0d5effd700 1 -- 172.21.15.88:0/3073637209 <== mon.3 v2:172.21.15.107:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d6c01ce10 con 0x7f0d7809d3f0 2023-09-30T01:33:04.901 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.798+0000 7f0d7d6e5700 5 monclient: authenticate success, global_id 4251 2023-09-30T01:33:04.901 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.798+0000 7f0d7d6e5700 10 monclient: _renew_subs 2023-09-30T01:33:04.901 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.798+0000 7f0d7d6e5700 10 monclient: _send_mon_message to mon.g at v2:172.21.15.107:3301/0 2023-09-30T01:33:04.901 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.798+0000 7f0d7d6e5700 1 -- 172.21.15.88:0/3073637209 --> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d7806e980 con 0x7f0d7809d3f0 2023-09-30T01:33:04.901 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.798+0000 7f0d5effd700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:33:04.901 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.798+0000 7f0d7d6e5700 10 monclient: _renew_subs 2023-09-30T01:33:04.901 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.798+0000 7f0d7d6e5700 10 monclient: _send_mon_message to mon.g at v2:172.21.15.107:3301/0 2023-09-30T01:33:04.901 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.798+0000 7f0d7d6e5700 1 -- 172.21.15.88:0/3073637209 --> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d7806ee00 con 0x7f0d7809d3f0 2023-09-30T01:33:04.902 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.798+0000 7f0d74ff9700 4 set_mon_vals no callback set 2023-09-30T01:33:04.902 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.798+0000 7f0d5effd700 1 -- 172.21.15.88:0/3073637209 <== mon.3 v2:172.21.15.107:3301/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f0d6c01a8d0 con 0x7f0d7809d3f0 2023-09-30T01:33:04.902 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.798+0000 7f0d5effd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:33:04.902 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.798+0000 7f0d5effd700 10 monclient: got monmap 1 from mon.g (according to old e1) 2023-09-30T01:33:04.902 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.798+0000 7f0d5effd700 10 monclient: dump: 2023-09-30T01:33:04.902 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:33:04.902 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:33:04.902 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:04.902 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:04.902 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:33:04.902 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:33:04.903 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:33:04.903 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:33:04.903 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:33:04.903 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:33:04.903 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:33:04.903 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:33:04.903 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:33:04.903 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:33:04.903 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:33:04.903 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:33:04.903 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.798+0000 7f0d5effd700 1 -- 172.21.15.88:0/3073637209 <== mon.3 v2:172.21.15.107:3301/0 4 ==== mgrmap(e 5) v1 ==== 40144+0+0 (secure 0 0 0) 0x7f0d6c01a450 con 0x7f0d7809d3f0 2023-09-30T01:33:04.903 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.798+0000 7f0d5effd700 1 --2- 172.21.15.88:0/3073637209 >> v2:172.21.15.107:6812/29926 conn(0x7f0d600338a0 0x7f0d60035d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:33:04.903 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.798+0000 7f0d5effd700 1 -- 172.21.15.88:0/3073637209 <== mon.3 v2:172.21.15.107:3301/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 3502+0+0 (secure 0 0 0) 0x7f0d6c044740 con 0x7f0d7809d3f0 2023-09-30T01:33:04.904 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.798+0000 7f0d76ffd700 1 --2- 172.21.15.88:0/3073637209 >> v2:172.21.15.107:6812/29926 conn(0x7f0d600338a0 0x7f0d60035d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:33:04.904 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.798+0000 7f0d76ffd700 10 monclient: get_auth_request con 0x7f0d600338a0 auth_method 0 2023-09-30T01:33:04.904 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.802+0000 7f0d76ffd700 1 --2- 172.21.15.88:0/3073637209 >> v2:172.21.15.107:6812/29926 conn(0x7f0d600338a0 0x7f0d60035d50 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f0d68006aa0 tx=0x7f0d68008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:33:04.904 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.806+0000 7f0d7d6e5700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:33:04.904 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.806+0000 7f0d7d6e5700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:33:04.904 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.806+0000 7f0d7d6e5700 10 monclient: _send_mon_message to mon.g at v2:172.21.15.107:3301/0 2023-09-30T01:33:04.904 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.806+0000 7f0d7d6e5700 1 -- 172.21.15.88:0/3073637209 --> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0d780a18b0 con 0x7f0d7809d3f0 2023-09-30T01:33:04.904 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.814+0000 7f0d5effd700 1 -- 172.21.15.88:0/3073637209 <== mon.3 v2:172.21.15.107:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f0d780a18b0 con 0x7f0d7809d3f0 2023-09-30T01:33:04.904 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.814+0000 7f0d5effd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:33:04.904 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.814+0000 7f0d5effd700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:33:04.935 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.934+0000 7f1fd1bbe700 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 5}] 2023-09-30T01:33:04.936 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.934+0000 7f1fd1bbe700 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2023-09-30T01:33:04.936 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.934+0000 7f1fd1bbe700 10 monclient: _send_mon_message to mon.d at v2:172.21.15.88:3303/0 2023-09-30T01:33:04.936 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.934+0000 7f1fd1bbe700 1 -- 172.21.15.88:0/2208255062 --> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7f1fb0005630 con 0x7f1fcc09d6d0 2023-09-30T01:33:04.936 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.934+0000 7f1fc37fe700 1 -- 172.21.15.88:0/2208255062 <== mon.6 v2:172.21.15.88:3303/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f1fbc00f240 con 0x7f1fcc09d6d0 2023-09-30T01:33:04.936 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.934+0000 7f1fc37fe700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2023-09-30T01:33:04.936 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.934+0000 7f1fc37fe700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:33:04.937 INFO:teuthology.orchestra.run.smithi088.stdout:47244640257 2023-09-30T01:33:04.938 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.934+0000 7f1fd1bbe700 1 -- 172.21.15.88:0/2208255062 >> v2:172.21.15.107:6812/29926 conn(0x7f1fac033360 msgr2=0x7f1fac035810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:33:04.938 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.934+0000 7f1fd1bbe700 1 --2- 172.21.15.88:0/2208255062 >> v2:172.21.15.107:6812/29926 conn(0x7f1fac033360 0x7f1fac035810 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f1fc4001460 tx=0x7f1fc4007c40 comp rx=0 tx=0).stop 2023-09-30T01:33:04.938 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.934+0000 7f1fd1bbe700 10 monclient: shutdown 2023-09-30T01:33:04.938 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.934+0000 7f1fd1bbe700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:33:04.939 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.934+0000 7f1fd1bbe700 1 -- 172.21.15.88:0/2208255062 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7f1fcc09d6d0 msgr2=0x7f1fcc06bbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:33:04.939 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.934+0000 7f1fd1bbe700 1 --2- 172.21.15.88:0/2208255062 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7f1fcc09d6d0 0x7f1fcc06bbb0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f1fbc00ec00 tx=0x7f1fbc0091a0 comp rx=0 tx=0).stop 2023-09-30T01:33:04.939 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.934+0000 7f1fd1bbe700 1 -- 172.21.15.88:0/2208255062 shutdown_connections 2023-09-30T01:33:04.939 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.934+0000 7f1fd1bbe700 1 --2- 172.21.15.88:0/2208255062 >> v2:172.21.15.107:6812/29926 conn(0x7f1fac033360 0x7f1fac035810 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:04.939 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.934+0000 7f1fd1bbe700 1 --2- 172.21.15.88:0/2208255062 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7f1fcc134a60 0x7f1fcc06c0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:04.939 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.934+0000 7f1fd1bbe700 1 --2- 172.21.15.88:0/2208255062 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7f1fcc09d6d0 0x7f1fcc06bbb0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:04.939 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.934+0000 7f1fd1bbe700 1 --2- 172.21.15.88:0/2208255062 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1fcc09d300 0x7f1fcc094860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:04.939 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.934+0000 7f1fd1bbe700 1 -- 172.21.15.88:0/2208255062 >> 172.21.15.88:0/2208255062 conn(0x7f1fcc0a24c0 msgr2=0x7f1fcc1393c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:33:04.939 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.934+0000 7f1fd1bbe700 1 -- 172.21.15.88:0/2208255062 shutdown_connections 2023-09-30T01:33:04.939 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.934+0000 7f1fd1bbe700 1 -- 172.21.15.88:0/2208255062 wait complete. 2023-09-30T01:33:04.948 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640258 got 47244640257 for osd.5 2023-09-30T01:33:04.973 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.970+0000 7f4a25309700 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 0}] 2023-09-30T01:33:04.973 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.970+0000 7f4a25309700 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2023-09-30T01:33:04.973 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.970+0000 7f4a25309700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.88:3300/0 2023-09-30T01:33:04.973 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.970+0000 7f4a25309700 1 -- 172.21.15.88:0/2569192909 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7f4a20071960 con 0x7f4a200a4b00 2023-09-30T01:33:04.973 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.970+0000 7f4a06ffd700 1 -- 172.21.15.88:0/2569192909 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f4a1402ee60 con 0x7f4a200a4b00 2023-09-30T01:33:04.973 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.970+0000 7f4a06ffd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2023-09-30T01:33:04.973 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.970+0000 7f4a06ffd700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:33:04.974 INFO:teuthology.orchestra.run.smithi088.stdout:47244640259 2023-09-30T01:33:04.976 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.974+0000 7f4a25309700 1 -- 172.21.15.88:0/2569192909 >> v2:172.21.15.107:6812/29926 conn(0x7f4a0c0332c0 msgr2=0x7f4a0c035770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:33:04.977 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.974+0000 7f4a25309700 1 --2- 172.21.15.88:0/2569192909 >> v2:172.21.15.107:6812/29926 conn(0x7f4a0c0332c0 0x7f4a0c035770 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f4a10001410 tx=0x7f4a100096a0 comp rx=0 tx=0).stop 2023-09-30T01:33:04.977 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.974+0000 7f4a25309700 10 monclient: shutdown 2023-09-30T01:33:04.977 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.974+0000 7f4a25309700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:33:04.977 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.974+0000 7f4a25309700 1 -- 172.21.15.88:0/2569192909 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4a200a4b00 msgr2=0x7f4a20142dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:33:04.977 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.974+0000 7f4a25309700 1 --2- 172.21.15.88:0/2569192909 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4a200a4b00 0x7f4a20142dc0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f4a14011ad0 tx=0x7f4a14016000 comp rx=0 tx=0).stop 2023-09-30T01:33:04.977 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.974+0000 7f4a25309700 1 -- 172.21.15.88:0/2569192909 shutdown_connections 2023-09-30T01:33:04.977 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.974+0000 7f4a25309700 1 --2- 172.21.15.88:0/2569192909 >> v2:172.21.15.107:6812/29926 conn(0x7f4a0c0332c0 0x7f4a0c035770 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:04.977 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.974+0000 7f4a25309700 1 --2- 172.21.15.88:0/2569192909 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7f4a201333c0 0x7f4a20143300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:04.977 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.974+0000 7f4a25309700 1 --2- 172.21.15.88:0/2569192909 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4a200a4b00 0x7f4a20142dc0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:04.977 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.974+0000 7f4a25309700 1 --2- 172.21.15.88:0/2569192909 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7f4a200a3270 0x7f4a20142880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:04.977 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.974+0000 7f4a25309700 1 -- 172.21.15.88:0/2569192909 >> 172.21.15.88:0/2569192909 conn(0x7f4a200a67f0 msgr2=0x7f4a200a20e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:33:04.978 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.974+0000 7f4a25309700 1 -- 172.21.15.88:0/2569192909 shutdown_connections 2023-09-30T01:33:04.978 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:04.974+0000 7f4a25309700 1 -- 172.21.15.88:0/2569192909 wait complete. 2023-09-30T01:33:04.991 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640260 got 47244640259 for osd.0 2023-09-30T01:33:05.028 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:05.026+0000 7f00dd7fa700 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 3}] 2023-09-30T01:33:05.028 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:05.026+0000 7f00dd7fa700 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2023-09-30T01:33:05.028 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:05.026+0000 7f00dd7fa700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.88:3301/0 2023-09-30T01:33:05.028 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:05.026+0000 7f00dd7fa700 1 -- 172.21.15.88:0/3746327561 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7f00f8071960 con 0x7f00f8099df0 2023-09-30T01:33:05.029 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:05.026+0000 7f00df7fe700 1 -- 172.21.15.88:0/3746327561 <== mon.2 v2:172.21.15.88:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f00ec0117c0 con 0x7f00f8099df0 2023-09-30T01:33:05.029 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:05.026+0000 7f00df7fe700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2023-09-30T01:33:05.029 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:05.026+0000 7f00df7fe700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:33:05.029 INFO:teuthology.orchestra.run.smithi088.stdout:47244640258 2023-09-30T01:33:05.032 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:05.030+0000 7f00dd7fa700 1 -- 172.21.15.88:0/3746327561 >> v2:172.21.15.107:6812/29926 conn(0x7f00e4033360 msgr2=0x7f00e4035810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:33:05.032 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:05.030+0000 7f00dd7fa700 1 --2- 172.21.15.88:0/3746327561 >> v2:172.21.15.107:6812/29926 conn(0x7f00e4033360 0x7f00e4035810 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f00e8009db0 tx=0x7f00e80091a0 comp rx=0 tx=0).stop 2023-09-30T01:33:05.033 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:05.030+0000 7f00dd7fa700 10 monclient: shutdown 2023-09-30T01:33:05.033 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:05.030+0000 7f00dd7fa700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:33:05.033 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:05.030+0000 7f00dd7fa700 1 -- 172.21.15.88:0/3746327561 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f00f8099df0 msgr2=0x7f00f8119fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:33:05.033 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:05.030+0000 7f00dd7fa700 1 --2- 172.21.15.88:0/3746327561 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f00f8099df0 0x7f00f8119fa0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f00ec009fd0 tx=0x7f00ec0091a0 comp rx=0 tx=0).stop 2023-09-30T01:33:05.034 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:05.030+0000 7f00dd7fa700 1 -- 172.21.15.88:0/3746327561 shutdown_connections 2023-09-30T01:33:05.034 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:05.030+0000 7f00dd7fa700 1 --2- 172.21.15.88:0/3746327561 >> v2:172.21.15.107:6812/29926 conn(0x7f00e4033360 0x7f00e4035810 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:05.034 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:05.030+0000 7f00dd7fa700 1 --2- 172.21.15.88:0/3746327561 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f00f8133d90 0x7f00f811a4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:05.034 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:05.030+0000 7f00dd7fa700 1 --2- 172.21.15.88:0/3746327561 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f00f8099df0 0x7f00f8119fa0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:05.034 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:05.030+0000 7f00dd7fa700 1 --2- 172.21.15.88:0/3746327561 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f00f8062c70 0x7f00f8119a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:05.034 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:05.030+0000 7f00dd7fa700 1 -- 172.21.15.88:0/3746327561 >> 172.21.15.88:0/3746327561 conn(0x7f00f809fe10 msgr2=0x7f00f80a1a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:33:05.034 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:05.030+0000 7f00dd7fa700 1 -- 172.21.15.88:0/3746327561 shutdown_connections 2023-09-30T01:33:05.034 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:05.030+0000 7f00dd7fa700 1 -- 172.21.15.88:0/3746327561 wait complete. 2023-09-30T01:33:05.048 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640259 got 47244640258 for osd.3 2023-09-30T01:33:05.055 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:05.054+0000 7f1c6875d700 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 4}] 2023-09-30T01:33:05.055 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:05.054+0000 7f1c6875d700 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 4}] 2023-09-30T01:33:05.055 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:05.054+0000 7f1c6875d700 10 monclient: _send_mon_message to mon.i at v2:172.21.15.107:3303/0 2023-09-30T01:33:05.055 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:05.054+0000 7f1c6875d700 1 -- 172.21.15.88:0/728105367 --> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7f1c48005630 con 0x7f1c6009c160 2023-09-30T01:33:05.056 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:05.054+0000 7f1c567fc700 1 -- 172.21.15.88:0/728105367 <== mon.7 v2:172.21.15.107:3303/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f1c5c011350 con 0x7f1c6009c160 2023-09-30T01:33:05.056 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:05.054+0000 7f1c567fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 4}] 2023-09-30T01:33:05.056 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:05.054+0000 7f1c567fc700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:33:05.056 INFO:teuthology.orchestra.run.smithi088.stdout:47244640257 2023-09-30T01:33:05.060 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:05.058+0000 7f1c6875d700 1 -- 172.21.15.88:0/728105367 >> v2:172.21.15.107:6812/29926 conn(0x7f1c44033360 msgr2=0x7f1c44035810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:33:05.060 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:05.058+0000 7f1c6875d700 1 --2- 172.21.15.88:0/728105367 >> v2:172.21.15.107:6812/29926 conn(0x7f1c44033360 0x7f1c44035810 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f1c6006ed70 tx=0x7f1c5800d040 comp rx=0 tx=0).stop 2023-09-30T01:33:05.060 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:05.058+0000 7f1c6875d700 10 monclient: shutdown 2023-09-30T01:33:05.060 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:05.058+0000 7f1c6875d700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:33:05.060 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:05.058+0000 7f1c6875d700 1 -- 172.21.15.88:0/728105367 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7f1c6009c160 msgr2=0x7f1c6012e8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:33:05.060 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:05.058+0000 7f1c6875d700 1 --2- 172.21.15.88:0/728105367 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7f1c6009c160 0x7f1c6012e8a0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f1c5c00ecb0 tx=0x7f1c5c009120 comp rx=0 tx=0).stop 2023-09-30T01:33:05.060 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:05.058+0000 7f1c6875d700 1 -- 172.21.15.88:0/728105367 shutdown_connections 2023-09-30T01:33:05.060 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:05.058+0000 7f1c6875d700 1 --2- 172.21.15.88:0/728105367 >> v2:172.21.15.107:6812/29926 conn(0x7f1c44033360 0x7f1c44035810 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:05.060 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:05.058+0000 7f1c6875d700 1 --2- 172.21.15.88:0/728105367 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1c6009dc10 0x7f1c6013e300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:05.060 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:05.058+0000 7f1c6875d700 1 --2- 172.21.15.88:0/728105367 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7f1c6009d300 0x7f1c6013ddc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:05.061 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:05.058+0000 7f1c6875d700 1 --2- 172.21.15.88:0/728105367 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7f1c6009c160 0x7f1c6012e8a0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:05.061 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:05.058+0000 7f1c6875d700 1 -- 172.21.15.88:0/728105367 >> 172.21.15.88:0/728105367 conn(0x7f1c600a24c0 msgr2=0x7f1c60131920 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:33:05.061 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:05.058+0000 7f1c6875d700 1 -- 172.21.15.88:0/728105367 shutdown_connections 2023-09-30T01:33:05.061 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:05.058+0000 7f1c6875d700 1 -- 172.21.15.88:0/728105367 wait complete. 2023-09-30T01:33:05.073 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:05.070+0000 7ff728995700 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 1}] 2023-09-30T01:33:05.073 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:05.070+0000 7ff728995700 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 1}] 2023-09-30T01:33:05.073 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:05.070+0000 7ff728995700 10 monclient: _send_mon_message to mon.e at v2:172.21.15.88:3304/0 2023-09-30T01:33:05.073 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:05.070+0000 7ff728995700 1 -- 172.21.15.88:0/3609853028 --> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7ff724071960 con 0x7ff724132450 2023-09-30T01:33:05.073 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:05.070+0000 7ff71a7fc700 1 -- 172.21.15.88:0/3609853028 <== mon.8 v2:172.21.15.88:3304/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7ff71c0133e0 con 0x7ff724132450 2023-09-30T01:33:05.074 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:05.070+0000 7ff71a7fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 1}] 2023-09-30T01:33:05.074 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:05.070+0000 7ff71a7fc700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:33:05.074 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640258 got 47244640257 for osd.4 2023-09-30T01:33:05.075 INFO:teuthology.orchestra.run.smithi088.stdout:47244640258 2023-09-30T01:33:05.077 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:05.074+0000 7ff728995700 1 -- 172.21.15.88:0/3609853028 >> v2:172.21.15.107:6812/29926 conn(0x7ff704033360 msgr2=0x7ff704035810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:33:05.077 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:05.074+0000 7ff728995700 1 --2- 172.21.15.88:0/3609853028 >> v2:172.21.15.107:6812/29926 conn(0x7ff704033360 0x7ff704035810 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7ff714009900 tx=0x7ff714016000 comp rx=0 tx=0).stop 2023-09-30T01:33:05.077 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:05.074+0000 7ff728995700 10 monclient: shutdown 2023-09-30T01:33:05.077 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:05.074+0000 7ff728995700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:33:05.078 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:05.074+0000 7ff728995700 1 -- 172.21.15.88:0/3609853028 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7ff724132450 msgr2=0x7ff7241328a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:33:05.078 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:05.074+0000 7ff728995700 1 --2- 172.21.15.88:0/3609853028 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7ff724132450 0x7ff7241328a0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7ff71c009db0 tx=0x7ff71c00b040 comp rx=0 tx=0).stop 2023-09-30T01:33:05.078 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:05.074+0000 7ff728995700 1 -- 172.21.15.88:0/3609853028 shutdown_connections 2023-09-30T01:33:05.078 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:05.074+0000 7ff728995700 1 --2- 172.21.15.88:0/3609853028 >> v2:172.21.15.107:6812/29926 conn(0x7ff704033360 0x7ff704035810 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:05.078 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:05.074+0000 7ff728995700 1 --2- 172.21.15.88:0/3609853028 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7ff724132450 0x7ff7241328a0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:05.078 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:05.074+0000 7ff728995700 1 --2- 172.21.15.88:0/3609853028 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7ff72409eef0 0x7ff724131f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:05.078 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:05.074+0000 7ff728995700 1 --2- 172.21.15.88:0/3609853028 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7ff72406ede0 0x7ff72409d370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:05.078 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:05.074+0000 7ff728995700 1 -- 172.21.15.88:0/3609853028 >> 172.21.15.88:0/3609853028 conn(0x7ff724097a40 msgr2=0x7ff724130120 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:33:05.078 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:05.074+0000 7ff728995700 1 -- 172.21.15.88:0/3609853028 shutdown_connections 2023-09-30T01:33:05.078 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:05.074+0000 7ff728995700 1 -- 172.21.15.88:0/3609853028 wait complete. 2023-09-30T01:33:05.089 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640259 got 47244640258 for osd.1 2023-09-30T01:33:05.090 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:05.086+0000 7f0d7d6e5700 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 2}] 2023-09-30T01:33:05.090 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:05.086+0000 7f0d7d6e5700 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2023-09-30T01:33:05.090 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:05.086+0000 7f0d7d6e5700 10 monclient: _send_mon_message to mon.g at v2:172.21.15.107:3301/0 2023-09-30T01:33:05.090 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:05.086+0000 7f0d7d6e5700 1 -- 172.21.15.88:0/3073637209 --> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7f0d780a18b0 con 0x7f0d7809d3f0 2023-09-30T01:33:05.090 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:05.086+0000 7f0d5effd700 1 -- 172.21.15.88:0/3073637209 <== mon.3 v2:172.21.15.107:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f0d780a18b0 con 0x7f0d7809d3f0 2023-09-30T01:33:05.090 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:05.086+0000 7f0d5effd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2023-09-30T01:33:05.090 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:05.086+0000 7f0d5effd700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:33:05.090 INFO:teuthology.orchestra.run.smithi088.stdout:47244640259 2023-09-30T01:33:05.092 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:05.090+0000 7f0d7d6e5700 1 -- 172.21.15.88:0/3073637209 >> v2:172.21.15.107:6812/29926 conn(0x7f0d600338a0 msgr2=0x7f0d60035d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:33:05.092 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:05.090+0000 7f0d7d6e5700 1 --2- 172.21.15.88:0/3073637209 >> v2:172.21.15.107:6812/29926 conn(0x7f0d600338a0 0x7f0d60035d50 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f0d68006aa0 tx=0x7f0d68008040 comp rx=0 tx=0).stop 2023-09-30T01:33:05.092 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:05.090+0000 7f0d7d6e5700 10 monclient: shutdown 2023-09-30T01:33:05.093 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:05.090+0000 7f0d7d6e5700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:33:05.093 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:05.090+0000 7f0d7d6e5700 1 -- 172.21.15.88:0/3073637209 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7f0d7809d3f0 msgr2=0x7f0d7806d0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:33:05.093 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:05.090+0000 7f0d7d6e5700 1 --2- 172.21.15.88:0/3073637209 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7f0d7809d3f0 0x7f0d7806d0b0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f0d7809a8b0 tx=0x7f0d6c017000 comp rx=0 tx=0).stop 2023-09-30T01:33:05.093 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:05.090+0000 7f0d7d6e5700 1 -- 172.21.15.88:0/3073637209 shutdown_connections 2023-09-30T01:33:05.093 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:05.090+0000 7f0d7d6e5700 1 --2- 172.21.15.88:0/3073637209 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7f0d7809d3f0 0x7f0d7806d0b0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:05.093 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:05.090+0000 7f0d7d6e5700 1 --2- 172.21.15.88:0/3073637209 >> v2:172.21.15.107:6812/29926 conn(0x7f0d600338a0 0x7f0d60035d50 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:05.093 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:05.090+0000 7f0d7d6e5700 1 --2- 172.21.15.88:0/3073637209 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f0d7809c620 0x7f0d7806cb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:05.093 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:05.090+0000 7f0d7d6e5700 1 --2- 172.21.15.88:0/3073637209 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0d7809c250 0x7f0d7806c630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:05.093 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:05.090+0000 7f0d7d6e5700 1 -- 172.21.15.88:0/3073637209 >> 172.21.15.88:0/3073637209 conn(0x7f0d780a24c0 msgr2=0x7f0d78131b00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:33:05.093 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:05.090+0000 7f0d7d6e5700 1 -- 172.21.15.88:0/3073637209 shutdown_connections 2023-09-30T01:33:05.093 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:05.090+0000 7f0d7d6e5700 1 -- 172.21.15.88:0/3073637209 wait complete. 2023-09-30T01:33:05.102 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640260 got 47244640259 for osd.2 2023-09-30T01:33:05.949 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2023-09-30T01:33:05.991 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2023-09-30T01:33:06.048 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2023-09-30T01:33:06.063 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.062+0000 7f4ea88e6700 10 monclient: get_monmap_and_config 2023-09-30T01:33:06.064 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.062+0000 7f4ea88e6700 10 monclient: build_initial_monmap 2023-09-30T01:33:06.064 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.062+0000 7f4ea88e6700 10 monclient: monmap: 2023-09-30T01:33:06.064 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2023-09-30T01:33:06.064 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:33:06.064 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:33:06.064745+0000 2023-09-30T01:33:06.064 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:33:06.064745+0000 2023-09-30T01:33:06.064 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2023-09-30T01:33:06.064 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:33:06.065 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a-legacy 2023-09-30T01:33:06.065 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.107:6789/0 mon.noname-f-legacy 2023-09-30T01:33:06.065 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.noname-b 2023-09-30T01:33:06.065 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.noname-g 2023-09-30T01:33:06.065 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.noname-c 2023-09-30T01:33:06.065 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.noname-h 2023-09-30T01:33:06.065 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.noname-d 2023-09-30T01:33:06.065 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.noname-i 2023-09-30T01:33:06.066 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.noname-e 2023-09-30T01:33:06.066 INFO:teuthology.orchestra.run.smithi088.stderr:9: v2:172.21.15.88:3300/0 mon.noname-a 2023-09-30T01:33:06.066 INFO:teuthology.orchestra.run.smithi088.stderr:10: v2:172.21.15.107:3300/0 mon.noname-f 2023-09-30T01:33:06.066 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:33:06.073 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.062+0000 7f4ea88e6700 1 Processor -- start 2023-09-30T01:33:06.073 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.062+0000 7f4ea88e6700 1 -- start start 2023-09-30T01:33:06.073 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.062+0000 7f4ea88e6700 10 monclient: init 2023-09-30T01:33:06.073 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.062+0000 7f4ea88e6700 10 monclient: _reopen_session rank -1 2023-09-30T01:33:06.073 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.062+0000 7f4ea88e6700 10 monclient: _add_conns ranks=[8,2,6,4,0,3,10,9,5,7,1] 2023-09-30T01:33:06.074 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.062+0000 7f4ea88e6700 1 --2- >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f4ea409d580 0x7f4ea406e470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:33:06.074 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.062+0000 7f4ea88e6700 10 monclient(hunting): picked mon.noname-e con 0x7f4ea409d580 addr [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] 2023-09-30T01:33:06.074 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.062+0000 7f4ea88e6700 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4ea409d950 0x7f4ea409dd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:33:06.074 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.062+0000 7f4ea88e6700 10 monclient(hunting): picked mon.noname-b con 0x7f4ea409d950 addr [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] 2023-09-30T01:33:06.074 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.062+0000 7f4ea88e6700 1 --2- >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7f4ea4131370 0x7f4ea41317a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:33:06.074 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.062+0000 7f4ea88e6700 10 monclient(hunting): picked mon.noname-d con 0x7f4ea4131370 addr [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] 2023-09-30T01:33:06.074 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.062+0000 7f4ea88e6700 10 monclient(hunting): start opening mon connection 2023-09-30T01:33:06.074 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.062+0000 7f4ea88e6700 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ea406d7b0 con 0x7f4ea409d950 2023-09-30T01:33:06.074 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.062+0000 7f4ea88e6700 10 monclient(hunting): start opening mon connection 2023-09-30T01:33:06.074 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.062+0000 7f4ea88e6700 1 -- --> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] -- mon_getmap magic: 0 v1 -- 0x7f4ea406d4d0 con 0x7f4ea4131370 2023-09-30T01:33:06.074 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.062+0000 7f4ea88e6700 10 monclient(hunting): start opening mon connection 2023-09-30T01:33:06.075 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.062+0000 7f4ea88e6700 1 -- --> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] -- mon_getmap magic: 0 v1 -- 0x7f4ea409eb40 con 0x7f4ea409d580 2023-09-30T01:33:06.075 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.062+0000 7f4ea88e6700 10 monclient(hunting): _renew_subs 2023-09-30T01:33:06.075 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.062+0000 7f4ea88e6700 10 monclient(hunting): authenticate will time out at 858.102827s 2023-09-30T01:33:06.075 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.062+0000 7f4ea37fe700 1 --2- >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f4ea409d580 0x7f4ea406e470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:33:06.075 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.062+0000 7f4ea3fff700 1 --2- >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7f4ea4131370 0x7f4ea41317a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:33:06.075 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.062+0000 7f4ea37fe700 1 --2- >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f4ea409d580 0x7f4ea406e470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3304/0 says I am v2:172.21.15.88:36752/0 (socket says 172.21.15.88:36752) 2023-09-30T01:33:06.075 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.062+0000 7f4ea37fe700 1 -- 172.21.15.88:0/727099684 learned_addr learned my addr 172.21.15.88:0/727099684 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-09-30T01:33:06.075 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.062+0000 7f4ea37fe700 10 monclient(hunting): get_auth_request con 0x7f4ea409d580 auth_method 0 2023-09-30T01:33:06.075 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.062+0000 7f4ea37fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:33:06.075 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.062+0000 7f4ea37fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:33:06.075 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.062+0000 7f4ea37fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:33:06.076 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.062+0000 7f4ea3fff700 10 monclient(hunting): get_auth_request con 0x7f4ea4131370 auth_method 0 2023-09-30T01:33:06.076 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.062+0000 7f4ea3fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:33:06.076 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.062+0000 7f4ea3fff700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:33:06.076 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.062+0000 7f4ea3fff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:33:06.076 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.062+0000 7f4ea2ffd700 1 --2- 172.21.15.88:0/727099684 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4ea409d950 0x7f4ea409dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:33:06.076 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.062+0000 7f4ea2ffd700 10 monclient(hunting): get_auth_request con 0x7f4ea409d950 auth_method 0 2023-09-30T01:33:06.076 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.062+0000 7f4ea2ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:33:06.076 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.062+0000 7f4ea2ffd700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:33:06.076 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.062+0000 7f4ea2ffd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:33:06.076 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.062+0000 7f4ea37fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:33:06.077 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.062+0000 7f4ea37fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:33:06.077 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.062+0000 7f4ea37fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:33:06.077 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.062+0000 7f4ea2ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:33:06.077 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.062+0000 7f4ea2ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:33:06.077 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.062+0000 7f4ea2ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:33:06.077 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.062+0000 7f4ea3fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:33:06.077 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.062+0000 7f4ea3fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:33:06.077 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.062+0000 7f4ea3fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:33:06.077 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.062+0000 7f4ea3fff700 10 monclient(hunting): handle_auth_done global_id 4263 payload 274 2023-09-30T01:33:06.077 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.062+0000 7f4ea3fff700 1 -- 172.21.15.88:0/727099684 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f4ea409d580 msgr2=0x7f4ea406e470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:33:06.077 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.062+0000 7f4ea3fff700 1 --2- 172.21.15.88:0/727099684 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f4ea409d580 0x7f4ea406e470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:06.078 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.062+0000 7f4ea3fff700 1 -- 172.21.15.88:0/727099684 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4ea409d950 msgr2=0x7f4ea409dd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:33:06.078 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.062+0000 7f4ea3fff700 1 --2- 172.21.15.88:0/727099684 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4ea409d950 0x7f4ea409dd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:06.078 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.062+0000 7f4ea3fff700 10 monclient: _finish_hunting 0 2023-09-30T01:33:06.078 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.062+0000 7f4ea3fff700 1 monclient: found mon.noname-d 2023-09-30T01:33:06.078 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.062+0000 7f4ea3fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:33:06.078 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.062+0000 7f4ea3fff700 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.88:3303/0 2023-09-30T01:33:06.078 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.062+0000 7f4ea3fff700 1 -- 172.21.15.88:0/727099684 --> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ea404bb20 con 0x7f4ea4131370 2023-09-30T01:33:06.078 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.062+0000 7f4ea2ffd700 1 --2- 172.21.15.88:0/727099684 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4ea409d950 0x7f4ea409dd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:33:06.078 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.062+0000 7f4ea37fe700 1 --2- 172.21.15.88:0/727099684 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f4ea409d580 0x7f4ea406e470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:33:06.078 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.066+0000 7f4ea3fff700 1 --2- 172.21.15.88:0/727099684 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7f4ea4131370 0x7f4ea41317a0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f4e9c009910 tx=0x7f4e9c0091a0 comp rx=0 tx=0).ready entity=mon.6 client_cookie=ed9e8ecb52eb2f33 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:33:06.078 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.066+0000 7f4ea27fc700 1 -- 172.21.15.88:0/727099684 <== mon.6 v2:172.21.15.88:3303/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f4e9c011ab0 con 0x7f4ea4131370 2023-09-30T01:33:06.078 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.066+0000 7f4ea27fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:33:06.079 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.066+0000 7f4ea27fc700 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2023-09-30T01:33:06.079 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.066+0000 7f4ea27fc700 10 monclient: dump: 2023-09-30T01:33:06.079 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:33:06.079 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:33:06.079 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:06.079 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:06.079 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:33:06.079 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:33:06.079 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:33:06.079 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:33:06.079 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:33:06.079 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:33:06.080 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:33:06.080 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:33:06.080 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:33:06.080 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:33:06.080 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:33:06.080 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:33:06.080 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.066+0000 7f4ea27fc700 10 monclient: _finish_auth 0 2023-09-30T01:33:06.080 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.066+0000 7f4ea27fc700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:33:06.080 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.066+0000 7f4ea27fc700 1 -- 172.21.15.88:0/727099684 <== mon.6 v2:172.21.15.88:3303/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e9c011c30 con 0x7f4ea4131370 2023-09-30T01:33:06.080 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.066+0000 7f4ea88e6700 5 monclient: authenticate success, global_id 4263 2023-09-30T01:33:06.080 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.066+0000 7f4ea88e6700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:33:06.080 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.066+0000 7f4ea27fc700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:33:06.080 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.066+0000 7f4ea88e6700 10 monclient: get_monmap_and_config success 2023-09-30T01:33:06.081 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.066+0000 7f4ea27fc700 1 -- 172.21.15.88:0/727099684 <== mon.6 v2:172.21.15.88:3303/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f4e9c011da0 con 0x7f4ea4131370 2023-09-30T01:33:06.081 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.066+0000 7f4ea27fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:33:06.081 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.066+0000 7f4ea88e6700 4 set_mon_vals no callback set 2023-09-30T01:33:06.081 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.066+0000 7f4ea27fc700 10 monclient: got monmap 1 from mon.d (according to old e1) 2023-09-30T01:33:06.081 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.066+0000 7f4ea27fc700 10 monclient: dump: 2023-09-30T01:33:06.081 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:33:06.081 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:33:06.081 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:06.081 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:06.081 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:33:06.081 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:33:06.081 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:33:06.082 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:33:06.082 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:33:06.082 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:33:06.082 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:33:06.082 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:33:06.082 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:33:06.082 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:33:06.082 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:33:06.082 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:33:06.082 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.066+0000 7f4ea88e6700 10 monclient: shutdown 2023-09-30T01:33:06.082 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.066+0000 7f4ea88e6700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:33:06.082 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.066+0000 7f4ea88e6700 1 -- 172.21.15.88:0/727099684 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7f4ea4131370 msgr2=0x7f4ea41317a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:33:06.083 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.066+0000 7f4ea88e6700 1 --2- 172.21.15.88:0/727099684 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7f4ea4131370 0x7f4ea41317a0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f4e9c009910 tx=0x7f4e9c0091a0 comp rx=0 tx=0).stop 2023-09-30T01:33:06.083 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.066+0000 7f4ea88e6700 1 -- 172.21.15.88:0/727099684 shutdown_connections 2023-09-30T01:33:06.083 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.066+0000 7f4ea88e6700 1 --2- 172.21.15.88:0/727099684 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7f4ea4131370 0x7f4ea41317a0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:06.083 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.066+0000 7f4ea88e6700 1 --2- 172.21.15.88:0/727099684 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4ea409d950 0x7f4ea409dd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:06.083 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.066+0000 7f4ea88e6700 1 --2- 172.21.15.88:0/727099684 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f4ea409d580 0x7f4ea406e470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:06.083 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.066+0000 7f4ea88e6700 1 -- 172.21.15.88:0/727099684 >> 172.21.15.88:0/727099684 conn(0x7f4ea4063610 msgr2=0x7f4ea40a2b90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:33:06.083 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.066+0000 7f4ea88e6700 1 -- 172.21.15.88:0/727099684 shutdown_connections 2023-09-30T01:33:06.083 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.066+0000 7f4ea88e6700 1 -- 172.21.15.88:0/727099684 wait complete. 2023-09-30T01:33:06.083 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.066+0000 7f4ea88e6700 10 monclient: build_initial_monmap 2023-09-30T01:33:06.083 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.066+0000 7f4ea88e6700 10 monclient: monmap: 2023-09-30T01:33:06.083 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2023-09-30T01:33:06.084 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:33:06.084 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 0.000000 2023-09-30T01:33:06.084 INFO:teuthology.orchestra.run.smithi088.stderr:created 0.000000 2023-09-30T01:33:06.084 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2023-09-30T01:33:06.084 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 1 2023-09-30T01:33:06.084 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.noname-a 2023-09-30T01:33:06.084 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.noname-f 2023-09-30T01:33:06.084 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.noname-b 2023-09-30T01:33:06.084 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.noname-g 2023-09-30T01:33:06.084 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.noname-c 2023-09-30T01:33:06.084 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.noname-h 2023-09-30T01:33:06.084 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.noname-d 2023-09-30T01:33:06.084 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.noname-i 2023-09-30T01:33:06.085 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.noname-e 2023-09-30T01:33:06.085 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:33:06.085 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.066+0000 7f4ea88e6700 1 Processor -- start 2023-09-30T01:33:06.085 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.066+0000 7f4ea88e6700 1 -- start start 2023-09-30T01:33:06.085 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.066+0000 7f4ea88e6700 10 monclient: init 2023-09-30T01:33:06.085 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.066+0000 7f4ea88e6700 10 monclient: _reopen_session rank -1 2023-09-30T01:33:06.085 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.066+0000 7f4ea88e6700 10 monclient: _add_conns ranks=[3,7,5,6,4,1,8,2,0] 2023-09-30T01:33:06.085 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.066+0000 7f4ea88e6700 1 --2- >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7f4ea409d580 0x7f4ea411a1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:33:06.085 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.066+0000 7f4ea88e6700 10 monclient(hunting): picked mon.noname-g con 0x7f4ea409d580 addr [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] 2023-09-30T01:33:06.085 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.066+0000 7f4ea88e6700 1 --2- >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7f4ea409d950 0x7f4ea411a700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:33:06.085 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.066+0000 7f4ea88e6700 10 monclient(hunting): picked mon.noname-i con 0x7f4ea409d950 addr [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] 2023-09-30T01:33:06.085 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.066+0000 7f4ea88e6700 1 --2- >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7f4ea4131370 0x7f4ea411ac40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:33:06.085 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.066+0000 7f4ea88e6700 10 monclient(hunting): picked mon.noname-h con 0x7f4ea4131370 addr [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] 2023-09-30T01:33:06.086 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.066+0000 7f4ea88e6700 10 monclient(hunting): start opening mon connection 2023-09-30T01:33:06.086 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.066+0000 7f4ea88e6700 1 -- --> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ea411c460 con 0x7f4ea409d580 2023-09-30T01:33:06.086 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.066+0000 7f4ea88e6700 10 monclient(hunting): start opening mon connection 2023-09-30T01:33:06.086 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.066+0000 7f4ea88e6700 1 -- --> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] -- mon_getmap magic: 0 v1 -- 0x7f4ea411d250 con 0x7f4ea4131370 2023-09-30T01:33:06.086 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.066+0000 7f4ea88e6700 10 monclient(hunting): start opening mon connection 2023-09-30T01:33:06.086 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.066+0000 7f4ea88e6700 1 -- --> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] -- mon_getmap magic: 0 v1 -- 0x7f4ea411d530 con 0x7f4ea409d950 2023-09-30T01:33:06.086 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.066+0000 7f4ea88e6700 10 monclient(hunting): _renew_subs 2023-09-30T01:33:06.086 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.066+0000 7f4ea88e6700 10 monclient(hunting): authenticate will time out at 858.107874s 2023-09-30T01:33:06.086 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.070+0000 7f4ea37fe700 1 --2- >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7f4ea409d580 0x7f4ea411a1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:33:06.086 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.070+0000 7f4ea2ffd700 1 --2- >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7f4ea409d950 0x7f4ea411a700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:33:06.086 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.070+0000 7f4ea3fff700 1 --2- >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7f4ea4131370 0x7f4ea411ac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:33:06.087 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.070+0000 7f4ea2ffd700 1 --2- >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7f4ea409d950 0x7f4ea411a700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3303/0 says I am v2:172.21.15.88:47104/0 (socket says 172.21.15.88:47104) 2023-09-30T01:33:06.087 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.070+0000 7f4ea2ffd700 1 -- 172.21.15.88:0/3240982109 learned_addr learned my addr 172.21.15.88:0/3240982109 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-09-30T01:33:06.087 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.070+0000 7f4ea2ffd700 10 monclient(hunting): get_auth_request con 0x7f4ea409d950 auth_method 0 2023-09-30T01:33:06.087 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.070+0000 7f4ea2ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:33:06.087 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.070+0000 7f4ea2ffd700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:33:06.087 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.070+0000 7f4ea2ffd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:33:06.087 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.070+0000 7f4ea37fe700 10 monclient(hunting): get_auth_request con 0x7f4ea409d580 auth_method 0 2023-09-30T01:33:06.087 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.070+0000 7f4ea37fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:33:06.087 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.070+0000 7f4ea37fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:33:06.087 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.070+0000 7f4ea37fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:33:06.087 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.070+0000 7f4ea3fff700 10 monclient(hunting): get_auth_request con 0x7f4ea4131370 auth_method 0 2023-09-30T01:33:06.087 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.070+0000 7f4ea3fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:33:06.087 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.070+0000 7f4ea3fff700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:33:06.088 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.070+0000 7f4ea3fff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:33:06.088 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.070+0000 7f4ea37fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:33:06.088 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.070+0000 7f4ea37fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:33:06.088 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.070+0000 7f4ea37fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:33:06.088 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.070+0000 7f4ea2ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:33:06.088 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.070+0000 7f4ea2ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:33:06.088 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.070+0000 7f4ea2ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:33:06.088 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.070+0000 7f4ea3fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:33:06.088 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.070+0000 7f4ea3fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:33:06.088 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.070+0000 7f4ea3fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:33:06.088 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.070+0000 7f4ea2ffd700 10 monclient(hunting): handle_auth_done global_id 4318 payload 824 2023-09-30T01:33:06.088 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.070+0000 7f4ea2ffd700 1 -- 172.21.15.88:0/3240982109 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7f4ea4131370 msgr2=0x7f4ea411ac40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:33:06.089 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.070+0000 7f4ea2ffd700 1 --2- 172.21.15.88:0/3240982109 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7f4ea4131370 0x7f4ea411ac40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:06.089 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.070+0000 7f4ea2ffd700 1 -- 172.21.15.88:0/3240982109 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7f4ea409d580 msgr2=0x7f4ea411a1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:33:06.089 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.070+0000 7f4ea2ffd700 1 --2- 172.21.15.88:0/3240982109 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7f4ea409d580 0x7f4ea411a1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:06.089 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.070+0000 7f4ea2ffd700 10 monclient: _finish_hunting 0 2023-09-30T01:33:06.089 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.070+0000 7f4ea2ffd700 1 monclient: found mon.noname-i 2023-09-30T01:33:06.089 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.070+0000 7f4ea2ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:33:06.089 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.070+0000 7f4ea2ffd700 10 monclient: _send_mon_message to mon.noname-i at v2:172.21.15.107:3303/0 2023-09-30T01:33:06.089 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.070+0000 7f4ea2ffd700 1 -- 172.21.15.88:0/3240982109 --> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e8c009ae0 con 0x7f4ea409d950 2023-09-30T01:33:06.089 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.070+0000 7f4ea3fff700 1 --2- 172.21.15.88:0/3240982109 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7f4ea4131370 0x7f4ea411ac40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:33:06.089 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.070+0000 7f4ea37fe700 1 --2- 172.21.15.88:0/3240982109 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7f4ea409d580 0x7f4ea411a1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:33:06.089 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.070+0000 7f4ea2ffd700 1 --2- 172.21.15.88:0/3240982109 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7f4ea409d950 0x7f4ea411a700 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f4e9400ec80 tx=0x7f4e940091a0 comp rx=0 tx=0).ready entity=mon.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:33:06.089 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.070+0000 7f4e9b7fe700 1 -- 172.21.15.88:0/3240982109 <== mon.7 v2:172.21.15.107:3303/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f4e9400ccd0 con 0x7f4ea409d950 2023-09-30T01:33:06.090 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.070+0000 7f4e9b7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:33:06.090 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.070+0000 7f4e9b7fe700 10 monclient: got monmap 1 from mon.noname-i (according to old e1) 2023-09-30T01:33:06.090 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.070+0000 7f4e9b7fe700 10 monclient: dump: 2023-09-30T01:33:06.090 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:33:06.090 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:33:06.090 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:06.090 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:06.090 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:33:06.090 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:33:06.090 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:33:06.090 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:33:06.090 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:33:06.091 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:33:06.091 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:33:06.091 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:33:06.091 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:33:06.091 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:33:06.091 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:33:06.091 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:33:06.091 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.070+0000 7f4e9b7fe700 10 monclient: _finish_auth 0 2023-09-30T01:33:06.091 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.070+0000 7f4e9b7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:33:06.091 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.070+0000 7f4e9b7fe700 1 -- 172.21.15.88:0/3240982109 <== mon.7 v2:172.21.15.107:3303/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e9400ce30 con 0x7f4ea409d950 2023-09-30T01:33:06.091 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.070+0000 7f4ea88e6700 5 monclient: authenticate success, global_id 4318 2023-09-30T01:33:06.091 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.070+0000 7f4ea88e6700 10 monclient: _renew_subs 2023-09-30T01:33:06.092 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.070+0000 7f4ea88e6700 10 monclient: _send_mon_message to mon.i at v2:172.21.15.107:3303/0 2023-09-30T01:33:06.092 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.070+0000 7f4ea88e6700 1 -- 172.21.15.88:0/3240982109 --> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e94009420 con 0x7f4ea409d950 2023-09-30T01:33:06.092 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.070+0000 7f4e9b7fe700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:33:06.092 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.070+0000 7f4e9b7fe700 1 -- 172.21.15.88:0/3240982109 <== mon.7 v2:172.21.15.107:3303/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f4e940138f0 con 0x7f4ea409d950 2023-09-30T01:33:06.092 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.070+0000 7f4e9b7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:33:06.092 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.070+0000 7f4e9b7fe700 10 monclient: got monmap 1 from mon.i (according to old e1) 2023-09-30T01:33:06.092 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.070+0000 7f4e9b7fe700 10 monclient: dump: 2023-09-30T01:33:06.092 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:33:06.092 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:33:06.092 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:06.092 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:06.092 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:33:06.092 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:33:06.093 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:33:06.093 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:33:06.093 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:33:06.093 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:33:06.093 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:33:06.093 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:33:06.093 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:33:06.093 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:33:06.093 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:33:06.093 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:33:06.093 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.070+0000 7f4ea17fa700 4 set_mon_vals no callback set 2023-09-30T01:33:06.093 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.070+0000 7f4ea88e6700 10 monclient: _renew_subs 2023-09-30T01:33:06.093 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.070+0000 7f4ea88e6700 10 monclient: _send_mon_message to mon.i at v2:172.21.15.107:3303/0 2023-09-30T01:33:06.094 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.070+0000 7f4ea88e6700 1 -- 172.21.15.88:0/3240982109 --> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e9c009420 con 0x7f4ea409d950 2023-09-30T01:33:06.094 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2023-09-30T01:33:06.094 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-09-30T01:33:06.096 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.070+0000 7f4ea88e6700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:33:06.096 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.070+0000 7f4ea88e6700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:33:06.096 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.070+0000 7f4ea88e6700 10 monclient: _send_mon_message to mon.i at v2:172.21.15.107:3303/0 2023-09-30T01:33:06.096 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.070+0000 7f4ea88e6700 1 -- 172.21.15.88:0/3240982109 --> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ea4071960 con 0x7f4ea409d950 2023-09-30T01:33:06.096 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.070+0000 7f4e9b7fe700 1 -- 172.21.15.88:0/3240982109 <== mon.7 v2:172.21.15.107:3303/0 4 ==== mgrmap(e 5) v1 ==== 40144+0+0 (secure 0 0 0) 0x7f4e94013470 con 0x7f4ea409d950 2023-09-30T01:33:06.096 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.070+0000 7f4e9b7fe700 1 --2- 172.21.15.88:0/3240982109 >> v2:172.21.15.107:6812/29926 conn(0x7f4e84033360 0x7f4e84035810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:33:06.096 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.070+0000 7f4e9b7fe700 1 -- 172.21.15.88:0/3240982109 <== mon.7 v2:172.21.15.107:3303/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 3502+0+0 (secure 0 0 0) 0x7f4e94043890 con 0x7f4ea409d950 2023-09-30T01:33:06.096 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.070+0000 7f4ea37fe700 1 --2- 172.21.15.88:0/3240982109 >> v2:172.21.15.107:6812/29926 conn(0x7f4e84033360 0x7f4e84035810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:33:06.096 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.074+0000 7f4ea37fe700 10 monclient: get_auth_request con 0x7f4e84033360 auth_method 0 2023-09-30T01:33:06.096 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.074+0000 7f4ea37fe700 1 --2- 172.21.15.88:0/3240982109 >> v2:172.21.15.107:6812/29926 conn(0x7f4e84033360 0x7f4e84035810 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f4e8c0072d0 tx=0x7f4e8c00b040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:33:06.096 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.082+0000 7f4e9b7fe700 1 -- 172.21.15.88:0/3240982109 <== mon.7 v2:172.21.15.107:3303/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f4e94013720 con 0x7f4ea409d950 2023-09-30T01:33:06.097 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.082+0000 7f4e9b7fe700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:33:06.097 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.082+0000 7f4e9b7fe700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:33:06.103 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2023-09-30T01:33:06.115 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.110+0000 7f4e3e77d700 10 monclient: get_monmap_and_config 2023-09-30T01:33:06.115 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.110+0000 7f4e3e77d700 10 monclient: build_initial_monmap 2023-09-30T01:33:06.115 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.110+0000 7f4e3e77d700 10 monclient: monmap: 2023-09-30T01:33:06.115 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2023-09-30T01:33:06.115 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:33:06.115 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:33:06.115444+0000 2023-09-30T01:33:06.115 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:33:06.115444+0000 2023-09-30T01:33:06.115 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2023-09-30T01:33:06.115 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:33:06.115 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a-legacy 2023-09-30T01:33:06.116 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.107:6789/0 mon.noname-f-legacy 2023-09-30T01:33:06.116 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.noname-b 2023-09-30T01:33:06.116 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.noname-g 2023-09-30T01:33:06.116 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.noname-c 2023-09-30T01:33:06.116 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.noname-h 2023-09-30T01:33:06.116 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.noname-d 2023-09-30T01:33:06.116 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.noname-i 2023-09-30T01:33:06.116 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.noname-e 2023-09-30T01:33:06.116 INFO:teuthology.orchestra.run.smithi088.stderr:9: v2:172.21.15.88:3300/0 mon.noname-a 2023-09-30T01:33:06.116 INFO:teuthology.orchestra.run.smithi088.stderr:10: v2:172.21.15.107:3300/0 mon.noname-f 2023-09-30T01:33:06.116 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:33:06.119 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.114+0000 7f4e3e77d700 1 Processor -- start 2023-09-30T01:33:06.119 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.114+0000 7f4e3e77d700 1 -- start start 2023-09-30T01:33:06.119 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.114+0000 7f4e3e77d700 10 monclient: init 2023-09-30T01:33:06.119 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.114+0000 7f4e3e77d700 10 monclient: _reopen_session rank -1 2023-09-30T01:33:06.119 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.114+0000 7f4e3e77d700 10 monclient: _add_conns ranks=[0,3,10,2,4,8,7,6,1,5,9] 2023-09-30T01:33:06.119 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.114+0000 7f4e3e77d700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f4e380976f0 addr v1:172.21.15.88:6789/0 2023-09-30T01:33:06.119 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.114+0000 7f4e3e77d700 1 --2- >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7f4e38098de0 0x7f4e3809f490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:33:06.119 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.114+0000 7f4e3e77d700 10 monclient(hunting): picked mon.noname-g con 0x7f4e38098de0 addr [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] 2023-09-30T01:33:06.119 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.114+0000 7f4e3e77d700 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f4e381357b0 0x7f4e38137b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:33:06.119 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.114+0000 7f4e3e77d700 10 monclient(hunting): picked mon.noname-f con 0x7f4e381357b0 addr v2:172.21.15.107:3300/0 2023-09-30T01:33:06.120 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.114+0000 7f4e3e77d700 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e3804bb20 con 0x7f4e380976f0 2023-09-30T01:33:06.120 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.114+0000 7f4e3e77d700 10 monclient(hunting): start opening mon connection 2023-09-30T01:33:06.120 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.114+0000 7f4e3e77d700 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4e38098420 con 0x7f4e381357b0 2023-09-30T01:33:06.120 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.114+0000 7f4e3e77d700 10 monclient(hunting): start opening mon connection 2023-09-30T01:33:06.120 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.114+0000 7f4e3e77d700 1 -- --> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e38098a30 con 0x7f4e38098de0 2023-09-30T01:33:06.120 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.114+0000 7f4e3e77d700 10 monclient(hunting): _renew_subs 2023-09-30T01:33:06.120 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.114+0000 7f4e3e77d700 10 monclient(hunting): authenticate will time out at 858.153357s 2023-09-30T01:33:06.120 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.114+0000 7f4e3d77b700 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f4e380976f0 0x7f4e3809ed80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:54120/0 (socket says 172.21.15.88:54120) 2023-09-30T01:33:06.120 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.114+0000 7f4e3d77b700 1 -- 172.21.15.88:0/2146071053 learned_addr learned my addr 172.21.15.88:0/2146071053 (peer_addr_for_me v1:172.21.15.88:0/0) 2023-09-30T01:33:06.120 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.114+0000 7f4e3cf7a700 1 --2- 172.21.15.88:0/2146071053 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7f4e38098de0 0x7f4e3809f490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:33:06.120 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.114+0000 7f4e3df7c700 1 --2- 172.21.15.88:0/2146071053 >> v2:172.21.15.107:3300/0 conn(0x7f4e381357b0 0x7f4e38137b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:33:06.121 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.114+0000 7f4e3df7c700 10 monclient(hunting): get_auth_request con 0x7f4e381357b0 auth_method 0 2023-09-30T01:33:06.121 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.114+0000 7f4e3df7c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:33:06.121 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.114+0000 7f4e3df7c700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:33:06.121 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.114+0000 7f4e3df7c700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:33:06.121 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.114+0000 7f4e2ffff700 1 -- 172.21.15.88:0/2146071053 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 210182572 0 0) 0x7f4e3804bb20 con 0x7f4e380976f0 2023-09-30T01:33:06.121 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.114+0000 7f4e3cf7a700 10 monclient(hunting): get_auth_request con 0x7f4e38098de0 auth_method 0 2023-09-30T01:33:06.121 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.114+0000 7f4e3cf7a700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:33:06.121 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.114+0000 7f4e3cf7a700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:33:06.121 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.114+0000 7f4e3cf7a700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:33:06.121 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.114+0000 7f4e2ffff700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:33:06.121 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.114+0000 7f4e2ffff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:33:06.121 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.114+0000 7f4e2ffff700 10 monclient(hunting): my global_id is 4374 2023-09-30T01:33:06.122 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.114+0000 7f4e2ffff700 1 -- 172.21.15.88:0/2146071053 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4e24003550 con 0x7f4e380976f0 2023-09-30T01:33:06.122 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.114+0000 7f4e3df7c700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:33:06.122 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.114+0000 7f4e3df7c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:33:06.122 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.114+0000 7f4e3df7c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:33:06.122 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.114+0000 7f4e3cf7a700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:33:06.122 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.114+0000 7f4e3cf7a700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:33:06.122 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.114+0000 7f4e3cf7a700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:33:06.122 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.114+0000 7f4e3df7c700 10 monclient(hunting): handle_auth_done global_id 4456 payload 274 2023-09-30T01:33:06.122 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.114+0000 7f4e3df7c700 1 -- 172.21.15.88:0/2146071053 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7f4e38098de0 msgr2=0x7f4e3809f490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:33:06.129 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.114+0000 7f4e3df7c700 1 --2- 172.21.15.88:0/2146071053 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7f4e38098de0 0x7f4e3809f490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:06.130 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.114+0000 7f4e3df7c700 1 -- 172.21.15.88:0/2146071053 >> v1:172.21.15.88:6789/0 conn(0x7f4e380976f0 legacy=0x7f4e3809ed80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:33:06.130 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.114+0000 7f4e3df7c700 10 monclient: _finish_hunting 0 2023-09-30T01:33:06.130 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.114+0000 7f4e3df7c700 1 monclient: found mon.noname-f 2023-09-30T01:33:06.130 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.114+0000 7f4e3df7c700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:33:06.130 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.114+0000 7f4e3df7c700 10 monclient: _send_mon_message to mon.noname-f at v2:172.21.15.107:3300/0 2023-09-30T01:33:06.130 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.114+0000 7f4e3df7c700 1 -- 172.21.15.88:0/2146071053 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e38097ac0 con 0x7f4e381357b0 2023-09-30T01:33:06.130 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.114+0000 7f4e3df7c700 1 --2- 172.21.15.88:0/2146071053 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4e381357b0 0x7f4e38137b90 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f4e34009910 tx=0x7f4e340091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b6d95ba614c04c9 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:33:06.130 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.114+0000 7f4e2ffff700 1 -- 172.21.15.88:0/2146071053 <== mon.1 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f4e30001df0 con 0x7f4e381357b0 2023-09-30T01:33:06.130 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.114+0000 7f4e2ffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:33:06.130 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.114+0000 7f4e2ffff700 10 monclient: got monmap 1 from mon.noname-f (according to old e1) 2023-09-30T01:33:06.130 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.114+0000 7f4e2ffff700 10 monclient: dump: 2023-09-30T01:33:06.131 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:33:06.131 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:33:06.131 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:06.131 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:06.131 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:33:06.131 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:33:06.131 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:33:06.131 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:33:06.131 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:33:06.131 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:33:06.131 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:33:06.132 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:33:06.132 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:33:06.132 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:33:06.132 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:33:06.132 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:33:06.132 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.114+0000 7f4e2ffff700 10 monclient: _finish_auth 0 2023-09-30T01:33:06.132 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.114+0000 7f4e2ffff700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:33:06.132 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.114+0000 7f4e2ffff700 1 -- 172.21.15.88:0/2146071053 <== mon.1 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e34010a10 con 0x7f4e381357b0 2023-09-30T01:33:06.132 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.114+0000 7f4e2ffff700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:33:06.132 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.114+0000 7f4e2ffff700 1 -- 172.21.15.88:0/2146071053 <== mon.1 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f4e34011420 con 0x7f4e381357b0 2023-09-30T01:33:06.132 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.114+0000 7f4e2ffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:33:06.132 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.114+0000 7f4e2ffff700 10 monclient: got monmap 1 from mon.f (according to old e1) 2023-09-30T01:33:06.133 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.114+0000 7f4e2ffff700 10 monclient: dump: 2023-09-30T01:33:06.133 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:33:06.133 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:33:06.133 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:06.133 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:06.133 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:33:06.133 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:33:06.133 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:33:06.133 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:33:06.133 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:33:06.133 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:33:06.133 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:33:06.133 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:33:06.134 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:33:06.134 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:33:06.134 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:33:06.134 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:33:06.134 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.114+0000 7f4e3e77d700 5 monclient: authenticate success, global_id 4456 2023-09-30T01:33:06.134 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.114+0000 7f4e3e77d700 10 monclient: get_monmap_and_config success 2023-09-30T01:33:06.134 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.114+0000 7f4e3e77d700 4 set_mon_vals no callback set 2023-09-30T01:33:06.134 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.118+0000 7f4e3e77d700 10 monclient: shutdown 2023-09-30T01:33:06.134 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.118+0000 7f4e3e77d700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:33:06.134 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.118+0000 7f4e3e77d700 1 -- 172.21.15.88:0/2146071053 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4e381357b0 msgr2=0x7f4e38137b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:33:06.134 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.118+0000 7f4e3e77d700 1 --2- 172.21.15.88:0/2146071053 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4e381357b0 0x7f4e38137b90 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f4e34009910 tx=0x7f4e340091a0 comp rx=0 tx=0).stop 2023-09-30T01:33:06.134 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.118+0000 7f4e3e77d700 1 -- 172.21.15.88:0/2146071053 shutdown_connections 2023-09-30T01:33:06.135 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.118+0000 7f4e3e77d700 1 --2- 172.21.15.88:0/2146071053 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7f4e38098de0 0x7f4e3809f490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:06.135 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.118+0000 7f4e3e77d700 1 --2- 172.21.15.88:0/2146071053 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4e381357b0 0x7f4e38137b90 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:06.135 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.118+0000 7f4e3e77d700 1 -- 172.21.15.88:0/2146071053 >> 172.21.15.88:0/2146071053 conn(0x7f4e380a0910 msgr2=0x7f4e380a33c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:33:06.135 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.118+0000 7f4e3e77d700 1 -- 172.21.15.88:0/2146071053 shutdown_connections 2023-09-30T01:33:06.135 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.118+0000 7f4e3e77d700 1 -- 172.21.15.88:0/2146071053 wait complete. 2023-09-30T01:33:06.135 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.118+0000 7f4e3e77d700 10 monclient: build_initial_monmap 2023-09-30T01:33:06.135 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.118+0000 7f4e3e77d700 10 monclient: monmap: 2023-09-30T01:33:06.135 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2023-09-30T01:33:06.135 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:33:06.135 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 0.000000 2023-09-30T01:33:06.135 INFO:teuthology.orchestra.run.smithi088.stderr:created 0.000000 2023-09-30T01:33:06.135 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2023-09-30T01:33:06.136 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 1 2023-09-30T01:33:06.136 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.noname-a 2023-09-30T01:33:06.136 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.noname-f 2023-09-30T01:33:06.136 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.noname-b 2023-09-30T01:33:06.136 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.noname-g 2023-09-30T01:33:06.136 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.noname-c 2023-09-30T01:33:06.136 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.noname-h 2023-09-30T01:33:06.136 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.noname-d 2023-09-30T01:33:06.136 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.noname-i 2023-09-30T01:33:06.136 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.noname-e 2023-09-30T01:33:06.136 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:33:06.136 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.118+0000 7f4e3e77d700 1 Processor -- start 2023-09-30T01:33:06.136 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.118+0000 7f4e3e77d700 1 -- start start 2023-09-30T01:33:06.137 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.118+0000 7f4e3e77d700 10 monclient: init 2023-09-30T01:33:06.137 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.118+0000 7f4e3e77d700 10 monclient: _reopen_session rank -1 2023-09-30T01:33:06.137 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.118+0000 7f4e3e77d700 10 monclient: _add_conns ranks=[7,3,8,1,6,0,5,4,2] 2023-09-30T01:33:06.137 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.118+0000 7f4e3e77d700 1 --2- >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7f4e380976f0 0x7f4e3813dde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:33:06.137 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.118+0000 7f4e3e77d700 10 monclient(hunting): picked mon.noname-i con 0x7f4e380976f0 addr [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] 2023-09-30T01:33:06.137 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.118+0000 7f4e3e77d700 1 --2- >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7f4e38098de0 0x7f4e38093380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:33:06.137 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.118+0000 7f4e3e77d700 10 monclient(hunting): picked mon.noname-g con 0x7f4e38098de0 addr [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] 2023-09-30T01:33:06.137 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.118+0000 7f4e3e77d700 1 --2- >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f4e381357b0 0x7f4e3806d860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:33:06.137 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.118+0000 7f4e3e77d700 10 monclient(hunting): picked mon.noname-e con 0x7f4e381357b0 addr [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] 2023-09-30T01:33:06.137 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.118+0000 7f4e3e77d700 10 monclient(hunting): start opening mon connection 2023-09-30T01:33:06.137 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.118+0000 7f4e3e77d700 1 -- --> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e380970b0 con 0x7f4e38098de0 2023-09-30T01:33:06.137 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.118+0000 7f4e3e77d700 10 monclient(hunting): start opening mon connection 2023-09-30T01:33:06.138 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.118+0000 7f4e3e77d700 1 -- --> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] -- mon_getmap magic: 0 v1 -- 0x7f4e3808a820 con 0x7f4e380976f0 2023-09-30T01:33:06.138 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.118+0000 7f4e3e77d700 10 monclient(hunting): start opening mon connection 2023-09-30T01:33:06.138 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.118+0000 7f4e3e77d700 1 -- --> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] -- mon_getmap magic: 0 v1 -- 0x7f4e3808a540 con 0x7f4e381357b0 2023-09-30T01:33:06.138 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.118+0000 7f4e3e77d700 10 monclient(hunting): _renew_subs 2023-09-30T01:33:06.138 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.118+0000 7f4e3e77d700 10 monclient(hunting): authenticate will time out at 858.158571s 2023-09-30T01:33:06.138 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.118+0000 7f4e3df7c700 1 --2- >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f4e381357b0 0x7f4e3806d860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:33:06.138 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.118+0000 7f4e3df7c700 1 --2- >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f4e381357b0 0x7f4e3806d860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3304/0 says I am v2:172.21.15.88:36762/0 (socket says 172.21.15.88:36762) 2023-09-30T01:33:06.138 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.118+0000 7f4e3df7c700 1 -- 172.21.15.88:0/405194235 learned_addr learned my addr 172.21.15.88:0/405194235 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-09-30T01:33:06.138 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.118+0000 7f4e3df7c700 10 monclient(hunting): get_auth_request con 0x7f4e381357b0 auth_method 0 2023-09-30T01:33:06.138 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.118+0000 7f4e3df7c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:33:06.138 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.118+0000 7f4e3df7c700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:33:06.138 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.118+0000 7f4e3df7c700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:33:06.139 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.118+0000 7f4e3cf7a700 1 --2- 172.21.15.88:0/405194235 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7f4e38098de0 0x7f4e38093380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:33:06.139 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.118+0000 7f4e3df7c700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:33:06.139 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.118+0000 7f4e3df7c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:33:06.139 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.118+0000 7f4e3df7c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:33:06.139 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.118+0000 7f4e3d77b700 1 --2- 172.21.15.88:0/405194235 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7f4e380976f0 0x7f4e3813dde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:33:06.139 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.118+0000 7f4e3cf7a700 10 monclient(hunting): get_auth_request con 0x7f4e38098de0 auth_method 0 2023-09-30T01:33:06.139 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.118+0000 7f4e3cf7a700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:33:06.139 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.118+0000 7f4e3cf7a700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:33:06.139 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.118+0000 7f4e3cf7a700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:33:06.139 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.118+0000 7f4e3d77b700 10 monclient(hunting): get_auth_request con 0x7f4e380976f0 auth_method 0 2023-09-30T01:33:06.139 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.118+0000 7f4e3d77b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:33:06.139 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.118+0000 7f4e3d77b700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:33:06.139 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.118+0000 7f4e3d77b700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:33:06.140 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.118+0000 7f4e3cf7a700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:33:06.140 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.118+0000 7f4e3cf7a700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:33:06.140 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.118+0000 7f4e3cf7a700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:33:06.140 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.118+0000 7f4e3d77b700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:33:06.140 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.118+0000 7f4e3d77b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:33:06.140 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.118+0000 7f4e3d77b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:33:06.140 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.118+0000 7f4e3df7c700 10 monclient(hunting): handle_auth_done global_id 4355 payload 824 2023-09-30T01:33:06.140 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.118+0000 7f4e3df7c700 1 -- 172.21.15.88:0/405194235 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7f4e380976f0 msgr2=0x7f4e3813dde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:33:06.140 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.118+0000 7f4e3df7c700 1 --2- 172.21.15.88:0/405194235 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7f4e380976f0 0x7f4e3813dde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:06.140 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.118+0000 7f4e3df7c700 1 -- 172.21.15.88:0/405194235 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7f4e38098de0 msgr2=0x7f4e38093380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:33:06.140 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.118+0000 7f4e3df7c700 1 --2- 172.21.15.88:0/405194235 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7f4e38098de0 0x7f4e38093380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:06.140 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.118+0000 7f4e3df7c700 10 monclient: _finish_hunting 0 2023-09-30T01:33:06.140 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.118+0000 7f4e3df7c700 1 monclient: found mon.noname-e 2023-09-30T01:33:06.141 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.118+0000 7f4e3df7c700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:33:06.141 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.118+0000 7f4e3df7c700 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.88:3304/0 2023-09-30T01:33:06.141 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.118+0000 7f4e3df7c700 1 -- 172.21.15.88:0/405194235 --> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e34009420 con 0x7f4e381357b0 2023-09-30T01:33:06.141 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.118+0000 7f4e3df7c700 1 --2- 172.21.15.88:0/405194235 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f4e381357b0 0x7f4e3806d860 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f4e340091e0 tx=0x7f4e34017000 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:33:06.141 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.118+0000 7f4e2d7fa700 1 -- 172.21.15.88:0/405194235 <== mon.8 v2:172.21.15.88:3304/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f4e3401c420 con 0x7f4e381357b0 2023-09-30T01:33:06.141 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.118+0000 7f4e2d7fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:33:06.141 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.118+0000 7f4e2d7fa700 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2023-09-30T01:33:06.141 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.118+0000 7f4e2d7fa700 10 monclient: dump: 2023-09-30T01:33:06.141 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:33:06.141 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:33:06.141 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:06.141 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:06.142 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:33:06.142 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:33:06.142 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:33:06.142 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:33:06.142 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:33:06.142 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:33:06.142 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:33:06.142 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:33:06.142 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:33:06.142 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:33:06.142 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:33:06.142 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:33:06.143 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.118+0000 7f4e2d7fa700 10 monclient: _finish_auth 0 2023-09-30T01:33:06.143 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.118+0000 7f4e2d7fa700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:33:06.143 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.118+0000 7f4e2d7fa700 1 -- 172.21.15.88:0/405194235 <== mon.8 v2:172.21.15.88:3304/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e3401c5e0 con 0x7f4e381357b0 2023-09-30T01:33:06.143 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.118+0000 7f4e3e77d700 5 monclient: authenticate success, global_id 4355 2023-09-30T01:33:06.143 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.118+0000 7f4e3e77d700 10 monclient: _renew_subs 2023-09-30T01:33:06.143 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.118+0000 7f4e3e77d700 10 monclient: _send_mon_message to mon.e at v2:172.21.15.88:3304/0 2023-09-30T01:33:06.143 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.118+0000 7f4e3e77d700 1 -- 172.21.15.88:0/405194235 --> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e3813ec10 con 0x7f4e381357b0 2023-09-30T01:33:06.143 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.122+0000 7f4e2d7fa700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:33:06.143 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.122+0000 7f4e2d7fa700 1 -- 172.21.15.88:0/405194235 <== mon.8 v2:172.21.15.88:3304/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f4e3401cbd0 con 0x7f4e381357b0 2023-09-30T01:33:06.143 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.122+0000 7f4e2d7fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:33:06.143 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.122+0000 7f4e2d7fa700 10 monclient: got monmap 1 from mon.e (according to old e1) 2023-09-30T01:33:06.143 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.122+0000 7f4e2d7fa700 10 monclient: dump: 2023-09-30T01:33:06.144 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:33:06.144 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:33:06.144 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:06.144 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:06.144 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:33:06.144 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:33:06.144 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:33:06.144 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:33:06.144 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:33:06.144 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:33:06.144 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:33:06.144 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:33:06.144 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:33:06.145 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:33:06.145 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:33:06.145 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:33:06.145 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.122+0000 7f4e3e77d700 10 monclient: _renew_subs 2023-09-30T01:33:06.145 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.122+0000 7f4e3e77d700 10 monclient: _send_mon_message to mon.e at v2:172.21.15.88:3304/0 2023-09-30T01:33:06.145 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.122+0000 7f4e3e77d700 1 -- 172.21.15.88:0/405194235 --> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e3813e320 con 0x7f4e381357b0 2023-09-30T01:33:06.145 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.122+0000 7f4e3e77d700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:33:06.145 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.122+0000 7f4e3e77d700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:33:06.145 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.122+0000 7f4e3e77d700 10 monclient: _send_mon_message to mon.e at v2:172.21.15.88:3304/0 2023-09-30T01:33:06.145 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.122+0000 7f4e3e77d700 1 -- 172.21.15.88:0/405194235 --> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4e380a13c0 con 0x7f4e381357b0 2023-09-30T01:33:06.145 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.122+0000 7f4e2d7fa700 1 -- 172.21.15.88:0/405194235 <== mon.8 v2:172.21.15.88:3304/0 4 ==== mgrmap(e 5) v1 ==== 40144+0+0 (secure 0 0 0) 0x7f4e3401cd30 con 0x7f4e381357b0 2023-09-30T01:33:06.145 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.122+0000 7f4e2d7fa700 1 --2- 172.21.15.88:0/405194235 >> v2:172.21.15.107:6812/29926 conn(0x7f4e24033c20 0x7f4e240360d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:33:06.146 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.122+0000 7f4e2d7fa700 1 -- 172.21.15.88:0/405194235 <== mon.8 v2:172.21.15.88:3304/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 3502+0+0 (secure 0 0 0) 0x7f4e34044a20 con 0x7f4e381357b0 2023-09-30T01:33:06.146 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.122+0000 7f4e3d77b700 1 --2- 172.21.15.88:0/405194235 >> v2:172.21.15.107:6812/29926 conn(0x7f4e24033c20 0x7f4e240360d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:33:06.146 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.122+0000 7f4e3d77b700 10 monclient: get_auth_request con 0x7f4e24033c20 auth_method 0 2023-09-30T01:33:06.146 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.122+0000 7f4e3d77b700 1 --2- 172.21.15.88:0/405194235 >> v2:172.21.15.107:6812/29926 conn(0x7f4e24033c20 0x7f4e240360d0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f4e30000f80 tx=0x7f4e30009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:33:06.146 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.126+0000 7f4e2effd700 4 set_mon_vals no callback set 2023-09-30T01:33:06.146 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.126+0000 7f4e2d7fa700 1 -- 172.21.15.88:0/405194235 <== mon.8 v2:172.21.15.88:3304/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f4e380a13c0 con 0x7f4e381357b0 2023-09-30T01:33:06.146 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.126+0000 7f4e2d7fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:33:06.147 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.126+0000 7f4e2d7fa700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:33:06.174 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.170+0000 7fc8c5fa1700 10 monclient: get_monmap_and_config 2023-09-30T01:33:06.174 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.170+0000 7fc8c5fa1700 10 monclient: build_initial_monmap 2023-09-30T01:33:06.174 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.170+0000 7fc8c5fa1700 10 monclient: monmap: 2023-09-30T01:33:06.174 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2023-09-30T01:33:06.174 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:33:06.174 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:33:06.174609+0000 2023-09-30T01:33:06.174 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:33:06.174609+0000 2023-09-30T01:33:06.175 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2023-09-30T01:33:06.175 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:33:06.175 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a-legacy 2023-09-30T01:33:06.175 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.107:6789/0 mon.noname-f-legacy 2023-09-30T01:33:06.175 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.noname-b 2023-09-30T01:33:06.175 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.noname-g 2023-09-30T01:33:06.175 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.noname-c 2023-09-30T01:33:06.175 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.noname-h 2023-09-30T01:33:06.175 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.noname-d 2023-09-30T01:33:06.175 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.noname-i 2023-09-30T01:33:06.175 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.noname-e 2023-09-30T01:33:06.175 INFO:teuthology.orchestra.run.smithi088.stderr:9: v2:172.21.15.88:3300/0 mon.noname-a 2023-09-30T01:33:06.176 INFO:teuthology.orchestra.run.smithi088.stderr:10: v2:172.21.15.107:3300/0 mon.noname-f 2023-09-30T01:33:06.176 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:33:06.179 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.174+0000 7fc8c5fa1700 1 Processor -- start 2023-09-30T01:33:06.179 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.174+0000 7fc8c5fa1700 1 -- start start 2023-09-30T01:33:06.179 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.174+0000 7fc8c5fa1700 10 monclient: init 2023-09-30T01:33:06.179 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.174+0000 7fc8c5fa1700 10 monclient: _reopen_session rank -1 2023-09-30T01:33:06.179 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.174+0000 7fc8c5fa1700 10 monclient: _add_conns ranks=[0,5,3,2,9,7,10,8,1,6,4] 2023-09-30T01:33:06.180 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.174+0000 7fc8c5fa1700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fc8c009c430 addr v1:172.21.15.88:6789/0 2023-09-30T01:33:06.180 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.174+0000 7fc8c5fa1700 1 --2- >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7fc8c00990f0 0x7fc8c01346c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:33:06.180 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.174+0000 7fc8c5fa1700 10 monclient(hunting): picked mon.noname-h con 0x7fc8c00990f0 addr [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] 2023-09-30T01:33:06.180 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.174+0000 7fc8c5fa1700 1 --2- >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7fc8c009cf30 0x7fc8c0137d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:33:06.180 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.174+0000 7fc8c5fa1700 10 monclient(hunting): picked mon.noname-g con 0x7fc8c009cf30 addr [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] 2023-09-30T01:33:06.180 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.174+0000 7fc8c5fa1700 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8c004bb20 con 0x7fc8c009c430 2023-09-30T01:33:06.180 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.174+0000 7fc8c5fa1700 10 monclient(hunting): start opening mon connection 2023-09-30T01:33:06.180 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.174+0000 7fc8c5fa1700 1 -- --> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8c006f420 con 0x7fc8c009cf30 2023-09-30T01:33:06.180 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.174+0000 7fc8c5fa1700 10 monclient(hunting): start opening mon connection 2023-09-30T01:33:06.180 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.174+0000 7fc8c5fa1700 1 -- --> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] -- mon_getmap magic: 0 v1 -- 0x7fc8c00994e0 con 0x7fc8c00990f0 2023-09-30T01:33:06.180 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.174+0000 7fc8c5fa1700 10 monclient(hunting): _renew_subs 2023-09-30T01:33:06.180 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.174+0000 7fc8c5fa1700 10 monclient(hunting): authenticate will time out at 858.213216s 2023-09-30T01:33:06.181 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.174+0000 7fc8c4f9f700 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7fc8c009c430 0x7fc8c00989e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:54124/0 (socket says 172.21.15.88:54124) 2023-09-30T01:33:06.181 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.174+0000 7fc8c4f9f700 1 -- 172.21.15.88:0/3939846617 learned_addr learned my addr 172.21.15.88:0/3939846617 (peer_addr_for_me v1:172.21.15.88:0/0) 2023-09-30T01:33:06.181 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.174+0000 7fc8c57a0700 1 --2- 172.21.15.88:0/3939846617 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7fc8c009cf30 0x7fc8c0137d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:33:06.181 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.174+0000 7fc8c57a0700 10 monclient(hunting): get_auth_request con 0x7fc8c009cf30 auth_method 0 2023-09-30T01:33:06.181 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.174+0000 7fc8c57a0700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:33:06.181 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.174+0000 7fc8c57a0700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:33:06.181 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.174+0000 7fc8c57a0700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:33:06.181 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.174+0000 7fc8bffff700 1 --2- 172.21.15.88:0/3939846617 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7fc8c00990f0 0x7fc8c01346c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:33:06.181 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.174+0000 7fc8bffff700 10 monclient(hunting): get_auth_request con 0x7fc8c00990f0 auth_method 0 2023-09-30T01:33:06.181 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.174+0000 7fc8bffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:33:06.181 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.174+0000 7fc8bffff700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:33:06.182 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.174+0000 7fc8bffff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:33:06.182 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.174+0000 7fc8c57a0700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:33:06.182 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.174+0000 7fc8c57a0700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:33:06.182 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.174+0000 7fc8c57a0700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:33:06.182 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.174+0000 7fc8bf7fe700 1 -- 172.21.15.88:0/3939846617 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 651104825 0 0) 0x7fc8c004bb20 con 0x7fc8c009c430 2023-09-30T01:33:06.182 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.174+0000 7fc8bf7fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:33:06.182 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.174+0000 7fc8bf7fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:33:06.182 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.174+0000 7fc8bf7fe700 10 monclient(hunting): my global_id is 4383 2023-09-30T01:33:06.182 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.174+0000 7fc8bf7fe700 1 -- 172.21.15.88:0/3939846617 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc8a8003570 con 0x7fc8c009c430 2023-09-30T01:33:06.182 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.174+0000 7fc8bffff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:33:06.182 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.174+0000 7fc8bffff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:33:06.182 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.174+0000 7fc8bffff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:33:06.183 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.174+0000 7fc8c57a0700 10 monclient(hunting): handle_auth_done global_id 4287 payload 274 2023-09-30T01:33:06.183 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.174+0000 7fc8c57a0700 1 -- 172.21.15.88:0/3939846617 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7fc8c00990f0 msgr2=0x7fc8c01346c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:33:06.183 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.174+0000 7fc8c57a0700 1 --2- 172.21.15.88:0/3939846617 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7fc8c00990f0 0x7fc8c01346c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:06.183 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.174+0000 7fc8bf7fe700 1 -- 172.21.15.88:0/3939846617 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2281318797 0 0) 0x7fc8a8003570 con 0x7fc8c009c430 2023-09-30T01:33:06.183 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.174+0000 7fc8c57a0700 1 -- 172.21.15.88:0/3939846617 >> v1:172.21.15.88:6789/0 conn(0x7fc8c009c430 legacy=0x7fc8c00989e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:33:06.183 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.174+0000 7fc8c57a0700 10 monclient: _finish_hunting 0 2023-09-30T01:33:06.183 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.174+0000 7fc8c57a0700 1 monclient: found mon.noname-g 2023-09-30T01:33:06.183 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.174+0000 7fc8c57a0700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:33:06.183 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.174+0000 7fc8c57a0700 10 monclient: _send_mon_message to mon.noname-g at v2:172.21.15.107:3301/0 2023-09-30T01:33:06.183 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.174+0000 7fc8c57a0700 1 -- 172.21.15.88:0/3939846617 --> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8c009bf80 con 0x7fc8c009cf30 2023-09-30T01:33:06.183 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.174+0000 7fc8bf7fe700 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2023-09-30T01:33:06.183 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.174+0000 7fc8c57a0700 1 --2- 172.21.15.88:0/3939846617 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7fc8c009cf30 0x7fc8c0137d20 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fc8b8009ec0 tx=0x7fc8b8009780 comp rx=0 tx=0).ready entity=mon.3 client_cookie=96782c9dfd94258f server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:33:06.183 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.174+0000 7fc8bf7fe700 1 -- 172.21.15.88:0/3939846617 <== mon.3 v2:172.21.15.107:3301/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7fc8b800e070 con 0x7fc8c009cf30 2023-09-30T01:33:06.184 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.174+0000 7fc8bf7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:33:06.184 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.174+0000 7fc8bf7fe700 10 monclient: got monmap 1 from mon.noname-g (according to old e1) 2023-09-30T01:33:06.184 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.174+0000 7fc8bf7fe700 10 monclient: dump: 2023-09-30T01:33:06.184 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:33:06.184 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:33:06.184 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:06.184 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:06.184 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:33:06.184 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:33:06.184 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:33:06.184 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:33:06.184 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:33:06.185 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:33:06.185 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:33:06.185 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:33:06.185 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:33:06.185 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:33:06.185 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:33:06.185 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:33:06.185 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.174+0000 7fc8bf7fe700 10 monclient: _finish_auth 0 2023-09-30T01:33:06.185 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.174+0000 7fc8bf7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:33:06.185 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.174+0000 7fc8c5fa1700 5 monclient: authenticate success, global_id 4287 2023-09-30T01:33:06.185 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.174+0000 7fc8c5fa1700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:33:06.185 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.174+0000 7fc8bf7fe700 1 -- 172.21.15.88:0/3939846617 <== mon.3 v2:172.21.15.107:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8b800ba80 con 0x7fc8c009cf30 2023-09-30T01:33:06.185 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.174+0000 7fc8bf7fe700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:33:06.186 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.174+0000 7fc8bf7fe700 1 -- 172.21.15.88:0/3939846617 <== mon.3 v2:172.21.15.107:3301/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7fc8b00018f0 con 0x7fc8c009cf30 2023-09-30T01:33:06.186 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.174+0000 7fc8bf7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:33:06.186 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.174+0000 7fc8bf7fe700 10 monclient: got monmap 1 from mon.g (according to old e1) 2023-09-30T01:33:06.186 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.174+0000 7fc8bf7fe700 10 monclient: dump: 2023-09-30T01:33:06.186 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:33:06.186 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:33:06.186 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:06.186 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:06.186 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:33:06.186 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:33:06.186 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:33:06.187 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:33:06.187 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:33:06.187 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:33:06.187 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:33:06.187 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:33:06.187 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:33:06.187 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:33:06.187 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:33:06.187 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:33:06.187 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.174+0000 7fc8c5fa1700 10 monclient: get_monmap_and_config success 2023-09-30T01:33:06.187 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.174+0000 7fc8c5fa1700 4 set_mon_vals no callback set 2023-09-30T01:33:06.187 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.178+0000 7fc8c5fa1700 10 monclient: shutdown 2023-09-30T01:33:06.187 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.178+0000 7fc8c5fa1700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:33:06.188 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.178+0000 7fc8c5fa1700 1 -- 172.21.15.88:0/3939846617 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7fc8c009cf30 msgr2=0x7fc8c0137d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:33:06.188 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.178+0000 7fc8c5fa1700 1 --2- 172.21.15.88:0/3939846617 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7fc8c009cf30 0x7fc8c0137d20 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fc8b8009ec0 tx=0x7fc8b8009780 comp rx=0 tx=0).stop 2023-09-30T01:33:06.188 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.178+0000 7fc8c5fa1700 1 -- 172.21.15.88:0/3939846617 shutdown_connections 2023-09-30T01:33:06.188 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.178+0000 7fc8c5fa1700 1 --2- 172.21.15.88:0/3939846617 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7fc8c009cf30 0x7fc8c0137d20 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:06.188 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.178+0000 7fc8c5fa1700 1 --2- 172.21.15.88:0/3939846617 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7fc8c00990f0 0x7fc8c01346c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:06.188 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.178+0000 7fc8c5fa1700 1 -- 172.21.15.88:0/3939846617 >> 172.21.15.88:0/3939846617 conn(0x7fc8c00a2540 msgr2=0x7fc8c00a2910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:33:06.188 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.178+0000 7fc8c5fa1700 1 -- 172.21.15.88:0/3939846617 shutdown_connections 2023-09-30T01:33:06.188 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.178+0000 7fc8c5fa1700 1 -- 172.21.15.88:0/3939846617 wait complete. 2023-09-30T01:33:06.191 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.178+0000 7fc8c5fa1700 10 monclient: build_initial_monmap 2023-09-30T01:33:06.192 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.178+0000 7fc8c5fa1700 10 monclient: monmap: 2023-09-30T01:33:06.192 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2023-09-30T01:33:06.192 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:33:06.192 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 0.000000 2023-09-30T01:33:06.192 INFO:teuthology.orchestra.run.smithi088.stderr:created 0.000000 2023-09-30T01:33:06.192 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2023-09-30T01:33:06.192 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 1 2023-09-30T01:33:06.192 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.noname-a 2023-09-30T01:33:06.192 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.noname-f 2023-09-30T01:33:06.192 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.noname-b 2023-09-30T01:33:06.192 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.noname-g 2023-09-30T01:33:06.193 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.noname-c 2023-09-30T01:33:06.193 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.noname-h 2023-09-30T01:33:06.193 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.noname-d 2023-09-30T01:33:06.193 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.noname-i 2023-09-30T01:33:06.193 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.noname-e 2023-09-30T01:33:06.193 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:33:06.193 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.178+0000 7fc8c5fa1700 1 Processor -- start 2023-09-30T01:33:06.193 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.178+0000 7fc8c5fa1700 1 -- start start 2023-09-30T01:33:06.193 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.178+0000 7fc8c5fa1700 10 monclient: init 2023-09-30T01:33:06.193 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.178+0000 7fc8c5fa1700 10 monclient: _reopen_session rank -1 2023-09-30T01:33:06.193 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.178+0000 7fc8c5fa1700 10 monclient: _add_conns ranks=[8,3,0,4,7,2,6,5,1] 2023-09-30T01:33:06.193 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.178+0000 7fc8c5fa1700 1 --2- >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7fc8c00990f0 0x7fc8c006cbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:33:06.194 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.178+0000 7fc8c5fa1700 10 monclient(hunting): picked mon.noname-e con 0x7fc8c00990f0 addr [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] 2023-09-30T01:33:06.194 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.178+0000 7fc8c5fa1700 1 --2- >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7fc8c009c430 0x7fc8c006d110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:33:06.194 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.178+0000 7fc8c5fa1700 10 monclient(hunting): picked mon.noname-g con 0x7fc8c009c430 addr [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] 2023-09-30T01:33:06.194 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.178+0000 7fc8c5fa1700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc8c009cf30 0x7fc8c006d650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:33:06.194 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.178+0000 7fc8c5fa1700 10 monclient(hunting): picked mon.noname-a con 0x7fc8c009cf30 addr [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] 2023-09-30T01:33:06.194 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.178+0000 7fc8c5fa1700 10 monclient(hunting): start opening mon connection 2023-09-30T01:33:06.194 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.178+0000 7fc8c5fa1700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8c0139fe0 con 0x7fc8c009cf30 2023-09-30T01:33:06.194 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.178+0000 7fc8c5fa1700 10 monclient(hunting): start opening mon connection 2023-09-30T01:33:06.194 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.178+0000 7fc8c5fa1700 1 -- --> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8c009ab50 con 0x7fc8c009c430 2023-09-30T01:33:06.194 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.178+0000 7fc8c5fa1700 10 monclient(hunting): start opening mon connection 2023-09-30T01:33:06.194 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.178+0000 7fc8c5fa1700 1 -- --> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] -- mon_getmap magic: 0 v1 -- 0x7fc8c006f590 con 0x7fc8c00990f0 2023-09-30T01:33:06.194 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.178+0000 7fc8c5fa1700 10 monclient(hunting): _renew_subs 2023-09-30T01:33:06.195 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.178+0000 7fc8c5fa1700 10 monclient(hunting): authenticate will time out at 858.218875s 2023-09-30T01:33:06.195 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.178+0000 7fc8c57a0700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc8c009cf30 0x7fc8c006d650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:33:06.195 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.178+0000 7fc8c4f9f700 1 --2- >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7fc8c00990f0 0x7fc8c006cbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:33:06.195 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.178+0000 7fc8c4f9f700 1 --2- >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7fc8c00990f0 0x7fc8c006cbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3304/0 says I am v2:172.21.15.88:36778/0 (socket says 172.21.15.88:36778) 2023-09-30T01:33:06.195 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.178+0000 7fc8c57a0700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc8c009cf30 0x7fc8c006d650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:57890/0 (socket says 172.21.15.88:57890) 2023-09-30T01:33:06.195 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.178+0000 7fc8c4f9f700 1 -- 172.21.15.88:0/2513588189 learned_addr learned my addr 172.21.15.88:0/2513588189 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-09-30T01:33:06.195 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.178+0000 7fc8c4f9f700 10 monclient(hunting): get_auth_request con 0x7fc8c00990f0 auth_method 0 2023-09-30T01:33:06.195 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.178+0000 7fc8c4f9f700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:33:06.195 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.178+0000 7fc8c4f9f700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:33:06.195 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.178+0000 7fc8c4f9f700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:33:06.195 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.178+0000 7fc8c57a0700 10 monclient(hunting): get_auth_request con 0x7fc8c009cf30 auth_method 0 2023-09-30T01:33:06.195 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.178+0000 7fc8c57a0700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:33:06.196 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.178+0000 7fc8c57a0700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:33:06.196 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.178+0000 7fc8c57a0700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:33:06.196 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.178+0000 7fc8c4f9f700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:33:06.196 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.178+0000 7fc8c4f9f700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:33:06.196 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.178+0000 7fc8c4f9f700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:33:06.196 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.178+0000 7fc8c57a0700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:33:06.196 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.178+0000 7fc8c57a0700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:33:06.196 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.178+0000 7fc8c57a0700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:33:06.196 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.178+0000 7fc8bffff700 1 --2- 172.21.15.88:0/2513588189 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7fc8c009c430 0x7fc8c006d110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:33:06.196 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.178+0000 7fc8bffff700 10 monclient(hunting): get_auth_request con 0x7fc8c009c430 auth_method 0 2023-09-30T01:33:06.196 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.178+0000 7fc8bffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:33:06.196 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.178+0000 7fc8bffff700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:33:06.197 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.178+0000 7fc8bffff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:33:06.197 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.178+0000 7fc8c4f9f700 10 monclient(hunting): handle_auth_done global_id 4364 payload 824 2023-09-30T01:33:06.197 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.178+0000 7fc8c4f9f700 1 -- 172.21.15.88:0/2513588189 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7fc8c009c430 msgr2=0x7fc8c006d110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:33:06.197 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.178+0000 7fc8c4f9f700 1 --2- 172.21.15.88:0/2513588189 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7fc8c009c430 0x7fc8c006d110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:06.197 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.178+0000 7fc8c4f9f700 1 -- 172.21.15.88:0/2513588189 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc8c009cf30 msgr2=0x7fc8c006d650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:33:06.197 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.178+0000 7fc8c4f9f700 1 --2- 172.21.15.88:0/2513588189 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc8c009cf30 0x7fc8c006d650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:06.197 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.178+0000 7fc8c4f9f700 10 monclient: _finish_hunting 0 2023-09-30T01:33:06.197 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.178+0000 7fc8c4f9f700 1 monclient: found mon.noname-e 2023-09-30T01:33:06.197 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.178+0000 7fc8c4f9f700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:33:06.197 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.178+0000 7fc8c4f9f700 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.88:3304/0 2023-09-30T01:33:06.197 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.178+0000 7fc8c4f9f700 1 -- 172.21.15.88:0/2513588189 --> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8b8009a20 con 0x7fc8c00990f0 2023-09-30T01:33:06.197 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.178+0000 7fc8c57a0700 1 --2- 172.21.15.88:0/2513588189 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc8c009cf30 0x7fc8c006d650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:33:06.197 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.182+0000 7fc8c4f9f700 1 --2- 172.21.15.88:0/2513588189 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7fc8c00990f0 0x7fc8c006cbd0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fc8b000c840 tx=0x7fc8b0001430 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:33:06.198 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.186+0000 7fc8bcff9700 1 -- 172.21.15.88:0/2513588189 <== mon.8 v2:172.21.15.88:3304/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7fc8b0004830 con 0x7fc8c00990f0 2023-09-30T01:33:06.198 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.186+0000 7fc8bcff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:33:06.198 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.186+0000 7fc8bcff9700 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2023-09-30T01:33:06.198 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.186+0000 7fc8bcff9700 10 monclient: dump: 2023-09-30T01:33:06.198 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:33:06.198 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:33:06.198 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:06.198 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:06.198 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:33:06.198 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:33:06.198 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:33:06.199 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:33:06.199 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:33:06.199 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:33:06.199 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:33:06.199 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:33:06.199 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:33:06.199 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:33:06.199 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:33:06.199 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:33:06.199 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.186+0000 7fc8bcff9700 10 monclient: _finish_auth 0 2023-09-30T01:33:06.199 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.186+0000 7fc8bcff9700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:33:06.199 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.186+0000 7fc8bcff9700 1 -- 172.21.15.88:0/2513588189 <== mon.8 v2:172.21.15.88:3304/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8b00049f0 con 0x7fc8c00990f0 2023-09-30T01:33:06.199 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.186+0000 7fc8bcff9700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:33:06.200 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.186+0000 7fc8bcff9700 1 -- 172.21.15.88:0/2513588189 <== mon.8 v2:172.21.15.88:3304/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7fc8b0014420 con 0x7fc8c00990f0 2023-09-30T01:33:06.200 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.186+0000 7fc8bcff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:33:06.200 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.186+0000 7fc8bcff9700 10 monclient: got monmap 1 from mon.e (according to old e1) 2023-09-30T01:33:06.200 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.186+0000 7fc8bcff9700 10 monclient: dump: 2023-09-30T01:33:06.200 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:33:06.200 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:33:06.200 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:06.200 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:06.200 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:33:06.200 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:33:06.200 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:33:06.200 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:33:06.201 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:33:06.201 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:33:06.201 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:33:06.201 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:33:06.201 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:33:06.201 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:33:06.201 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:33:06.201 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:33:06.201 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.186+0000 7fc8be7fc700 4 set_mon_vals no callback set 2023-09-30T01:33:06.201 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.186+0000 7fc8c5fa1700 5 monclient: authenticate success, global_id 4364 2023-09-30T01:33:06.201 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.186+0000 7fc8c5fa1700 10 monclient: _renew_subs 2023-09-30T01:33:06.201 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.186+0000 7fc8c5fa1700 10 monclient: _send_mon_message to mon.e at v2:172.21.15.88:3304/0 2023-09-30T01:33:06.201 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.186+0000 7fc8c5fa1700 1 -- 172.21.15.88:0/2513588189 --> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc8c01340f0 con 0x7fc8c00990f0 2023-09-30T01:33:06.202 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.186+0000 7fc8c5fa1700 10 monclient: _renew_subs 2023-09-30T01:33:06.202 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.186+0000 7fc8c5fa1700 10 monclient: _send_mon_message to mon.e at v2:172.21.15.88:3304/0 2023-09-30T01:33:06.202 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.186+0000 7fc8c5fa1700 1 -- 172.21.15.88:0/2513588189 --> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc8c0133200 con 0x7fc8c00990f0 2023-09-30T01:33:06.202 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.186+0000 7fc8bcff9700 1 -- 172.21.15.88:0/2513588189 <== mon.8 v2:172.21.15.88:3304/0 4 ==== mgrmap(e 5) v1 ==== 40144+0+0 (secure 0 0 0) 0x7fc8b0004b60 con 0x7fc8c00990f0 2023-09-30T01:33:06.202 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.186+0000 7fc8bcff9700 1 --2- 172.21.15.88:0/2513588189 >> v2:172.21.15.107:6812/29926 conn(0x7fc8a8033b60 0x7fc8a8036010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:33:06.202 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.186+0000 7fc8bcff9700 1 -- 172.21.15.88:0/2513588189 <== mon.8 v2:172.21.15.88:3304/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 3502+0+0 (secure 0 0 0) 0x7fc8b00431f0 con 0x7fc8c00990f0 2023-09-30T01:33:06.202 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.186+0000 7fc8bffff700 1 --2- 172.21.15.88:0/2513588189 >> v2:172.21.15.107:6812/29926 conn(0x7fc8a8033b60 0x7fc8a8036010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:33:06.202 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.186+0000 7fc8bffff700 10 monclient: get_auth_request con 0x7fc8a8033b60 auth_method 0 2023-09-30T01:33:06.202 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.186+0000 7fc8c5fa1700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:33:06.202 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.186+0000 7fc8c5fa1700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:33:06.202 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.186+0000 7fc8c5fa1700 10 monclient: _send_mon_message to mon.e at v2:172.21.15.88:3304/0 2023-09-30T01:33:06.203 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.186+0000 7fc8c5fa1700 1 -- 172.21.15.88:0/2513588189 --> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc8a00050b0 con 0x7fc8c00990f0 2023-09-30T01:33:06.203 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.186+0000 7fc8bffff700 1 --2- 172.21.15.88:0/2513588189 >> v2:172.21.15.107:6812/29926 conn(0x7fc8a8033b60 0x7fc8a8036010 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fc8c0071da0 tx=0x7fc8b4008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:33:06.203 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.194+0000 7fc8bcff9700 1 -- 172.21.15.88:0/2513588189 <== mon.8 v2:172.21.15.88:3304/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fc8b0014580 con 0x7fc8c00990f0 2023-09-30T01:33:06.203 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.194+0000 7fc8bcff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:33:06.203 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.194+0000 7fc8bcff9700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:33:06.225 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.222+0000 7fb583bad700 10 monclient: get_monmap_and_config 2023-09-30T01:33:06.225 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.222+0000 7fb583bad700 10 monclient: build_initial_monmap 2023-09-30T01:33:06.225 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.222+0000 7fb583bad700 10 monclient: monmap: 2023-09-30T01:33:06.225 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2023-09-30T01:33:06.225 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:33:06.225 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:33:06.225498+0000 2023-09-30T01:33:06.225 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:33:06.225498+0000 2023-09-30T01:33:06.225 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2023-09-30T01:33:06.225 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:33:06.225 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a-legacy 2023-09-30T01:33:06.226 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.107:6789/0 mon.noname-f-legacy 2023-09-30T01:33:06.226 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.noname-b 2023-09-30T01:33:06.226 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.noname-g 2023-09-30T01:33:06.226 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.noname-c 2023-09-30T01:33:06.226 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.noname-h 2023-09-30T01:33:06.226 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.noname-d 2023-09-30T01:33:06.226 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.noname-i 2023-09-30T01:33:06.226 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.noname-e 2023-09-30T01:33:06.226 INFO:teuthology.orchestra.run.smithi088.stderr:9: v2:172.21.15.88:3300/0 mon.noname-a 2023-09-30T01:33:06.226 INFO:teuthology.orchestra.run.smithi088.stderr:10: v2:172.21.15.107:3300/0 mon.noname-f 2023-09-30T01:33:06.226 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:33:06.229 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.222+0000 7fb583bad700 1 Processor -- start 2023-09-30T01:33:06.229 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.222+0000 7fb583bad700 1 -- start start 2023-09-30T01:33:06.229 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.222+0000 7fb583bad700 10 monclient: init 2023-09-30T01:33:06.229 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.222+0000 7fb583bad700 10 monclient: _reopen_session rank -1 2023-09-30T01:33:06.230 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.222+0000 7fb583bad700 10 monclient: _add_conns ranks=[4,3,6,9,10,8,0,2,7,1,5] 2023-09-30T01:33:06.230 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.222+0000 7fb583bad700 1 --2- >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7fb57c09f9e0 0x7fb57c098c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:33:06.230 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.222+0000 7fb583bad700 10 monclient(hunting): picked mon.noname-c con 0x7fb57c09f9e0 addr [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] 2023-09-30T01:33:06.230 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.222+0000 7fb583bad700 1 --2- >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7fb57c09a460 0x7fb57c06f070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:33:06.230 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.222+0000 7fb583bad700 10 monclient(hunting): picked mon.noname-g con 0x7fb57c09a460 addr [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] 2023-09-30T01:33:06.230 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.222+0000 7fb583bad700 1 --2- >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7fb57c062c70 0x7fb57c063040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:33:06.230 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.222+0000 7fb583bad700 10 monclient(hunting): picked mon.noname-d con 0x7fb57c062c70 addr [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] 2023-09-30T01:33:06.230 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.222+0000 7fb583bad700 10 monclient(hunting): start opening mon connection 2023-09-30T01:33:06.230 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.222+0000 7fb583bad700 1 -- --> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb57c09fdb0 con 0x7fb57c09a460 2023-09-30T01:33:06.231 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.222+0000 7fb583bad700 10 monclient(hunting): start opening mon connection 2023-09-30T01:33:06.231 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.222+0000 7fb583bad700 1 -- --> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] -- mon_getmap magic: 0 v1 -- 0x7fb57c09bb20 con 0x7fb57c09f9e0 2023-09-30T01:33:06.231 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.222+0000 7fb583bad700 10 monclient(hunting): start opening mon connection 2023-09-30T01:33:06.231 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.222+0000 7fb583bad700 1 -- --> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] -- mon_getmap magic: 0 v1 -- 0x7fb57c09b840 con 0x7fb57c062c70 2023-09-30T01:33:06.231 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.222+0000 7fb583bad700 10 monclient(hunting): _renew_subs 2023-09-30T01:33:06.231 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.222+0000 7fb583bad700 10 monclient(hunting): authenticate will time out at 858.263203s 2023-09-30T01:33:06.231 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.222+0000 7fb581949700 1 --2- >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7fb57c09f9e0 0x7fb57c098c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:33:06.231 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.222+0000 7fb581949700 1 --2- >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7fb57c09f9e0 0x7fb57c098c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3302/0 says I am v2:172.21.15.88:50454/0 (socket says 172.21.15.88:50454) 2023-09-30T01:33:06.231 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.222+0000 7fb581949700 1 -- 172.21.15.88:0/3265574889 learned_addr learned my addr 172.21.15.88:0/3265574889 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-09-30T01:33:06.231 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.222+0000 7fb581949700 10 monclient(hunting): get_auth_request con 0x7fb57c09f9e0 auth_method 0 2023-09-30T01:33:06.231 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.222+0000 7fb581949700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:33:06.231 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.222+0000 7fb581949700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:33:06.232 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.222+0000 7fb581949700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:33:06.232 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.222+0000 7fb58214a700 1 --2- 172.21.15.88:0/3265574889 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7fb57c062c70 0x7fb57c063040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:33:06.232 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.222+0000 7fb58214a700 10 monclient(hunting): get_auth_request con 0x7fb57c062c70 auth_method 0 2023-09-30T01:33:06.232 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.222+0000 7fb58214a700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:33:06.232 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.222+0000 7fb58214a700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:33:06.232 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.222+0000 7fb58214a700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:33:06.232 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.222+0000 7fb58214a700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:33:06.232 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.222+0000 7fb58214a700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:33:06.232 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.222+0000 7fb58214a700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:33:06.232 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.222+0000 7fb581148700 1 --2- 172.21.15.88:0/3265574889 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7fb57c09a460 0x7fb57c06f070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:33:06.232 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.222+0000 7fb581148700 10 monclient(hunting): get_auth_request con 0x7fb57c09a460 auth_method 0 2023-09-30T01:33:06.233 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.222+0000 7fb581148700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:33:06.233 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.222+0000 7fb581148700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:33:06.233 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.222+0000 7fb581148700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:33:06.233 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.222+0000 7fb581949700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:33:06.233 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.222+0000 7fb581949700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:33:06.233 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.222+0000 7fb581949700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:33:06.233 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.226+0000 7fb581148700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:33:06.233 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.226+0000 7fb581148700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:33:06.233 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.226+0000 7fb581148700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:33:06.233 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.226+0000 7fb58214a700 10 monclient(hunting): handle_auth_done global_id 4272 payload 274 2023-09-30T01:33:06.233 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.226+0000 7fb58214a700 1 -- 172.21.15.88:0/3265574889 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7fb57c09f9e0 msgr2=0x7fb57c098c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:33:06.234 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.226+0000 7fb58214a700 1 --2- 172.21.15.88:0/3265574889 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7fb57c09f9e0 0x7fb57c098c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:06.234 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.226+0000 7fb58214a700 1 -- 172.21.15.88:0/3265574889 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7fb57c09a460 msgr2=0x7fb57c06f070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:33:06.234 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.226+0000 7fb58214a700 1 --2- 172.21.15.88:0/3265574889 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7fb57c09a460 0x7fb57c06f070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:06.234 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.226+0000 7fb58214a700 10 monclient: _finish_hunting 0 2023-09-30T01:33:06.234 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.226+0000 7fb58214a700 1 monclient: found mon.noname-d 2023-09-30T01:33:06.234 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.226+0000 7fb58214a700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:33:06.234 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.226+0000 7fb58214a700 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.88:3303/0 2023-09-30T01:33:06.234 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.226+0000 7fb58214a700 1 -- 172.21.15.88:0/3265574889 --> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb57c05ecb0 con 0x7fb57c062c70 2023-09-30T01:33:06.234 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.226+0000 7fb581949700 1 --2- 172.21.15.88:0/3265574889 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7fb57c09f9e0 0x7fb57c098c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:33:06.234 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.226+0000 7fb58214a700 1 --2- 172.21.15.88:0/3265574889 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7fb57c062c70 0x7fb57c063040 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fb57400af20 tx=0x7fb57400a7d0 comp rx=0 tx=0).ready entity=mon.6 client_cookie=a148a927cd92ce8b server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:33:06.234 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.226+0000 7fb580947700 1 -- 172.21.15.88:0/3265574889 <== mon.6 v2:172.21.15.88:3303/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7fb574015070 con 0x7fb57c062c70 2023-09-30T01:33:06.235 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.226+0000 7fb580947700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:33:06.235 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.226+0000 7fb580947700 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2023-09-30T01:33:06.235 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.226+0000 7fb580947700 10 monclient: dump: 2023-09-30T01:33:06.235 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:33:06.235 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:33:06.235 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:06.235 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:06.235 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:33:06.235 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:33:06.235 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:33:06.235 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:33:06.235 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:33:06.236 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:33:06.236 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:33:06.236 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:33:06.236 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:33:06.236 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:33:06.236 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:33:06.236 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:33:06.236 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.226+0000 7fb580947700 10 monclient: _finish_auth 0 2023-09-30T01:33:06.236 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.226+0000 7fb580947700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:33:06.236 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.226+0000 7fb583bad700 5 monclient: authenticate success, global_id 4272 2023-09-30T01:33:06.236 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.226+0000 7fb583bad700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:33:06.237 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.226+0000 7fb580947700 1 -- 172.21.15.88:0/3265574889 <== mon.6 v2:172.21.15.88:3303/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb57400a2e0 con 0x7fb57c062c70 2023-09-30T01:33:06.237 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.226+0000 7fb580947700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:33:06.237 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.226+0000 7fb583bad700 10 monclient: get_monmap_and_config success 2023-09-30T01:33:06.237 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.226+0000 7fb583bad700 4 set_mon_vals no callback set 2023-09-30T01:33:06.237 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.226+0000 7fb580947700 1 -- 172.21.15.88:0/3265574889 <== mon.6 v2:172.21.15.88:3303/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7fb57400bde0 con 0x7fb57c062c70 2023-09-30T01:33:06.237 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.226+0000 7fb580947700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:33:06.237 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.226+0000 7fb580947700 10 monclient: got monmap 1 from mon.d (according to old e1) 2023-09-30T01:33:06.237 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.226+0000 7fb580947700 10 monclient: dump: 2023-09-30T01:33:06.237 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:33:06.237 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:33:06.237 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:06.237 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:06.238 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:33:06.238 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:33:06.238 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:33:06.238 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:33:06.238 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:33:06.238 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:33:06.238 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:33:06.238 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:33:06.238 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:33:06.238 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:33:06.238 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:33:06.238 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:33:06.239 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.226+0000 7fb583bad700 10 monclient: shutdown 2023-09-30T01:33:06.239 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.226+0000 7fb583bad700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:33:06.239 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.226+0000 7fb583bad700 1 -- 172.21.15.88:0/3265574889 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7fb57c062c70 msgr2=0x7fb57c063040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:33:06.239 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.226+0000 7fb583bad700 1 --2- 172.21.15.88:0/3265574889 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7fb57c062c70 0x7fb57c063040 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fb57400af20 tx=0x7fb57400a7d0 comp rx=0 tx=0).stop 2023-09-30T01:33:06.240 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.230+0000 7fb583bad700 1 -- 172.21.15.88:0/3265574889 shutdown_connections 2023-09-30T01:33:06.240 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.230+0000 7fb583bad700 1 --2- 172.21.15.88:0/3265574889 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7fb57c062c70 0x7fb57c063040 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:06.240 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.230+0000 7fb583bad700 1 --2- 172.21.15.88:0/3265574889 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7fb57c09a460 0x7fb57c06f070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:06.240 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.230+0000 7fb583bad700 1 --2- 172.21.15.88:0/3265574889 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7fb57c09f9e0 0x7fb57c098c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:06.240 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.230+0000 7fb583bad700 1 -- 172.21.15.88:0/3265574889 >> 172.21.15.88:0/3265574889 conn(0x7fb57c0a2660 msgr2=0x7fb57c0a2a30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:33:06.240 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.230+0000 7fb583bad700 1 -- 172.21.15.88:0/3265574889 shutdown_connections 2023-09-30T01:33:06.240 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.230+0000 7fb583bad700 1 -- 172.21.15.88:0/3265574889 wait complete. 2023-09-30T01:33:06.240 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.230+0000 7fb583bad700 10 monclient: build_initial_monmap 2023-09-30T01:33:06.240 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.230+0000 7fb583bad700 10 monclient: monmap: 2023-09-30T01:33:06.241 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2023-09-30T01:33:06.241 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:33:06.241 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 0.000000 2023-09-30T01:33:06.241 INFO:teuthology.orchestra.run.smithi088.stderr:created 0.000000 2023-09-30T01:33:06.241 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2023-09-30T01:33:06.241 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 1 2023-09-30T01:33:06.241 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.noname-a 2023-09-30T01:33:06.241 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.noname-f 2023-09-30T01:33:06.241 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.noname-b 2023-09-30T01:33:06.241 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.noname-g 2023-09-30T01:33:06.241 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.noname-c 2023-09-30T01:33:06.241 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.noname-h 2023-09-30T01:33:06.242 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.noname-d 2023-09-30T01:33:06.242 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.noname-i 2023-09-30T01:33:06.242 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.noname-e 2023-09-30T01:33:06.242 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:33:06.242 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.230+0000 7f54e16e1700 10 monclient: get_monmap_and_config 2023-09-30T01:33:06.242 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.230+0000 7f54e16e1700 10 monclient: build_initial_monmap 2023-09-30T01:33:06.242 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.230+0000 7f54e16e1700 10 monclient: monmap: 2023-09-30T01:33:06.242 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2023-09-30T01:33:06.243 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:33:06.243 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:33:06.232955+0000 2023-09-30T01:33:06.243 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:33:06.232955+0000 2023-09-30T01:33:06.243 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2023-09-30T01:33:06.243 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:33:06.243 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a-legacy 2023-09-30T01:33:06.243 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.107:6789/0 mon.noname-f-legacy 2023-09-30T01:33:06.243 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.noname-b 2023-09-30T01:33:06.243 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.noname-g 2023-09-30T01:33:06.243 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.noname-c 2023-09-30T01:33:06.243 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.noname-h 2023-09-30T01:33:06.243 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.noname-d 2023-09-30T01:33:06.244 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.noname-i 2023-09-30T01:33:06.244 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.noname-e 2023-09-30T01:33:06.244 INFO:teuthology.orchestra.run.smithi088.stderr:9: v2:172.21.15.88:3300/0 mon.noname-a 2023-09-30T01:33:06.244 INFO:teuthology.orchestra.run.smithi088.stderr:10: v2:172.21.15.107:3300/0 mon.noname-f 2023-09-30T01:33:06.244 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:33:06.244 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.230+0000 7fb583bad700 1 Processor -- start 2023-09-30T01:33:06.245 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.230+0000 7fb583bad700 1 -- start start 2023-09-30T01:33:06.245 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.230+0000 7fb583bad700 10 monclient: init 2023-09-30T01:33:06.245 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.230+0000 7f54e16e1700 1 Processor -- start 2023-09-30T01:33:06.245 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.230+0000 7f54e16e1700 1 -- start start 2023-09-30T01:33:06.245 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.230+0000 7f54e16e1700 10 monclient: init 2023-09-30T01:33:06.246 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.230+0000 7fb583bad700 10 monclient: _reopen_session rank -1 2023-09-30T01:33:06.246 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.230+0000 7fb583bad700 10 monclient: _add_conns ranks=[1,3,0,7,5,8,2,6,4] 2023-09-30T01:33:06.247 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.230+0000 7fb583bad700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb57c062c70 0x7fb57c095bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:33:06.247 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.230+0000 7fb583bad700 10 monclient(hunting): picked mon.noname-f con 0x7fb57c062c70 addr [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] 2023-09-30T01:33:06.247 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.230+0000 7fb583bad700 1 --2- >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7fb57c09a460 0x7fb57c06cf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:33:06.247 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.230+0000 7fb583bad700 10 monclient(hunting): picked mon.noname-g con 0x7fb57c09a460 addr [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] 2023-09-30T01:33:06.247 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.230+0000 7fb583bad700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb57c09f9e0 0x7fb57c06d4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:33:06.247 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.230+0000 7fb583bad700 10 monclient(hunting): picked mon.noname-a con 0x7fb57c09f9e0 addr [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] 2023-09-30T01:33:06.247 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.230+0000 7fb583bad700 10 monclient(hunting): start opening mon connection 2023-09-30T01:33:06.247 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.230+0000 7fb583bad700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb57c093030 con 0x7fb57c09f9e0 2023-09-30T01:33:06.247 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.230+0000 7fb583bad700 10 monclient(hunting): start opening mon connection 2023-09-30T01:33:06.247 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.230+0000 7fb583bad700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb57c092d50 con 0x7fb57c062c70 2023-09-30T01:33:06.247 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.230+0000 7fb583bad700 10 monclient(hunting): start opening mon connection 2023-09-30T01:33:06.248 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.230+0000 7fb583bad700 1 -- --> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb57c093310 con 0x7fb57c09a460 2023-09-30T01:33:06.248 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.230+0000 7fb583bad700 10 monclient(hunting): _renew_subs 2023-09-30T01:33:06.248 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.230+0000 7fb583bad700 10 monclient(hunting): authenticate will time out at 858.270491s 2023-09-30T01:33:06.248 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.230+0000 7f54e16e1700 10 monclient: _reopen_session rank -1 2023-09-30T01:33:06.248 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.230+0000 7fb58214a700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb57c09f9e0 0x7fb57c06d4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:33:06.249 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.230+0000 7f54e16e1700 10 monclient: _add_conns ranks=[6,8,10,4,5,1,9,7,3,0,2] 2023-09-30T01:33:06.249 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.230+0000 7f54e16e1700 1 --2- >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7f54dc0966a0 0x7f54dc096a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:33:06.249 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.230+0000 7f54e16e1700 10 monclient(hunting): picked mon.noname-d con 0x7f54dc0966a0 addr [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] 2023-09-30T01:33:06.249 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.230+0000 7fb581949700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb57c062c70 0x7fb57c095bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:33:06.249 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.230+0000 7fb58214a700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb57c09f9e0 0x7fb57c06d4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:57906/0 (socket says 172.21.15.88:57906) 2023-09-30T01:33:06.250 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.230+0000 7fb58214a700 1 -- 172.21.15.88:0/1762600973 learned_addr learned my addr 172.21.15.88:0/1762600973 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-09-30T01:33:06.250 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.230+0000 7fb58214a700 10 monclient(hunting): get_auth_request con 0x7fb57c09f9e0 auth_method 0 2023-09-30T01:33:06.250 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.230+0000 7fb58214a700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:33:06.250 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.230+0000 7fb58214a700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:33:06.250 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.230+0000 7fb58214a700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:33:06.250 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.230+0000 7fb581949700 10 monclient(hunting): get_auth_request con 0x7fb57c062c70 auth_method 0 2023-09-30T01:33:06.250 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.230+0000 7fb581949700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:33:06.250 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.230+0000 7fb581949700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:33:06.250 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.230+0000 7fb581949700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:33:06.250 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.230+0000 7f54e16e1700 1 --2- >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f54dc097cd0 0x7f54dc0980a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:33:06.251 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.230+0000 7f54e16e1700 10 monclient(hunting): picked mon.noname-e con 0x7f54dc097cd0 addr [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] 2023-09-30T01:33:06.251 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.230+0000 7fb581148700 1 --2- 172.21.15.88:0/1762600973 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7fb57c09a460 0x7fb57c06cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:33:06.251 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.230+0000 7f54e16e1700 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f54dc09ed80 0x7f54dc09f1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:33:06.251 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.230+0000 7f54e16e1700 10 monclient(hunting): picked mon.noname-f con 0x7f54dc09ed80 addr v2:172.21.15.107:3300/0 2023-09-30T01:33:06.251 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.230+0000 7f54e16e1700 10 monclient(hunting): start opening mon connection 2023-09-30T01:33:06.251 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.230+0000 7f54e16e1700 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f54dc097920 con 0x7f54dc09ed80 2023-09-30T01:33:06.252 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.230+0000 7f54e16e1700 10 monclient(hunting): start opening mon connection 2023-09-30T01:33:06.252 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.230+0000 7f54e16e1700 1 -- --> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] -- mon_getmap magic: 0 v1 -- 0x7f54dc09a340 con 0x7f54dc0966a0 2023-09-30T01:33:06.252 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.230+0000 7f54e16e1700 10 monclient(hunting): start opening mon connection 2023-09-30T01:33:06.252 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.230+0000 7f54e16e1700 1 -- --> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] -- mon_getmap magic: 0 v1 -- 0x7f54dc0975d0 con 0x7f54dc097cd0 2023-09-30T01:33:06.252 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.230+0000 7f54e16e1700 10 monclient(hunting): _renew_subs 2023-09-30T01:33:06.252 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.230+0000 7f54e16e1700 10 monclient(hunting): authenticate will time out at 858.270927s 2023-09-30T01:33:06.253 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.230+0000 7fb581148700 10 monclient(hunting): get_auth_request con 0x7fb57c09a460 auth_method 0 2023-09-30T01:33:06.253 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.230+0000 7fb581148700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:33:06.253 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.230+0000 7fb581148700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:33:06.253 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.230+0000 7fb581148700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:33:06.253 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.230+0000 7fb58214a700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:33:06.253 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.230+0000 7fb58214a700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:33:06.253 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.230+0000 7fb58214a700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:33:06.253 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.230+0000 7fb581949700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:33:06.253 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.230+0000 7fb581949700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:33:06.253 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.230+0000 7fb581949700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:33:06.254 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.230+0000 7f54e0ee0700 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f54dc09ed80 0x7f54dc09f1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:33:06.254 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.230+0000 7fb581148700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:33:06.254 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.230+0000 7fb581148700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:33:06.254 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.230+0000 7fb581148700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:33:06.254 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.230+0000 7f54e0ee0700 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f54dc09ed80 0x7f54dc09f1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.88:57644/0 (socket says 172.21.15.88:57644) 2023-09-30T01:33:06.255 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.230+0000 7f54e0ee0700 1 -- 172.21.15.88:0/701772924 learned_addr learned my addr 172.21.15.88:0/701772924 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-09-30T01:33:06.255 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.230+0000 7f54e0ee0700 10 monclient(hunting): get_auth_request con 0x7f54dc09ed80 auth_method 0 2023-09-30T01:33:06.255 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.230+0000 7f54e0ee0700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:33:06.255 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.230+0000 7f54e0ee0700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:33:06.255 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.230+0000 7f54e0ee0700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:33:06.255 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.230+0000 7fb58214a700 10 monclient(hunting): handle_auth_done global_id 4401 payload 824 2023-09-30T01:33:06.255 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.230+0000 7f54dbfff700 1 --2- 172.21.15.88:0/701772924 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7f54dc0966a0 0x7f54dc096a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:33:06.256 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.230+0000 7fb58214a700 1 -- 172.21.15.88:0/1762600973 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7fb57c09a460 msgr2=0x7fb57c06cf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:33:06.256 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.230+0000 7fb58214a700 1 --2- 172.21.15.88:0/1762600973 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7fb57c09a460 0x7fb57c06cf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:06.256 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.230+0000 7fb58214a700 1 -- 172.21.15.88:0/1762600973 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb57c062c70 msgr2=0x7fb57c095bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:33:06.256 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.230+0000 7fb58214a700 1 --2- 172.21.15.88:0/1762600973 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb57c062c70 0x7fb57c095bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:06.256 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.230+0000 7fb58214a700 10 monclient: _finish_hunting 0 2023-09-30T01:33:06.256 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.230+0000 7fb58214a700 1 monclient: found mon.noname-a 2023-09-30T01:33:06.257 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.230+0000 7fb58214a700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:33:06.257 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.230+0000 7fb58214a700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.88:3300/0 2023-09-30T01:33:06.257 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.230+0000 7fb58214a700 1 -- 172.21.15.88:0/1762600973 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb578009480 con 0x7fb57c09f9e0 2023-09-30T01:33:06.258 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.230+0000 7f54dbfff700 10 monclient(hunting): get_auth_request con 0x7f54dc0966a0 auth_method 0 2023-09-30T01:33:06.258 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.230+0000 7f54dbfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:33:06.258 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.230+0000 7f54dbfff700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:33:06.258 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.230+0000 7f54dbfff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:33:06.258 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.230+0000 7f54dbfff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:33:06.258 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.230+0000 7f54dbfff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:33:06.258 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.230+0000 7f54db7fe700 1 --2- 172.21.15.88:0/701772924 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f54dc097cd0 0x7f54dc0980a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:33:06.258 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.230+0000 7f54dbfff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:33:06.258 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.230+0000 7f54db7fe700 10 monclient(hunting): get_auth_request con 0x7f54dc097cd0 auth_method 0 2023-09-30T01:33:06.259 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.230+0000 7f54db7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:33:06.259 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.230+0000 7f54db7fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:33:06.259 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.230+0000 7f54db7fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:33:06.259 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.230+0000 7f54e0ee0700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:33:06.259 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.230+0000 7f54e0ee0700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:33:06.259 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.230+0000 7f54e0ee0700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:33:06.259 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.230+0000 7f54db7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:33:06.259 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.230+0000 7f54db7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:33:06.259 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.230+0000 7f54db7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:33:06.259 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.230+0000 7f54dbfff700 10 monclient(hunting): handle_auth_done global_id 4281 payload 274 2023-09-30T01:33:06.259 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.230+0000 7f54dbfff700 1 -- 172.21.15.88:0/701772924 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f54dc097cd0 msgr2=0x7f54dc0980a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:33:06.260 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.230+0000 7f54dbfff700 1 --2- 172.21.15.88:0/701772924 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f54dc097cd0 0x7f54dc0980a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:06.260 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.230+0000 7f54dbfff700 1 -- 172.21.15.88:0/701772924 >> v2:172.21.15.107:3300/0 conn(0x7f54dc09ed80 msgr2=0x7f54dc09f1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:33:06.260 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.230+0000 7f54dbfff700 1 --2- 172.21.15.88:0/701772924 >> v2:172.21.15.107:3300/0 conn(0x7f54dc09ed80 0x7f54dc09f1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:06.260 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.230+0000 7fb58214a700 1 --2- 172.21.15.88:0/1762600973 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb57c09f9e0 0x7fb57c06d4d0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fb574012000 tx=0x7fb574015000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:33:06.260 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.230+0000 7f54dbfff700 10 monclient: _finish_hunting 0 2023-09-30T01:33:06.260 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.230+0000 7f54dbfff700 1 monclient: found mon.noname-d 2023-09-30T01:33:06.260 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.230+0000 7f54dbfff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:33:06.260 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.230+0000 7f54dbfff700 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.88:3303/0 2023-09-30T01:33:06.261 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.234+0000 7fb571ffb700 1 -- 172.21.15.88:0/1762600973 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7fb5740032e0 con 0x7fb57c09f9e0 2023-09-30T01:33:06.261 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.234+0000 7fb571ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:33:06.261 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.234+0000 7fb571ffb700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T01:33:06.261 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.230+0000 7f54dbfff700 1 -- 172.21.15.88:0/701772924 --> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54dc04bb20 con 0x7f54dc0966a0 2023-09-30T01:33:06.261 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.234+0000 7f54dbfff700 1 --2- 172.21.15.88:0/701772924 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7f54dc0966a0 0x7f54dc096a70 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f54d00098d0 tx=0x7f54d0009120 comp rx=0 tx=0).ready entity=mon.6 client_cookie=1604a47dbd845ea9 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:33:06.262 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.234+0000 7fb571ffb700 10 monclient: dump: 2023-09-30T01:33:06.262 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:33:06.262 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:33:06.262 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:06.262 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:06.262 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:33:06.262 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:33:06.262 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:33:06.263 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:33:06.263 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:33:06.263 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:33:06.263 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:33:06.263 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:33:06.263 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:33:06.263 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:33:06.263 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:33:06.263 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:33:06.263 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.234+0000 7fb571ffb700 10 monclient: _finish_auth 0 2023-09-30T01:33:06.263 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.234+0000 7fb571ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:33:06.263 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.234+0000 7fb571ffb700 1 -- 172.21.15.88:0/1762600973 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb5740088d0 con 0x7fb57c09f9e0 2023-09-30T01:33:06.263 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.234+0000 7fb571ffb700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:33:06.264 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.234+0000 7fb571ffb700 1 -- 172.21.15.88:0/1762600973 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7fb57401c430 con 0x7fb57c09f9e0 2023-09-30T01:33:06.264 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.234+0000 7fb571ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:33:06.264 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.234+0000 7fb571ffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:33:06.264 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.234+0000 7fb573fff700 4 set_mon_vals no callback set 2023-09-30T01:33:06.264 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.234+0000 7fb571ffb700 10 monclient: dump: 2023-09-30T01:33:06.264 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:33:06.264 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:33:06.264 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:06.264 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:06.264 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:33:06.264 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:33:06.264 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:33:06.265 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:33:06.265 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:33:06.265 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:33:06.265 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:33:06.265 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:33:06.265 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:33:06.265 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:33:06.265 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:33:06.265 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:33:06.265 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.234+0000 7fb583bad700 5 monclient: authenticate success, global_id 4401 2023-09-30T01:33:06.266 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.234+0000 7f54daffd700 1 -- 172.21.15.88:0/701772924 <== mon.6 v2:172.21.15.88:3303/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f54d0005c10 con 0x7f54dc0966a0 2023-09-30T01:33:06.266 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.234+0000 7f54daffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:33:06.266 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.234+0000 7f54daffd700 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2023-09-30T01:33:06.266 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.234+0000 7f54daffd700 10 monclient: dump: 2023-09-30T01:33:06.266 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:33:06.266 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:33:06.266 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:06.267 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:06.267 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:33:06.267 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:33:06.267 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:33:06.267 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:33:06.267 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:33:06.267 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:33:06.267 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:33:06.267 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:33:06.267 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:33:06.267 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:33:06.267 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:33:06.268 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:33:06.268 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.234+0000 7f54daffd700 10 monclient: _finish_auth 0 2023-09-30T01:33:06.268 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.234+0000 7f54daffd700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:33:06.268 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.234+0000 7f54daffd700 1 -- 172.21.15.88:0/701772924 <== mon.6 v2:172.21.15.88:3303/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f54d0005d90 con 0x7f54dc0966a0 2023-09-30T01:33:06.268 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.234+0000 7f54daffd700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:33:06.268 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.234+0000 7f54daffd700 1 -- 172.21.15.88:0/701772924 <== mon.6 v2:172.21.15.88:3303/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f54d0011420 con 0x7f54dc0966a0 2023-09-30T01:33:06.268 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.234+0000 7f54daffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:33:06.268 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.234+0000 7f54daffd700 10 monclient: got monmap 1 from mon.d (according to old e1) 2023-09-30T01:33:06.268 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.234+0000 7f54daffd700 10 monclient: dump: 2023-09-30T01:33:06.268 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:33:06.268 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:33:06.268 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:06.269 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:06.269 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:33:06.269 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:33:06.269 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:33:06.269 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:33:06.269 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:33:06.269 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:33:06.269 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:33:06.269 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:33:06.269 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:33:06.269 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:33:06.269 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:33:06.269 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:33:06.270 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.234+0000 7f54e16e1700 5 monclient: authenticate success, global_id 4281 2023-09-30T01:33:06.270 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.234+0000 7f54e16e1700 10 monclient: get_monmap_and_config success 2023-09-30T01:33:06.270 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.234+0000 7f54e16e1700 4 set_mon_vals no callback set 2023-09-30T01:33:06.270 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.234+0000 7fb583bad700 10 monclient: _renew_subs 2023-09-30T01:33:06.270 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.234+0000 7fb583bad700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.88:3300/0 2023-09-30T01:33:06.270 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.234+0000 7fb583bad700 1 -- 172.21.15.88:0/1762600973 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb57400aa30 con 0x7fb57c09f9e0 2023-09-30T01:33:06.270 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.234+0000 7fb583bad700 10 monclient: _renew_subs 2023-09-30T01:33:06.270 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.234+0000 7fb583bad700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.88:3300/0 2023-09-30T01:33:06.271 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.234+0000 7fb583bad700 1 -- 172.21.15.88:0/1762600973 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb57c06eab0 con 0x7fb57c09f9e0 2023-09-30T01:33:06.271 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.230+0000 7f0bd1292700 10 monclient: get_monmap_and_config 2023-09-30T01:33:06.271 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.230+0000 7f0bd1292700 10 monclient: build_initial_monmap 2023-09-30T01:33:06.271 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.230+0000 7f0bd1292700 10 monclient: monmap: 2023-09-30T01:33:06.271 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2023-09-30T01:33:06.271 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:33:06.272 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:33:06.233893+0000 2023-09-30T01:33:06.272 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:33:06.233893+0000 2023-09-30T01:33:06.272 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2023-09-30T01:33:06.272 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:33:06.272 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a-legacy 2023-09-30T01:33:06.272 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.107:6789/0 mon.noname-f-legacy 2023-09-30T01:33:06.272 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.noname-b 2023-09-30T01:33:06.272 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.noname-g 2023-09-30T01:33:06.272 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.noname-c 2023-09-30T01:33:06.272 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.noname-h 2023-09-30T01:33:06.272 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.noname-d 2023-09-30T01:33:06.272 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.noname-i 2023-09-30T01:33:06.272 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.noname-e 2023-09-30T01:33:06.273 INFO:teuthology.orchestra.run.smithi088.stderr:9: v2:172.21.15.88:3300/0 mon.noname-a 2023-09-30T01:33:06.273 INFO:teuthology.orchestra.run.smithi088.stderr:10: v2:172.21.15.107:3300/0 mon.noname-f 2023-09-30T01:33:06.273 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:33:06.273 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.234+0000 7f54e16e1700 10 monclient: shutdown 2023-09-30T01:33:06.273 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.234+0000 7f54e16e1700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:33:06.273 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.234+0000 7f54e16e1700 1 -- 172.21.15.88:0/701772924 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7f54dc0966a0 msgr2=0x7f54dc096a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:33:06.273 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.234+0000 7f54e16e1700 1 --2- 172.21.15.88:0/701772924 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7f54dc0966a0 0x7f54dc096a70 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f54d00098d0 tx=0x7f54d0009120 comp rx=0 tx=0).stop 2023-09-30T01:33:06.274 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.234+0000 7fb571ffb700 1 -- 172.21.15.88:0/1762600973 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 40144+0+0 (secure 0 0 0) 0x7fb574003030 con 0x7fb57c09f9e0 2023-09-30T01:33:06.274 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.234+0000 7fb583bad700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:33:06.274 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.234+0000 7fb583bad700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:33:06.274 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.234+0000 7fb583bad700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.88:3300/0 2023-09-30T01:33:06.274 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.234+0000 7fb583bad700 1 -- 172.21.15.88:0/1762600973 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb57c1329f0 con 0x7fb57c09f9e0 2023-09-30T01:33:06.275 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.234+0000 7fb571ffb700 1 --2- 172.21.15.88:0/1762600973 >> v2:172.21.15.107:6812/29926 conn(0x7fb560033360 0x7fb560035810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:33:06.275 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.234+0000 7fb571ffb700 1 -- 172.21.15.88:0/1762600973 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 3502+0+0 (secure 0 0 0) 0x7fb574044fe0 con 0x7fb57c09f9e0 2023-09-30T01:33:06.275 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.234+0000 7f54e16e1700 1 -- 172.21.15.88:0/701772924 shutdown_connections 2023-09-30T01:33:06.275 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.234+0000 7f54e16e1700 1 --2- 172.21.15.88:0/701772924 >> v2:172.21.15.107:3300/0 conn(0x7f54dc09ed80 0x7f54dc09f1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:06.275 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.234+0000 7f54e16e1700 1 --2- 172.21.15.88:0/701772924 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f54dc097cd0 0x7f54dc0980a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:06.275 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.234+0000 7f54e16e1700 1 --2- 172.21.15.88:0/701772924 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7f54dc0966a0 0x7f54dc096a70 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:06.276 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.234+0000 7f54e16e1700 1 -- 172.21.15.88:0/701772924 >> 172.21.15.88:0/701772924 conn(0x7f54dc0a0910 msgr2=0x7f54dc0a33c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:33:06.276 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.234+0000 7f54e16e1700 1 -- 172.21.15.88:0/701772924 shutdown_connections 2023-09-30T01:33:06.276 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.234+0000 7fb581949700 1 --2- 172.21.15.88:0/1762600973 >> v2:172.21.15.107:6812/29926 conn(0x7fb560033360 0x7fb560035810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:33:06.276 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.234+0000 7fb581949700 10 monclient: get_auth_request con 0x7fb560033360 auth_method 0 2023-09-30T01:33:06.277 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.234+0000 7f54e16e1700 1 -- 172.21.15.88:0/701772924 wait complete. 2023-09-30T01:33:06.277 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.234+0000 7f54e16e1700 10 monclient: build_initial_monmap 2023-09-30T01:33:06.277 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.234+0000 7f54e16e1700 10 monclient: monmap: 2023-09-30T01:33:06.277 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2023-09-30T01:33:06.277 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:33:06.277 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 0.000000 2023-09-30T01:33:06.277 INFO:teuthology.orchestra.run.smithi088.stderr:created 0.000000 2023-09-30T01:33:06.277 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2023-09-30T01:33:06.277 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 1 2023-09-30T01:33:06.277 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.noname-a 2023-09-30T01:33:06.277 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.noname-f 2023-09-30T01:33:06.278 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.noname-b 2023-09-30T01:33:06.278 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.noname-g 2023-09-30T01:33:06.278 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.noname-c 2023-09-30T01:33:06.278 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.noname-h 2023-09-30T01:33:06.278 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.noname-d 2023-09-30T01:33:06.278 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.noname-i 2023-09-30T01:33:06.278 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.noname-e 2023-09-30T01:33:06.278 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:33:06.278 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.234+0000 7fb581949700 1 --2- 172.21.15.88:0/1762600973 >> v2:172.21.15.107:6812/29926 conn(0x7fb560033360 0x7fb560035810 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fb578009e90 tx=0x7fb578009220 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:33:06.282 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.234+0000 7f54e16e1700 1 Processor -- start 2023-09-30T01:33:06.282 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.234+0000 7f54e16e1700 1 -- start start 2023-09-30T01:33:06.282 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.234+0000 7f54e16e1700 10 monclient: init 2023-09-30T01:33:06.282 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.238+0000 7f54e16e1700 10 monclient: _reopen_session rank -1 2023-09-30T01:33:06.282 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.238+0000 7f54e16e1700 10 monclient: _add_conns ranks=[3,2,1,7,8,0,5,6,4] 2023-09-30T01:33:06.282 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.238+0000 7f54e16e1700 1 --2- >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7f54dc0966a0 0x7f54dc142690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:33:06.282 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.238+0000 7f54e16e1700 10 monclient(hunting): picked mon.noname-g con 0x7f54dc0966a0 addr [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] 2023-09-30T01:33:06.282 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.238+0000 7f54e16e1700 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f54dc097cd0 0x7f54dc142bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:33:06.282 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.238+0000 7f54e16e1700 10 monclient(hunting): picked mon.noname-b con 0x7f54dc097cd0 addr [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] 2023-09-30T01:33:06.282 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.238+0000 7f54e16e1700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f54dc09ed80 0x7f54dc143110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:33:06.283 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.238+0000 7f54e16e1700 10 monclient(hunting): picked mon.noname-f con 0x7f54dc09ed80 addr [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] 2023-09-30T01:33:06.283 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.238+0000 7f54e16e1700 10 monclient(hunting): start opening mon connection 2023-09-30T01:33:06.283 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.238+0000 7f54e16e1700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54dc0996c0 con 0x7f54dc09ed80 2023-09-30T01:33:06.283 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.238+0000 7f54e16e1700 10 monclient(hunting): start opening mon connection 2023-09-30T01:33:06.283 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.238+0000 7f54e16e1700 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54dc0904e0 con 0x7f54dc097cd0 2023-09-30T01:33:06.283 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.238+0000 7f54e16e1700 10 monclient(hunting): start opening mon connection 2023-09-30T01:33:06.283 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.238+0000 7f54e16e1700 1 -- --> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54dc08fe20 con 0x7f54dc0966a0 2023-09-30T01:33:06.283 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.238+0000 7f54e16e1700 10 monclient(hunting): _renew_subs 2023-09-30T01:33:06.283 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.238+0000 7f54e16e1700 10 monclient(hunting): authenticate will time out at 858.276533s 2023-09-30T01:33:06.283 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.238+0000 7f54db7fe700 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f54dc097cd0 0x7f54dc142bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:33:06.283 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.238+0000 7f54db7fe700 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f54dc097cd0 0x7f54dc142bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3301/0 says I am v2:172.21.15.88:52492/0 (socket says 172.21.15.88:52492) 2023-09-30T01:33:06.283 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.238+0000 7f54db7fe700 1 -- 172.21.15.88:0/1794351274 learned_addr learned my addr 172.21.15.88:0/1794351274 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-09-30T01:33:06.284 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.238+0000 7f54db7fe700 10 monclient(hunting): get_auth_request con 0x7f54dc097cd0 auth_method 0 2023-09-30T01:33:06.284 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.238+0000 7f54db7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:33:06.284 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.238+0000 7f54db7fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:33:06.284 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.238+0000 7f54db7fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:33:06.284 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.238+0000 7f54db7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:33:06.284 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.238+0000 7f54db7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:33:06.284 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.238+0000 7f54dbfff700 1 --2- 172.21.15.88:0/1794351274 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7f54dc0966a0 0x7f54dc142690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:33:06.284 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.238+0000 7f54db7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:33:06.284 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.238+0000 7f54e0ee0700 1 --2- 172.21.15.88:0/1794351274 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f54dc09ed80 0x7f54dc143110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:33:06.284 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.238+0000 7f54dbfff700 10 monclient(hunting): get_auth_request con 0x7f54dc0966a0 auth_method 0 2023-09-30T01:33:06.284 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.238+0000 7f54dbfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:33:06.285 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.238+0000 7f54dbfff700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:33:06.285 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.238+0000 7f54dbfff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:33:06.285 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.238+0000 7f54e0ee0700 10 monclient(hunting): get_auth_request con 0x7f54dc09ed80 auth_method 0 2023-09-30T01:33:06.285 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.238+0000 7f54e0ee0700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:33:06.285 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.238+0000 7f54e0ee0700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:33:06.285 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.238+0000 7f54e0ee0700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:33:06.285 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.238+0000 7f54db7fe700 10 monclient(hunting): handle_auth_done global_id 4322 payload 824 2023-09-30T01:33:06.285 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.238+0000 7f54db7fe700 1 -- 172.21.15.88:0/1794351274 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7f54dc0966a0 msgr2=0x7f54dc142690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:33:06.285 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.238+0000 7f54db7fe700 1 --2- 172.21.15.88:0/1794351274 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7f54dc0966a0 0x7f54dc142690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:06.285 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.238+0000 7f54db7fe700 1 -- 172.21.15.88:0/1794351274 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f54dc09ed80 msgr2=0x7f54dc143110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:33:06.285 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.238+0000 7f54db7fe700 1 --2- 172.21.15.88:0/1794351274 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f54dc09ed80 0x7f54dc143110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:06.285 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.238+0000 7f54db7fe700 10 monclient: _finish_hunting 0 2023-09-30T01:33:06.286 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.238+0000 7f54db7fe700 1 monclient: found mon.noname-b 2023-09-30T01:33:06.286 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.238+0000 7f54db7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:33:06.286 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.238+0000 7f54db7fe700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.88:3301/0 2023-09-30T01:33:06.286 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.238+0000 7f54db7fe700 1 -- 172.21.15.88:0/1794351274 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54d00093c0 con 0x7f54dc097cd0 2023-09-30T01:33:06.286 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.238+0000 7f54dbfff700 1 --2- 172.21.15.88:0/1794351274 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7f54dc0966a0 0x7f54dc142690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-30T01:33:06.286 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.238+0000 7f54e0ee0700 1 --2- 172.21.15.88:0/1794351274 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f54dc09ed80 0x7f54dc143110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-30T01:33:06.286 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.238+0000 7f54db7fe700 1 --2- 172.21.15.88:0/1794351274 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f54dc097cd0 0x7f54dc142bd0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f54d4009ef0 tx=0x7f54d400b040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:33:06.286 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.238+0000 7f54c3fff700 1 -- 172.21.15.88:0/1794351274 <== mon.2 v2:172.21.15.88:3301/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f54d400cc50 con 0x7f54dc097cd0 2023-09-30T01:33:06.286 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.238+0000 7f54c3fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:33:06.286 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.238+0000 7f54c3fff700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T01:33:06.286 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.238+0000 7f54c3fff700 10 monclient: dump: 2023-09-30T01:33:06.286 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:33:06.287 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:33:06.287 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:06.287 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:06.287 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:33:06.287 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:33:06.287 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:33:06.287 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:33:06.287 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:33:06.287 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:33:06.287 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:33:06.287 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:33:06.287 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:33:06.288 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:33:06.288 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:33:06.288 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:33:06.288 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.238+0000 7f54c3fff700 10 monclient: _finish_auth 0 2023-09-30T01:33:06.288 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.238+0000 7f54c3fff700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:33:06.288 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.238+0000 7f54c3fff700 1 -- 172.21.15.88:0/1794351274 <== mon.2 v2:172.21.15.88:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f54d400f040 con 0x7f54dc097cd0 2023-09-30T01:33:06.288 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.238+0000 7f54c3fff700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:33:06.288 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.238+0000 7f54c3fff700 1 -- 172.21.15.88:0/1794351274 <== mon.2 v2:172.21.15.88:3301/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f54d400ce10 con 0x7f54dc097cd0 2023-09-30T01:33:06.288 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.238+0000 7f54c3fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:33:06.288 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.238+0000 7f54c3fff700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T01:33:06.288 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.238+0000 7f54d9ffb700 4 set_mon_vals no callback set 2023-09-30T01:33:06.288 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.238+0000 7f54c3fff700 10 monclient: dump: 2023-09-30T01:33:06.289 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:33:06.289 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:33:06.289 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:06.289 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:06.289 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:33:06.289 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:33:06.289 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:33:06.289 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:33:06.289 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:33:06.289 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:33:06.289 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:33:06.289 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:33:06.290 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:33:06.290 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:33:06.290 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:33:06.290 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:33:06.290 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.238+0000 7f54e16e1700 5 monclient: authenticate success, global_id 4322 2023-09-30T01:33:06.290 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.238+0000 7f54e16e1700 10 monclient: _renew_subs 2023-09-30T01:33:06.290 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.238+0000 7f54e16e1700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.88:3301/0 2023-09-30T01:33:06.290 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.238+0000 7f54e16e1700 1 -- 172.21.15.88:0/1794351274 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f54dc063630 con 0x7f54dc097cd0 2023-09-30T01:33:06.290 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.238+0000 7f54e16e1700 10 monclient: _renew_subs 2023-09-30T01:33:06.290 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.238+0000 7f54e16e1700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.88:3301/0 2023-09-30T01:33:06.290 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.238+0000 7f54e16e1700 1 -- 172.21.15.88:0/1794351274 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f54dc08f8a0 con 0x7f54dc097cd0 2023-09-30T01:33:06.290 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.238+0000 7f54c3fff700 1 -- 172.21.15.88:0/1794351274 <== mon.2 v2:172.21.15.88:3301/0 4 ==== mgrmap(e 5) v1 ==== 40144+0+0 (secure 0 0 0) 0x7f54d4003020 con 0x7f54dc097cd0 2023-09-30T01:33:06.291 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.238+0000 7f54c3fff700 1 --2- 172.21.15.88:0/1794351274 >> v2:172.21.15.107:6812/29926 conn(0x7f54c8033360 0x7f54c8035810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:33:06.291 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.238+0000 7f54c3fff700 1 -- 172.21.15.88:0/1794351274 <== mon.2 v2:172.21.15.88:3301/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 3502+0+0 (secure 0 0 0) 0x7f54d4043c10 con 0x7f54dc097cd0 2023-09-30T01:33:06.291 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.238+0000 7f54dbfff700 1 --2- 172.21.15.88:0/1794351274 >> v2:172.21.15.107:6812/29926 conn(0x7f54c8033360 0x7f54c8035810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:33:06.291 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.238+0000 7f54dbfff700 10 monclient: get_auth_request con 0x7f54c8033360 auth_method 0 2023-09-30T01:33:06.291 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.242+0000 7f0bd1292700 1 Processor -- start 2023-09-30T01:33:06.291 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.242+0000 7f0bd1292700 1 -- start start 2023-09-30T01:33:06.291 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.242+0000 7f0bd1292700 10 monclient: init 2023-09-30T01:33:06.292 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.242+0000 7f54dbfff700 1 --2- 172.21.15.88:0/1794351274 >> v2:172.21.15.107:6812/29926 conn(0x7f54c8033360 0x7f54c8035810 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f54d0000c00 tx=0x7f54d0016000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:33:06.294 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.242+0000 7f0bd1292700 10 monclient: _reopen_session rank -1 2023-09-30T01:33:06.294 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.242+0000 7f0bd1292700 10 monclient: _add_conns ranks=[2,5,6,10,3,9,7,4,8,1,0] 2023-09-30T01:33:06.294 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.242+0000 7f0bd1292700 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0bcc09ceb0 0x7f0bcc09d280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:33:06.294 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.242+0000 7f0bd1292700 10 monclient(hunting): picked mon.noname-b con 0x7f0bcc09ceb0 addr [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] 2023-09-30T01:33:06.294 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.242+0000 7f0bd1292700 1 --2- >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7f0bcc09e0e0 0x7f0bcc09d7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:33:06.294 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.242+0000 7f0bd1292700 10 monclient(hunting): picked mon.noname-h con 0x7f0bcc09e0e0 addr [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] 2023-09-30T01:33:06.294 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.242+0000 7f0bd1292700 1 --2- >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7f0bcc1316a0 0x7f0bcc131af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:33:06.294 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.242+0000 7f0bd1292700 10 monclient(hunting): picked mon.noname-d con 0x7f0bcc1316a0 addr [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] 2023-09-30T01:33:06.295 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.242+0000 7f0bd1292700 10 monclient(hunting): start opening mon connection 2023-09-30T01:33:06.295 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.242+0000 7f0bd1292700 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0bcc09c940 con 0x7f0bcc09ceb0 2023-09-30T01:33:06.295 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.242+0000 7f0bd1292700 10 monclient(hunting): start opening mon connection 2023-09-30T01:33:06.295 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.242+0000 7f0bd1292700 1 -- --> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] -- mon_getmap magic: 0 v1 -- 0x7f0bcc09ee00 con 0x7f0bcc09e0e0 2023-09-30T01:33:06.295 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.242+0000 7f0bd1292700 10 monclient(hunting): start opening mon connection 2023-09-30T01:33:06.295 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.242+0000 7f0bd1292700 1 -- --> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] -- mon_getmap magic: 0 v1 -- 0x7f0bcc09dd00 con 0x7f0bcc1316a0 2023-09-30T01:33:06.295 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.242+0000 7f0bd1292700 10 monclient(hunting): _renew_subs 2023-09-30T01:33:06.295 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.242+0000 7f0bd1292700 10 monclient(hunting): authenticate will time out at 858.280939s 2023-09-30T01:33:06.295 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.242+0000 7f0bcb7fe700 1 --2- >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7f0bcc1316a0 0x7f0bcc131af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:33:06.295 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.242+0000 7f0bcaffd700 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0bcc09ceb0 0x7f0bcc09d280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:33:06.295 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.242+0000 7f0bcb7fe700 1 --2- >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7f0bcc1316a0 0x7f0bcc131af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3303/0 says I am v2:172.21.15.88:53606/0 (socket says 172.21.15.88:53606) 2023-09-30T01:33:06.296 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.242+0000 7f0bcb7fe700 1 -- 172.21.15.88:0/3030362939 learned_addr learned my addr 172.21.15.88:0/3030362939 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-09-30T01:33:06.296 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.242+0000 7f0bcb7fe700 10 monclient(hunting): get_auth_request con 0x7f0bcc1316a0 auth_method 0 2023-09-30T01:33:06.296 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.242+0000 7f0bcb7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:33:06.296 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.242+0000 7f0bcb7fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:33:06.296 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.242+0000 7f0bcb7fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:33:06.296 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.242+0000 7f0bcaffd700 10 monclient(hunting): get_auth_request con 0x7f0bcc09ceb0 auth_method 0 2023-09-30T01:33:06.296 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.242+0000 7f0bcaffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:33:06.296 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.242+0000 7f0bcaffd700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:33:06.296 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.242+0000 7f0bcaffd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:33:06.296 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.242+0000 7f0bcb7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:33:06.296 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.242+0000 7f0bcb7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:33:06.296 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.242+0000 7f0bcb7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:33:06.297 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.242+0000 7f0bcaffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:33:06.297 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.242+0000 7f0bcaffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:33:06.297 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.242+0000 7f0bcaffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:33:06.297 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.242+0000 7f0bcb7fe700 10 monclient(hunting): handle_auth_done global_id 4290 payload 274 2023-09-30T01:33:06.297 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.242+0000 7f0bcb7fe700 1 -- 172.21.15.88:0/3030362939 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7f0bcc09e0e0 msgr2=0x7f0bcc09d7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:33:06.297 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.242+0000 7f0bcb7fe700 1 --2- 172.21.15.88:0/3030362939 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7f0bcc09e0e0 0x7f0bcc09d7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:06.297 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.242+0000 7f0bcb7fe700 1 -- 172.21.15.88:0/3030362939 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0bcc09ceb0 msgr2=0x7f0bcc09d280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:33:06.297 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.242+0000 7f0bcb7fe700 1 --2- 172.21.15.88:0/3030362939 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0bcc09ceb0 0x7f0bcc09d280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:06.297 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.242+0000 7f0bcb7fe700 10 monclient: _finish_hunting 0 2023-09-30T01:33:06.297 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.242+0000 7f0bcb7fe700 1 monclient: found mon.noname-d 2023-09-30T01:33:06.297 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.242+0000 7f0bcb7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:33:06.297 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.242+0000 7f0bcb7fe700 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.88:3303/0 2023-09-30T01:33:06.298 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.242+0000 7f0bcb7fe700 1 -- 172.21.15.88:0/3030362939 --> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0bcc05ecb0 con 0x7f0bcc1316a0 2023-09-30T01:33:06.298 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.242+0000 7f0bcaffd700 1 --2- 172.21.15.88:0/3030362939 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0bcc09ceb0 0x7f0bcc09d280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:33:06.298 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.242+0000 7f0bcb7fe700 1 --2- 172.21.15.88:0/3030362939 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7f0bcc1316a0 0x7f0bcc131af0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f0bbc0099e0 tx=0x7f0bbc009250 comp rx=0 tx=0).ready entity=mon.6 client_cookie=b05873bf3ccef449 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:33:06.298 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.242+0000 7f0bc9ffb700 1 -- 172.21.15.88:0/3030362939 <== mon.6 v2:172.21.15.88:3303/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f0bbc00bce0 con 0x7f0bcc1316a0 2023-09-30T01:33:06.298 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.242+0000 7f0bc9ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:33:06.298 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.242+0000 7f0bc9ffb700 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2023-09-30T01:33:06.298 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.242+0000 7f0bc9ffb700 10 monclient: dump: 2023-09-30T01:33:06.298 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:33:06.298 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:33:06.298 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:06.298 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:06.298 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:33:06.299 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:33:06.299 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:33:06.299 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:33:06.299 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:33:06.299 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:33:06.299 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:33:06.299 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:33:06.299 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:33:06.299 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:33:06.299 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:33:06.299 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:33:06.299 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.242+0000 7f0bc9ffb700 10 monclient: _finish_auth 0 2023-09-30T01:33:06.300 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.242+0000 7f0bc9ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:33:06.300 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.242+0000 7f0bc9ffb700 1 -- 172.21.15.88:0/3030362939 <== mon.6 v2:172.21.15.88:3303/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0bbc00be60 con 0x7f0bcc1316a0 2023-09-30T01:33:06.300 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.242+0000 7f0bd1292700 5 monclient: authenticate success, global_id 4290 2023-09-30T01:33:06.300 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.242+0000 7f0bd1292700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:33:06.300 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.242+0000 7f0bc9ffb700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:33:06.300 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.242+0000 7f0bd1292700 10 monclient: get_monmap_and_config success 2023-09-30T01:33:06.300 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.242+0000 7f0bd1292700 4 set_mon_vals no callback set 2023-09-30T01:33:06.300 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.242+0000 7f0bc9ffb700 1 -- 172.21.15.88:0/3030362939 <== mon.6 v2:172.21.15.88:3303/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f0bbc0109e0 con 0x7f0bcc1316a0 2023-09-30T01:33:06.300 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.242+0000 7f0bc9ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:33:06.300 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.242+0000 7f0bc9ffb700 10 monclient: got monmap 1 from mon.d (according to old e1) 2023-09-30T01:33:06.300 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.242+0000 7f0bc9ffb700 10 monclient: dump: 2023-09-30T01:33:06.300 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:33:06.300 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:33:06.301 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:06.301 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:06.301 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:33:06.301 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:33:06.301 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:33:06.301 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:33:06.301 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:33:06.301 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:33:06.301 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:33:06.301 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:33:06.301 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:33:06.301 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:33:06.302 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:33:06.302 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:33:06.302 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.242+0000 7fb571ffb700 1 -- 172.21.15.88:0/1762600973 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fb57c1329f0 con 0x7fb57c09f9e0 2023-09-30T01:33:06.302 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.242+0000 7fb571ffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:33:06.302 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.242+0000 7fb571ffb700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:33:06.303 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.242+0000 7f0bd1292700 10 monclient: shutdown 2023-09-30T01:33:06.303 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.242+0000 7f0bd1292700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:33:06.303 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.242+0000 7f0bd1292700 1 -- 172.21.15.88:0/3030362939 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7f0bcc1316a0 msgr2=0x7f0bcc131af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:33:06.303 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.242+0000 7f0bd1292700 1 --2- 172.21.15.88:0/3030362939 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7f0bcc1316a0 0x7f0bcc131af0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f0bbc0099e0 tx=0x7f0bbc009250 comp rx=0 tx=0).stop 2023-09-30T01:33:06.303 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.246+0000 7f0bd1292700 1 -- 172.21.15.88:0/3030362939 shutdown_connections 2023-09-30T01:33:06.303 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.246+0000 7f0bd1292700 1 --2- 172.21.15.88:0/3030362939 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7f0bcc1316a0 0x7f0bcc131af0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:06.303 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.246+0000 7f0bd1292700 1 --2- 172.21.15.88:0/3030362939 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7f0bcc09e0e0 0x7f0bcc09d7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:06.303 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.246+0000 7f0bd1292700 1 --2- 172.21.15.88:0/3030362939 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0bcc09ceb0 0x7f0bcc09d280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:06.303 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.246+0000 7f0bd1292700 1 -- 172.21.15.88:0/3030362939 >> 172.21.15.88:0/3030362939 conn(0x7f0bcc0a24c0 msgr2=0x7f0bcc0a2890 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:33:06.304 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.246+0000 7f54c1ffb700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:33:06.304 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.246+0000 7f54c1ffb700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:33:06.304 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.246+0000 7f54c1ffb700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.88:3301/0 2023-09-30T01:33:06.304 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.246+0000 7f54c1ffb700 1 -- 172.21.15.88:0/1794351274 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f54dc067520 con 0x7f54dc097cd0 2023-09-30T01:33:06.306 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.246+0000 7f0bd1292700 1 -- 172.21.15.88:0/3030362939 shutdown_connections 2023-09-30T01:33:06.306 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.246+0000 7f0bd1292700 1 -- 172.21.15.88:0/3030362939 wait complete. 2023-09-30T01:33:06.306 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.246+0000 7f0bd1292700 10 monclient: build_initial_monmap 2023-09-30T01:33:06.306 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.246+0000 7f0bd1292700 10 monclient: monmap: 2023-09-30T01:33:06.306 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2023-09-30T01:33:06.306 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:33:06.306 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 0.000000 2023-09-30T01:33:06.306 INFO:teuthology.orchestra.run.smithi088.stderr:created 0.000000 2023-09-30T01:33:06.306 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2023-09-30T01:33:06.307 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 1 2023-09-30T01:33:06.307 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.noname-a 2023-09-30T01:33:06.307 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.noname-f 2023-09-30T01:33:06.307 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.noname-b 2023-09-30T01:33:06.307 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.noname-g 2023-09-30T01:33:06.307 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.noname-c 2023-09-30T01:33:06.307 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.noname-h 2023-09-30T01:33:06.307 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.noname-d 2023-09-30T01:33:06.307 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.noname-i 2023-09-30T01:33:06.307 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.noname-e 2023-09-30T01:33:06.307 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:33:06.307 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.246+0000 7f0bd1292700 1 Processor -- start 2023-09-30T01:33:06.307 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.246+0000 7f0bd1292700 1 -- start start 2023-09-30T01:33:06.307 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.246+0000 7f0bd1292700 10 monclient: init 2023-09-30T01:33:06.308 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.246+0000 7f0bd1292700 10 monclient: _reopen_session rank -1 2023-09-30T01:33:06.308 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.246+0000 7f0bd1292700 10 monclient: _add_conns ranks=[8,0,1,6,3,4,5,7,2] 2023-09-30T01:33:06.308 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.246+0000 7f0bd1292700 1 --2- >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f0bcc09ceb0 0x7f0bcc13e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:33:06.308 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.246+0000 7f0bd1292700 10 monclient(hunting): picked mon.noname-e con 0x7f0bcc09ceb0 addr [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] 2023-09-30T01:33:06.308 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.246+0000 7f0bd1292700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0bcc09e0e0 0x7f0bcc13ecb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:33:06.308 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.246+0000 7f0bd1292700 10 monclient(hunting): picked mon.noname-a con 0x7f0bcc09e0e0 addr [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] 2023-09-30T01:33:06.308 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.246+0000 7f0bd1292700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0bcc1316a0 0x7f0bcc13f1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:33:06.308 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.246+0000 7f0bd1292700 10 monclient(hunting): picked mon.noname-f con 0x7f0bcc1316a0 addr [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] 2023-09-30T01:33:06.308 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.246+0000 7f0bd1292700 10 monclient(hunting): start opening mon connection 2023-09-30T01:33:06.308 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.246+0000 7f0bd1292700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bcc12c8c0 con 0x7f0bcc09e0e0 2023-09-30T01:33:06.308 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.246+0000 7f0bd1292700 10 monclient(hunting): start opening mon connection 2023-09-30T01:33:06.308 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.246+0000 7f0bd1292700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bcc129a50 con 0x7f0bcc1316a0 2023-09-30T01:33:06.309 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.246+0000 7f0bd1292700 10 monclient(hunting): start opening mon connection 2023-09-30T01:33:06.309 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.246+0000 7f0bd1292700 1 -- --> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] -- mon_getmap magic: 0 v1 -- 0x7f0bcc09e7c0 con 0x7f0bcc09ceb0 2023-09-30T01:33:06.309 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.246+0000 7f0bd1292700 10 monclient(hunting): _renew_subs 2023-09-30T01:33:06.309 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.246+0000 7f0bd1292700 10 monclient(hunting): authenticate will time out at 858.286033s 2023-09-30T01:33:06.309 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.246+0000 7f0bcaffd700 1 --2- >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f0bcc09ceb0 0x7f0bcc13e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:33:06.309 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.246+0000 7f0bcaffd700 1 --2- >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f0bcc09ceb0 0x7f0bcc13e770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3304/0 says I am v2:172.21.15.88:36796/0 (socket says 172.21.15.88:36796) 2023-09-30T01:33:06.309 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.246+0000 7f0bcaffd700 1 -- 172.21.15.88:0/671106497 learned_addr learned my addr 172.21.15.88:0/671106497 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-09-30T01:33:06.309 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.246+0000 7f0bcaffd700 10 monclient(hunting): get_auth_request con 0x7f0bcc09ceb0 auth_method 0 2023-09-30T01:33:06.309 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.246+0000 7f0bcaffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:33:06.309 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.246+0000 7f0bcaffd700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:33:06.309 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.246+0000 7f0bcaffd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:33:06.309 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.246+0000 7f0bcaffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:33:06.310 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.246+0000 7f0bcaffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:33:06.310 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.246+0000 7f0bcaffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:33:06.310 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.246+0000 7f0bcb7fe700 1 --2- 172.21.15.88:0/671106497 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0bcc1316a0 0x7f0bcc13f1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:33:06.310 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.246+0000 7f0bcb7fe700 10 monclient(hunting): get_auth_request con 0x7f0bcc1316a0 auth_method 0 2023-09-30T01:33:06.310 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.246+0000 7f0bcb7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:33:06.310 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.246+0000 7f0bcb7fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:33:06.310 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.246+0000 7f0bcb7fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:33:06.310 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.246+0000 7f0bcaffd700 10 monclient(hunting): handle_auth_done global_id 4382 payload 824 2023-09-30T01:33:06.310 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.246+0000 7f0bcaffd700 1 -- 172.21.15.88:0/671106497 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0bcc1316a0 msgr2=0x7f0bcc13f1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:33:06.310 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.246+0000 7f0bcaffd700 1 --2- 172.21.15.88:0/671106497 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0bcc1316a0 0x7f0bcc13f1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:06.310 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.246+0000 7f0bcaffd700 1 -- 172.21.15.88:0/671106497 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0bcc09e0e0 msgr2=0x7f0bcc13ecb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:33:06.310 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.246+0000 7f0bcaffd700 1 --2- 172.21.15.88:0/671106497 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0bcc09e0e0 0x7f0bcc13ecb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:06.311 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.246+0000 7f0bcaffd700 10 monclient: _finish_hunting 0 2023-09-30T01:33:06.311 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.246+0000 7f0bcaffd700 1 monclient: found mon.noname-e 2023-09-30T01:33:06.311 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.246+0000 7f0bcaffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:33:06.311 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.246+0000 7f0bcaffd700 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.88:3304/0 2023-09-30T01:33:06.311 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.246+0000 7f0bcaffd700 1 -- 172.21.15.88:0/671106497 --> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0bc0009420 con 0x7f0bcc09ceb0 2023-09-30T01:33:06.311 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.246+0000 7f0bcb7fe700 1 --2- 172.21.15.88:0/671106497 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0bcc1316a0 0x7f0bcc13f1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-30T01:33:06.311 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.246+0000 7f0bcaffd700 1 --2- 172.21.15.88:0/671106497 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f0bcc09ceb0 0x7f0bcc13e770 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f0bc000ecb0 tx=0x7f0bc00091a0 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:33:06.311 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.254+0000 7f0bbaffd700 1 -- 172.21.15.88:0/671106497 <== mon.8 v2:172.21.15.88:3304/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f0bc00097d0 con 0x7f0bcc09ceb0 2023-09-30T01:33:06.311 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.254+0000 7f0bbaffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:33:06.311 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.254+0000 7f0bbaffd700 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2023-09-30T01:33:06.311 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.254+0000 7f0bbaffd700 10 monclient: dump: 2023-09-30T01:33:06.312 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:33:06.312 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:33:06.312 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:06.312 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:06.312 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:33:06.312 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:33:06.312 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:33:06.312 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:33:06.312 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:33:06.312 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:33:06.312 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:33:06.312 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:33:06.312 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:33:06.313 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:33:06.313 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:33:06.313 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:33:06.313 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.254+0000 7f0bbaffd700 10 monclient: _finish_auth 0 2023-09-30T01:33:06.313 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.254+0000 7f0bbaffd700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:33:06.313 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.254+0000 7f0bbaffd700 1 -- 172.21.15.88:0/671106497 <== mon.8 v2:172.21.15.88:3304/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0bc0013470 con 0x7f0bcc09ceb0 2023-09-30T01:33:06.313 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.254+0000 7f0bbaffd700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:33:06.313 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.254+0000 7f0bbaffd700 1 -- 172.21.15.88:0/671106497 <== mon.8 v2:172.21.15.88:3304/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f0bc0013a60 con 0x7f0bcc09ceb0 2023-09-30T01:33:06.313 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.254+0000 7f0bbaffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:33:06.313 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.254+0000 7f0bbaffd700 10 monclient: got monmap 1 from mon.e (according to old e1) 2023-09-30T01:33:06.313 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.254+0000 7f0bbaffd700 10 monclient: dump: 2023-09-30T01:33:06.313 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:33:06.314 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:33:06.314 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:06.314 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:06.314 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:33:06.314 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:33:06.314 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:33:06.314 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:33:06.314 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:33:06.314 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:33:06.314 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:33:06.314 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:33:06.314 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:33:06.314 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:33:06.315 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:33:06.315 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:33:06.315 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.254+0000 7f54c3fff700 1 -- 172.21.15.88:0/1794351274 <== mon.2 v2:172.21.15.88:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f54d4012af0 con 0x7f54dc097cd0 2023-09-30T01:33:06.315 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.254+0000 7f54c3fff700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:33:06.315 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.254+0000 7f54c3fff700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:33:06.316 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.254+0000 7f0bd1292700 5 monclient: authenticate success, global_id 4382 2023-09-30T01:33:06.316 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.254+0000 7f0bd1292700 10 monclient: _renew_subs 2023-09-30T01:33:06.316 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.254+0000 7f0bd1292700 10 monclient: _send_mon_message to mon.e at v2:172.21.15.88:3304/0 2023-09-30T01:33:06.316 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.254+0000 7f0bd1292700 1 -- 172.21.15.88:0/671106497 --> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0bbc0094f0 con 0x7f0bcc09ceb0 2023-09-30T01:33:06.317 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.254+0000 7f0bd1292700 10 monclient: _renew_subs 2023-09-30T01:33:06.317 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.254+0000 7f0bd1292700 10 monclient: _send_mon_message to mon.e at v2:172.21.15.88:3304/0 2023-09-30T01:33:06.317 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.254+0000 7f0bc97fa700 4 set_mon_vals no callback set 2023-09-30T01:33:06.317 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.254+0000 7f0bd1292700 1 -- 172.21.15.88:0/671106497 --> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0bcc12a210 con 0x7f0bcc09ceb0 2023-09-30T01:33:06.317 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.258+0000 7f0bbaffd700 1 -- 172.21.15.88:0/671106497 <== mon.8 v2:172.21.15.88:3304/0 4 ==== mgrmap(e 5) v1 ==== 40144+0+0 (secure 0 0 0) 0x7f0bc0013bc0 con 0x7f0bcc09ceb0 2023-09-30T01:33:06.317 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.258+0000 7f0bbaffd700 1 --2- 172.21.15.88:0/671106497 >> v2:172.21.15.107:6812/29926 conn(0x7f0bb40338a0 0x7f0bb4035d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:33:06.317 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.258+0000 7f0bbaffd700 1 -- 172.21.15.88:0/671106497 <== mon.8 v2:172.21.15.88:3304/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 3502+0+0 (secure 0 0 0) 0x7f0bc00439d0 con 0x7f0bcc09ceb0 2023-09-30T01:33:06.317 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.258+0000 7f0bca7fc700 1 --2- 172.21.15.88:0/671106497 >> v2:172.21.15.107:6812/29926 conn(0x7f0bb40338a0 0x7f0bb4035d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:33:06.317 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.258+0000 7f0bca7fc700 10 monclient: get_auth_request con 0x7f0bb40338a0 auth_method 0 2023-09-30T01:33:06.317 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.262+0000 7f0bd1292700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:33:06.317 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.262+0000 7f0bd1292700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:33:06.317 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.262+0000 7f0bd1292700 10 monclient: _send_mon_message to mon.e at v2:172.21.15.88:3304/0 2023-09-30T01:33:06.318 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.262+0000 7f0bca7fc700 1 --2- 172.21.15.88:0/671106497 >> v2:172.21.15.107:6812/29926 conn(0x7f0bb40338a0 0x7f0bb4035d50 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f0bc400acf0 tx=0x7f0bc4009370 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:33:06.318 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.262+0000 7f0bd1292700 1 -- 172.21.15.88:0/671106497 --> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ba80050b0 con 0x7f0bcc09ceb0 2023-09-30T01:33:06.318 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.266+0000 7f0bbaffd700 1 -- 172.21.15.88:0/671106497 <== mon.8 v2:172.21.15.88:3304/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f0bc000cb80 con 0x7f0bcc09ceb0 2023-09-30T01:33:06.318 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.266+0000 7f0bbaffd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:33:06.318 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.266+0000 7f0bbaffd700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:33:06.318 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.294+0000 7f4ea88e6700 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 5}] 2023-09-30T01:33:06.319 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.294+0000 7f4ea88e6700 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2023-09-30T01:33:06.319 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.294+0000 7f4ea88e6700 10 monclient: _send_mon_message to mon.i at v2:172.21.15.107:3303/0 2023-09-30T01:33:06.319 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.294+0000 7f4ea88e6700 1 -- 172.21.15.88:0/3240982109 --> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7f4ea409cfd0 con 0x7f4ea409d950 2023-09-30T01:33:06.319 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.294+0000 7f4e9b7fe700 1 -- 172.21.15.88:0/3240982109 <== mon.7 v2:172.21.15.107:3303/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f4e9401bba0 con 0x7f4ea409d950 2023-09-30T01:33:06.319 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.294+0000 7f4e9b7fe700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2023-09-30T01:33:06.319 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.294+0000 7f4e9b7fe700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:33:06.319 INFO:teuthology.orchestra.run.smithi088.stdout:47244640257 2023-09-30T01:33:06.320 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.298+0000 7f4ea88e6700 1 -- 172.21.15.88:0/3240982109 >> v2:172.21.15.107:6812/29926 conn(0x7f4e84033360 msgr2=0x7f4e84035810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:33:06.321 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.298+0000 7f4ea88e6700 1 --2- 172.21.15.88:0/3240982109 >> v2:172.21.15.107:6812/29926 conn(0x7f4e84033360 0x7f4e84035810 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f4e8c0072d0 tx=0x7f4e8c00b040 comp rx=0 tx=0).stop 2023-09-30T01:33:06.321 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.298+0000 7f4ea88e6700 10 monclient: shutdown 2023-09-30T01:33:06.321 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.298+0000 7f4ea88e6700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:33:06.321 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.298+0000 7f4ea88e6700 1 -- 172.21.15.88:0/3240982109 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7f4ea409d950 msgr2=0x7f4ea411a700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:33:06.321 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.298+0000 7f4ea88e6700 1 --2- 172.21.15.88:0/3240982109 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7f4ea409d950 0x7f4ea411a700 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f4e9400ec80 tx=0x7f4e940091a0 comp rx=0 tx=0).stop 2023-09-30T01:33:06.321 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.298+0000 7f4ea88e6700 1 -- 172.21.15.88:0/3240982109 shutdown_connections 2023-09-30T01:33:06.321 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.298+0000 7f4ea88e6700 1 --2- 172.21.15.88:0/3240982109 >> v2:172.21.15.107:6812/29926 conn(0x7f4e84033360 0x7f4e84035810 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:06.321 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.298+0000 7f4ea88e6700 1 --2- 172.21.15.88:0/3240982109 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7f4ea4131370 0x7f4ea411ac40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:06.321 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.298+0000 7f4ea88e6700 1 --2- 172.21.15.88:0/3240982109 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7f4ea409d950 0x7f4ea411a700 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:06.321 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.298+0000 7f4ea88e6700 1 --2- 172.21.15.88:0/3240982109 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7f4ea409d580 0x7f4ea411a1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:06.321 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.298+0000 7f4ea88e6700 1 -- 172.21.15.88:0/3240982109 >> 172.21.15.88:0/3240982109 conn(0x7f4ea4063610 msgr2=0x7f4ea4097a40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:33:06.322 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.298+0000 7f4ea88e6700 1 -- 172.21.15.88:0/3240982109 shutdown_connections 2023-09-30T01:33:06.322 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.298+0000 7f4ea88e6700 1 -- 172.21.15.88:0/3240982109 wait complete. 2023-09-30T01:33:06.322 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640258 got 47244640257 for osd.5 2023-09-30T01:33:06.401 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.398+0000 7f4e3e77d700 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 0}] 2023-09-30T01:33:06.402 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.398+0000 7f4e3e77d700 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2023-09-30T01:33:06.402 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.398+0000 7f4e3e77d700 10 monclient: _send_mon_message to mon.e at v2:172.21.15.88:3304/0 2023-09-30T01:33:06.402 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.398+0000 7f4e3e77d700 1 -- 172.21.15.88:0/405194235 --> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7f4e380a13c0 con 0x7f4e381357b0 2023-09-30T01:33:06.402 INFO:teuthology.orchestra.run.smithi088.stdout:47244640260 2023-09-30T01:33:06.402 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.398+0000 7f4e2d7fa700 1 -- 172.21.15.88:0/405194235 <== mon.8 v2:172.21.15.88:3304/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f4e380a13c0 con 0x7f4e381357b0 2023-09-30T01:33:06.402 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.398+0000 7f4e2d7fa700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2023-09-30T01:33:06.403 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.398+0000 7f4e2d7fa700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:33:06.404 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.402+0000 7f4e3e77d700 1 -- 172.21.15.88:0/405194235 >> v2:172.21.15.107:6812/29926 conn(0x7f4e24033c20 msgr2=0x7f4e240360d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:33:06.404 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.402+0000 7f4e3e77d700 1 --2- 172.21.15.88:0/405194235 >> v2:172.21.15.107:6812/29926 conn(0x7f4e24033c20 0x7f4e240360d0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f4e30000f80 tx=0x7f4e30009040 comp rx=0 tx=0).stop 2023-09-30T01:33:06.405 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.402+0000 7f4e3e77d700 10 monclient: shutdown 2023-09-30T01:33:06.405 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.402+0000 7f4e3e77d700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:33:06.405 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.402+0000 7f4e3e77d700 1 -- 172.21.15.88:0/405194235 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f4e381357b0 msgr2=0x7f4e3806d860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:33:06.405 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.402+0000 7f4e3e77d700 1 --2- 172.21.15.88:0/405194235 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f4e381357b0 0x7f4e3806d860 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f4e340091e0 tx=0x7f4e34017000 comp rx=0 tx=0).stop 2023-09-30T01:33:06.405 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.402+0000 7f4e3e77d700 1 -- 172.21.15.88:0/405194235 shutdown_connections 2023-09-30T01:33:06.405 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.402+0000 7f4e3e77d700 1 --2- 172.21.15.88:0/405194235 >> v2:172.21.15.107:6812/29926 conn(0x7f4e24033c20 0x7f4e240360d0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:06.405 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.402+0000 7f4e3e77d700 1 --2- 172.21.15.88:0/405194235 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f4e381357b0 0x7f4e3806d860 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:06.405 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.402+0000 7f4e3e77d700 1 --2- 172.21.15.88:0/405194235 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7f4e38098de0 0x7f4e38093380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:06.405 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.402+0000 7f4e3e77d700 1 --2- 172.21.15.88:0/405194235 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7f4e380976f0 0x7f4e3813dde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:06.406 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.402+0000 7f4e3e77d700 1 -- 172.21.15.88:0/405194235 >> 172.21.15.88:0/405194235 conn(0x7f4e380a0910 msgr2=0x7f4e3809ec00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:33:06.406 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.402+0000 7f4e3e77d700 1 -- 172.21.15.88:0/405194235 shutdown_connections 2023-09-30T01:33:06.406 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.402+0000 7f4e3e77d700 1 -- 172.21.15.88:0/405194235 wait complete. 2023-09-30T01:33:06.418 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640260 got 47244640260 for osd.0 2023-09-30T01:33:06.418 DEBUG:teuthology.parallel:result is None 2023-09-30T01:33:06.451 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.450+0000 7f0bd1292700 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 1}] 2023-09-30T01:33:06.451 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.450+0000 7f0bd1292700 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 1}] 2023-09-30T01:33:06.451 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.450+0000 7f0bd1292700 10 monclient: _send_mon_message to mon.e at v2:172.21.15.88:3304/0 2023-09-30T01:33:06.452 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.450+0000 7f0bd1292700 1 -- 172.21.15.88:0/671106497 --> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f0ba8005360 con 0x7f0bcc09ceb0 2023-09-30T01:33:06.452 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.450+0000 7f0bbaffd700 1 -- 172.21.15.88:0/671106497 <== mon.8 v2:172.21.15.88:3304/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f0bc00192c0 con 0x7f0bcc09ceb0 2023-09-30T01:33:06.452 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.450+0000 7f0bbaffd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 1}] 2023-09-30T01:33:06.452 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.450+0000 7f0bbaffd700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:33:06.452 INFO:teuthology.orchestra.run.smithi088.stdout:47244640259 2023-09-30T01:33:06.458 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.454+0000 7f0bd1292700 1 -- 172.21.15.88:0/671106497 >> v2:172.21.15.107:6812/29926 conn(0x7f0bb40338a0 msgr2=0x7f0bb4035d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:33:06.458 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.454+0000 7f0bd1292700 1 --2- 172.21.15.88:0/671106497 >> v2:172.21.15.107:6812/29926 conn(0x7f0bb40338a0 0x7f0bb4035d50 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f0bc400acf0 tx=0x7f0bc4009370 comp rx=0 tx=0).stop 2023-09-30T01:33:06.458 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.454+0000 7f0bd1292700 10 monclient: shutdown 2023-09-30T01:33:06.458 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.454+0000 7f0bd1292700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:33:06.458 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.454+0000 7f0bd1292700 1 -- 172.21.15.88:0/671106497 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f0bcc09ceb0 msgr2=0x7f0bcc13e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:33:06.458 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.454+0000 7f0bd1292700 1 --2- 172.21.15.88:0/671106497 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f0bcc09ceb0 0x7f0bcc13e770 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f0bc000ecb0 tx=0x7f0bc00091a0 comp rx=0 tx=0).stop 2023-09-30T01:33:06.458 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.454+0000 7f0bd1292700 1 -- 172.21.15.88:0/671106497 shutdown_connections 2023-09-30T01:33:06.458 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.454+0000 7f0bd1292700 1 --2- 172.21.15.88:0/671106497 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0bcc1316a0 0x7f0bcc13f1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:06.458 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.454+0000 7f0bd1292700 1 --2- 172.21.15.88:0/671106497 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0bcc09e0e0 0x7f0bcc13ecb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:06.459 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.454+0000 7f0bd1292700 1 --2- 172.21.15.88:0/671106497 >> v2:172.21.15.107:6812/29926 conn(0x7f0bb40338a0 0x7f0bb4035d50 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:06.459 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.454+0000 7f0bd1292700 1 --2- 172.21.15.88:0/671106497 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f0bcc09ceb0 0x7f0bcc13e770 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:06.459 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.454+0000 7f0bd1292700 1 -- 172.21.15.88:0/671106497 >> 172.21.15.88:0/671106497 conn(0x7f0bcc0a24c0 msgr2=0x7f0bcc0a0b50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:33:06.459 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.454+0000 7f0bd1292700 1 -- 172.21.15.88:0/671106497 shutdown_connections 2023-09-30T01:33:06.459 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.454+0000 7f0bd1292700 1 -- 172.21.15.88:0/671106497 wait complete. 2023-09-30T01:33:06.471 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640259 got 47244640259 for osd.1 2023-09-30T01:33:06.471 DEBUG:teuthology.parallel:result is None 2023-09-30T01:33:06.495 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.490+0000 7fc8c5fa1700 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 3}] 2023-09-30T01:33:06.495 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.490+0000 7fc8c5fa1700 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2023-09-30T01:33:06.495 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.490+0000 7fc8c5fa1700 10 monclient: _send_mon_message to mon.e at v2:172.21.15.88:3304/0 2023-09-30T01:33:06.495 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.490+0000 7fc8c5fa1700 1 -- 172.21.15.88:0/2513588189 --> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7fc8a0005630 con 0x7fc8c00990f0 2023-09-30T01:33:06.495 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.494+0000 7fc8bcff9700 1 -- 172.21.15.88:0/2513588189 <== mon.8 v2:172.21.15.88:3304/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fc8b001b300 con 0x7fc8c00990f0 2023-09-30T01:33:06.495 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.494+0000 7fc8bcff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2023-09-30T01:33:06.495 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.494+0000 7fc8bcff9700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:33:06.496 INFO:teuthology.orchestra.run.smithi088.stdout:47244640259 2023-09-30T01:33:06.498 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.494+0000 7fc8c5fa1700 1 -- 172.21.15.88:0/2513588189 >> v2:172.21.15.107:6812/29926 conn(0x7fc8a8033b60 msgr2=0x7fc8a8036010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:33:06.499 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.494+0000 7fc8c5fa1700 1 --2- 172.21.15.88:0/2513588189 >> v2:172.21.15.107:6812/29926 conn(0x7fc8a8033b60 0x7fc8a8036010 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fc8c0071da0 tx=0x7fc8b4008040 comp rx=0 tx=0).stop 2023-09-30T01:33:06.499 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.494+0000 7fc8c5fa1700 10 monclient: shutdown 2023-09-30T01:33:06.499 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.494+0000 7fc8c5fa1700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:33:06.499 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.494+0000 7fc8c5fa1700 1 -- 172.21.15.88:0/2513588189 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7fc8c00990f0 msgr2=0x7fc8c006cbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:33:06.499 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.494+0000 7fc8c5fa1700 1 --2- 172.21.15.88:0/2513588189 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7fc8c00990f0 0x7fc8c006cbd0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fc8b000c840 tx=0x7fc8b0001430 comp rx=0 tx=0).stop 2023-09-30T01:33:06.499 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.494+0000 7fc8c5fa1700 1 -- 172.21.15.88:0/2513588189 shutdown_connections 2023-09-30T01:33:06.499 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.494+0000 7fc8c5fa1700 1 --2- 172.21.15.88:0/2513588189 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc8c009cf30 0x7fc8c006d650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:06.499 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.494+0000 7fc8c5fa1700 1 --2- 172.21.15.88:0/2513588189 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7fc8c009c430 0x7fc8c006d110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:06.499 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.494+0000 7fc8c5fa1700 1 --2- 172.21.15.88:0/2513588189 >> v2:172.21.15.107:6812/29926 conn(0x7fc8a8033b60 0x7fc8a8036010 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:06.499 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.494+0000 7fc8c5fa1700 1 --2- 172.21.15.88:0/2513588189 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7fc8c00990f0 0x7fc8c006cbd0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:06.499 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.494+0000 7fc8c5fa1700 1 -- 172.21.15.88:0/2513588189 >> 172.21.15.88:0/2513588189 conn(0x7fc8c00a2540 msgr2=0x7fc8c009d330 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:33:06.500 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.494+0000 7fc8c5fa1700 1 -- 172.21.15.88:0/2513588189 shutdown_connections 2023-09-30T01:33:06.500 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.494+0000 7fc8c5fa1700 1 -- 172.21.15.88:0/2513588189 wait complete. 2023-09-30T01:33:06.500 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.498+0000 7f54c1ffb700 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 2}] 2023-09-30T01:33:06.500 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.498+0000 7f54c1ffb700 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2023-09-30T01:33:06.501 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.498+0000 7f54c1ffb700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.88:3301/0 2023-09-30T01:33:06.501 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.498+0000 7f54c1ffb700 1 -- 172.21.15.88:0/1794351274 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7f54dc0916e0 con 0x7f54dc097cd0 2023-09-30T01:33:06.501 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.498+0000 7f54c3fff700 1 -- 172.21.15.88:0/1794351274 <== mon.2 v2:172.21.15.88:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f54d4012af0 con 0x7f54dc097cd0 2023-09-30T01:33:06.501 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.498+0000 7f54c3fff700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2023-09-30T01:33:06.501 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.498+0000 7f54c3fff700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:33:06.501 INFO:teuthology.orchestra.run.smithi088.stdout:47244640260 2023-09-30T01:33:06.502 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.498+0000 7f54c1ffb700 1 -- 172.21.15.88:0/1794351274 >> v2:172.21.15.107:6812/29926 conn(0x7f54c8033360 msgr2=0x7f54c8035810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:33:06.502 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.498+0000 7f54c1ffb700 1 --2- 172.21.15.88:0/1794351274 >> v2:172.21.15.107:6812/29926 conn(0x7f54c8033360 0x7f54c8035810 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f54d0000c00 tx=0x7f54d0016000 comp rx=0 tx=0).stop 2023-09-30T01:33:06.502 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.498+0000 7f54c1ffb700 10 monclient: shutdown 2023-09-30T01:33:06.502 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.498+0000 7f54c1ffb700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:33:06.503 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.498+0000 7f54c1ffb700 1 -- 172.21.15.88:0/1794351274 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f54dc097cd0 msgr2=0x7f54dc142bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:33:06.503 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.498+0000 7f54c1ffb700 1 --2- 172.21.15.88:0/1794351274 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f54dc097cd0 0x7f54dc142bd0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f54d4009ef0 tx=0x7f54d400b040 comp rx=0 tx=0).stop 2023-09-30T01:33:06.503 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.498+0000 7f54c1ffb700 1 -- 172.21.15.88:0/1794351274 shutdown_connections 2023-09-30T01:33:06.503 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.498+0000 7f54c1ffb700 1 --2- 172.21.15.88:0/1794351274 >> v2:172.21.15.107:6812/29926 conn(0x7f54c8033360 0x7f54c8035810 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:06.503 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.498+0000 7f54c1ffb700 1 --2- 172.21.15.88:0/1794351274 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f54dc09ed80 0x7f54dc143110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:06.503 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.498+0000 7f54c1ffb700 1 --2- 172.21.15.88:0/1794351274 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f54dc097cd0 0x7f54dc142bd0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:06.503 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.498+0000 7f54c1ffb700 1 --2- 172.21.15.88:0/1794351274 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7f54dc0966a0 0x7f54dc142690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:06.503 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.498+0000 7f54c1ffb700 1 -- 172.21.15.88:0/1794351274 >> 172.21.15.88:0/1794351274 conn(0x7f54dc0a0910 msgr2=0x7f54dc09f1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:33:06.503 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.498+0000 7f54c1ffb700 1 -- 172.21.15.88:0/1794351274 shutdown_connections 2023-09-30T01:33:06.503 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.498+0000 7f54c1ffb700 1 -- 172.21.15.88:0/1794351274 wait complete. 2023-09-30T01:33:06.511 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.510+0000 7fb583bad700 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 4}] 2023-09-30T01:33:06.511 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.510+0000 7fb583bad700 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 4}] 2023-09-30T01:33:06.511 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.510+0000 7fb583bad700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.88:3300/0 2023-09-30T01:33:06.512 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.510+0000 7fb583bad700 1 -- 172.21.15.88:0/1762600973 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7fb57c1329f0 con 0x7fb57c09f9e0 2023-09-30T01:33:06.512 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.510+0000 7fb571ffb700 1 -- 172.21.15.88:0/1762600973 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fb57c1329f0 con 0x7fb57c09f9e0 2023-09-30T01:33:06.512 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.510+0000 7fb571ffb700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 4}] 2023-09-30T01:33:06.512 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.510+0000 7fb571ffb700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:33:06.513 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640259 got 47244640259 for osd.3 2023-09-30T01:33:06.513 INFO:teuthology.orchestra.run.smithi088.stdout:47244640258 2023-09-30T01:33:06.514 DEBUG:teuthology.parallel:result is None 2023-09-30T01:33:06.514 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640260 got 47244640260 for osd.2 2023-09-30T01:33:06.515 DEBUG:teuthology.parallel:result is None 2023-09-30T01:33:06.515 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.510+0000 7fb583bad700 1 -- 172.21.15.88:0/1762600973 >> v2:172.21.15.107:6812/29926 conn(0x7fb560033360 msgr2=0x7fb560035810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:33:06.515 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.510+0000 7fb583bad700 1 --2- 172.21.15.88:0/1762600973 >> v2:172.21.15.107:6812/29926 conn(0x7fb560033360 0x7fb560035810 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fb578009e90 tx=0x7fb578009220 comp rx=0 tx=0).stop 2023-09-30T01:33:06.515 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.510+0000 7fb583bad700 10 monclient: shutdown 2023-09-30T01:33:06.515 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.510+0000 7fb583bad700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:33:06.515 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.510+0000 7fb583bad700 1 -- 172.21.15.88:0/1762600973 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb57c09f9e0 msgr2=0x7fb57c06d4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:33:06.516 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.510+0000 7fb583bad700 1 --2- 172.21.15.88:0/1762600973 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb57c09f9e0 0x7fb57c06d4d0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fb574012000 tx=0x7fb574015000 comp rx=0 tx=0).stop 2023-09-30T01:33:06.516 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.510+0000 7fb583bad700 1 -- 172.21.15.88:0/1762600973 shutdown_connections 2023-09-30T01:33:06.516 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.510+0000 7fb583bad700 1 --2- 172.21.15.88:0/1762600973 >> v2:172.21.15.107:6812/29926 conn(0x7fb560033360 0x7fb560035810 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:06.516 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.510+0000 7fb583bad700 1 --2- 172.21.15.88:0/1762600973 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb57c09f9e0 0x7fb57c06d4d0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:06.516 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.510+0000 7fb583bad700 1 --2- 172.21.15.88:0/1762600973 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7fb57c09a460 0x7fb57c06cf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:06.516 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.510+0000 7fb583bad700 1 --2- 172.21.15.88:0/1762600973 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb57c062c70 0x7fb57c095bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:06.516 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.510+0000 7fb583bad700 1 -- 172.21.15.88:0/1762600973 >> 172.21.15.88:0/1762600973 conn(0x7fb57c0a2660 msgr2=0x7fb57c131b10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:33:06.516 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.510+0000 7fb583bad700 1 -- 172.21.15.88:0/1762600973 shutdown_connections 2023-09-30T01:33:06.516 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:06.510+0000 7fb583bad700 1 -- 172.21.15.88:0/1762600973 wait complete. 2023-09-30T01:33:06.524 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640258 got 47244640258 for osd.4 2023-09-30T01:33:06.524 DEBUG:teuthology.parallel:result is None 2023-09-30T01:33:07.323 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2023-09-30T01:33:07.439 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.438+0000 7f7fc93f4700 10 monclient: get_monmap_and_config 2023-09-30T01:33:07.439 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.438+0000 7f7fc93f4700 10 monclient: build_initial_monmap 2023-09-30T01:33:07.439 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.438+0000 7f7fc93f4700 10 monclient: monmap: 2023-09-30T01:33:07.439 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2023-09-30T01:33:07.440 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:33:07.440 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:33:07.440022+0000 2023-09-30T01:33:07.440 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:33:07.440022+0000 2023-09-30T01:33:07.440 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2023-09-30T01:33:07.440 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:33:07.440 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a-legacy 2023-09-30T01:33:07.440 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.107:6789/0 mon.noname-f-legacy 2023-09-30T01:33:07.440 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.noname-b 2023-09-30T01:33:07.440 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.noname-g 2023-09-30T01:33:07.440 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.noname-c 2023-09-30T01:33:07.440 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.noname-h 2023-09-30T01:33:07.440 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.noname-d 2023-09-30T01:33:07.441 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.noname-i 2023-09-30T01:33:07.441 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.noname-e 2023-09-30T01:33:07.441 INFO:teuthology.orchestra.run.smithi088.stderr:9: v2:172.21.15.88:3300/0 mon.noname-a 2023-09-30T01:33:07.441 INFO:teuthology.orchestra.run.smithi088.stderr:10: v2:172.21.15.107:3300/0 mon.noname-f 2023-09-30T01:33:07.441 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:33:07.444 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.438+0000 7f7fc93f4700 1 Processor -- start 2023-09-30T01:33:07.445 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.438+0000 7f7fc93f4700 1 -- start start 2023-09-30T01:33:07.445 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.438+0000 7f7fc93f4700 10 monclient: init 2023-09-30T01:33:07.445 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.438+0000 7f7fc93f4700 10 monclient: _reopen_session rank -1 2023-09-30T01:33:07.445 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.438+0000 7f7fc93f4700 10 monclient: _add_conns ranks=[0,5,8,10,4,3,1,9,2,6,7] 2023-09-30T01:33:07.445 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.438+0000 7f7fc93f4700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f7fc4097a40 addr v1:172.21.15.88:6789/0 2023-09-30T01:33:07.445 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.438+0000 7f7fc93f4700 1 --2- >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7f7fc4132e10 0x7f7fc4138650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:33:07.445 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.438+0000 7f7fc93f4700 10 monclient(hunting): picked mon.noname-h con 0x7f7fc4132e10 addr [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] 2023-09-30T01:33:07.445 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.438+0000 7f7fc93f4700 1 --2- >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f7fc41311c0 0x7f7fc413aba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:33:07.445 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.438+0000 7f7fc93f4700 10 monclient(hunting): picked mon.noname-e con 0x7f7fc41311c0 addr [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] 2023-09-30T01:33:07.445 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.438+0000 7f7fc93f4700 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7fc404bb20 con 0x7f7fc4097a40 2023-09-30T01:33:07.445 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.438+0000 7f7fc93f4700 10 monclient(hunting): start opening mon connection 2023-09-30T01:33:07.445 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.438+0000 7f7fc93f4700 1 -- --> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] -- mon_getmap magic: 0 v1 -- 0x7f7fc409a5d0 con 0x7f7fc4132e10 2023-09-30T01:33:07.446 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.438+0000 7f7fc93f4700 10 monclient(hunting): start opening mon connection 2023-09-30T01:33:07.446 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.438+0000 7f7fc93f4700 1 -- --> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] -- mon_getmap magic: 0 v1 -- 0x7f7fc4097e10 con 0x7f7fc41311c0 2023-09-30T01:33:07.446 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.438+0000 7f7fc93f4700 10 monclient(hunting): _renew_subs 2023-09-30T01:33:07.446 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.438+0000 7f7fc93f4700 10 monclient(hunting): authenticate will time out at 859.477517s 2023-09-30T01:33:07.446 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.438+0000 7f7fc3fff700 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f7fc4097a40 0x7f7fc4132700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:54140/0 (socket says 172.21.15.88:54140) 2023-09-30T01:33:07.446 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.438+0000 7f7fc8bf3700 1 --2- >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f7fc41311c0 0x7f7fc413aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:33:07.446 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.438+0000 7f7fc3fff700 1 -- 172.21.15.88:0/2997816105 learned_addr learned my addr 172.21.15.88:0/2997816105 (peer_addr_for_me v1:172.21.15.88:0/0) 2023-09-30T01:33:07.446 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.438+0000 7f7fc8bf3700 10 monclient(hunting): get_auth_request con 0x7f7fc41311c0 auth_method 0 2023-09-30T01:33:07.446 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.438+0000 7f7fc8bf3700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:33:07.446 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.438+0000 7f7fc8bf3700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:33:07.446 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.438+0000 7f7fc8bf3700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:33:07.447 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.438+0000 7f7fc37fe700 1 --2- 172.21.15.88:0/2997816105 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7f7fc4132e10 0x7f7fc4138650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:33:07.447 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.438+0000 7f7fc8bf3700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:33:07.447 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.438+0000 7f7fc8bf3700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:33:07.447 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.438+0000 7f7fc8bf3700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:33:07.447 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.438+0000 7f7fc37fe700 10 monclient(hunting): get_auth_request con 0x7f7fc4132e10 auth_method 0 2023-09-30T01:33:07.447 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.438+0000 7f7fc37fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:33:07.447 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.438+0000 7f7fc37fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:33:07.447 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.438+0000 7f7fc37fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:33:07.447 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.438+0000 7f7fc37fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:33:07.447 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.438+0000 7f7fc37fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:33:07.447 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.438+0000 7f7fc2ffd700 1 -- 172.21.15.88:0/2997816105 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 234058146 0 0) 0x7f7fc404bb20 con 0x7f7fc4097a40 2023-09-30T01:33:07.447 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.438+0000 7f7fc37fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:33:07.447 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.438+0000 7f7fc2ffd700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:33:07.448 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.438+0000 7f7fc2ffd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:33:07.448 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.438+0000 7f7fc2ffd700 10 monclient(hunting): my global_id is 4410 2023-09-30T01:33:07.448 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.438+0000 7f7fc2ffd700 1 -- 172.21.15.88:0/2997816105 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7fb0003570 con 0x7f7fc4097a40 2023-09-30T01:33:07.448 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.438+0000 7f7fc8bf3700 10 monclient(hunting): handle_auth_done global_id 4391 payload 274 2023-09-30T01:33:07.448 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.438+0000 7f7fc8bf3700 1 -- 172.21.15.88:0/2997816105 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7f7fc4132e10 msgr2=0x7f7fc4138650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:33:07.448 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.438+0000 7f7fc8bf3700 1 --2- 172.21.15.88:0/2997816105 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7f7fc4132e10 0x7f7fc4138650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:07.448 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.438+0000 7f7fc8bf3700 1 -- 172.21.15.88:0/2997816105 >> v1:172.21.15.88:6789/0 conn(0x7f7fc4097a40 legacy=0x7f7fc4132700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:33:07.448 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.438+0000 7f7fc8bf3700 10 monclient: _finish_hunting 0 2023-09-30T01:33:07.448 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.438+0000 7f7fc8bf3700 1 monclient: found mon.noname-e 2023-09-30T01:33:07.448 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.438+0000 7f7fc8bf3700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:33:07.448 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.438+0000 7f7fc8bf3700 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.88:3304/0 2023-09-30T01:33:07.448 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.438+0000 7f7fc8bf3700 1 -- 172.21.15.88:0/2997816105 --> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7fc4098d40 con 0x7f7fc41311c0 2023-09-30T01:33:07.449 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.438+0000 7f7fc8bf3700 1 --2- 172.21.15.88:0/2997816105 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f7fc41311c0 0x7f7fc413aba0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f7fb4009e90 tx=0x7f7fb4009780 comp rx=0 tx=0).ready entity=mon.8 client_cookie=3525af3da591da17 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:33:07.449 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.438+0000 7f7fc2ffd700 1 -- 172.21.15.88:0/2997816105 <== mon.8 v2:172.21.15.88:3304/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f7fb400e070 con 0x7f7fc41311c0 2023-09-30T01:33:07.449 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.438+0000 7f7fc2ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:33:07.449 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.438+0000 7f7fc2ffd700 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2023-09-30T01:33:07.449 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.438+0000 7f7fc2ffd700 10 monclient: dump: 2023-09-30T01:33:07.449 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:33:07.449 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:33:07.449 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:07.449 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:07.449 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:33:07.449 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:33:07.449 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:33:07.450 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:33:07.450 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:33:07.450 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:33:07.450 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:33:07.450 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:33:07.450 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:33:07.450 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:33:07.450 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:33:07.450 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:33:07.450 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.438+0000 7f7fc2ffd700 10 monclient: _finish_auth 0 2023-09-30T01:33:07.450 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.438+0000 7f7fc2ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:33:07.450 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.438+0000 7f7fc2ffd700 1 -- 172.21.15.88:0/2997816105 <== mon.8 v2:172.21.15.88:3304/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7fb4012430 con 0x7f7fc41311c0 2023-09-30T01:33:07.451 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.438+0000 7f7fc93f4700 5 monclient: authenticate success, global_id 4391 2023-09-30T01:33:07.451 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.438+0000 7f7fc93f4700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:33:07.451 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.438+0000 7f7fc2ffd700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:33:07.451 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.438+0000 7f7fc93f4700 10 monclient: get_monmap_and_config success 2023-09-30T01:33:07.451 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.438+0000 7f7fc2ffd700 1 -- 172.21.15.88:0/2997816105 <== mon.8 v2:172.21.15.88:3304/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f7fb4012a20 con 0x7f7fc41311c0 2023-09-30T01:33:07.451 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.438+0000 7f7fc93f4700 4 set_mon_vals no callback set 2023-09-30T01:33:07.451 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.438+0000 7f7fc2ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:33:07.451 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.438+0000 7f7fc2ffd700 10 monclient: got monmap 1 from mon.e (according to old e1) 2023-09-30T01:33:07.451 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.438+0000 7f7fc2ffd700 10 monclient: dump: 2023-09-30T01:33:07.451 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:33:07.451 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:33:07.451 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:07.451 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:07.452 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:33:07.452 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:33:07.452 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:33:07.452 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:33:07.452 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:33:07.452 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:33:07.452 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:33:07.452 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:33:07.452 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:33:07.452 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:33:07.452 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:33:07.452 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:33:07.452 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.442+0000 7f7fc93f4700 10 monclient: shutdown 2023-09-30T01:33:07.453 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.442+0000 7f7fc93f4700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:33:07.453 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.442+0000 7f7fc93f4700 1 -- 172.21.15.88:0/2997816105 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f7fc41311c0 msgr2=0x7f7fc413aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:33:07.453 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.442+0000 7f7fc93f4700 1 --2- 172.21.15.88:0/2997816105 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f7fc41311c0 0x7f7fc413aba0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f7fb4009e90 tx=0x7f7fb4009780 comp rx=0 tx=0).stop 2023-09-30T01:33:07.453 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.442+0000 7f7fc93f4700 1 -- 172.21.15.88:0/2997816105 shutdown_connections 2023-09-30T01:33:07.453 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.442+0000 7f7fc93f4700 1 --2- 172.21.15.88:0/2997816105 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f7fc41311c0 0x7f7fc413aba0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:07.453 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.442+0000 7f7fc93f4700 1 --2- 172.21.15.88:0/2997816105 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7f7fc4132e10 0x7f7fc4138650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:07.453 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.442+0000 7f7fc93f4700 1 -- 172.21.15.88:0/2997816105 >> 172.21.15.88:0/2997816105 conn(0x7f7fc406e5e0 msgr2=0x7f7fc406e9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:33:07.453 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.442+0000 7f7fc93f4700 1 -- 172.21.15.88:0/2997816105 shutdown_connections 2023-09-30T01:33:07.453 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.442+0000 7f7fc93f4700 1 -- 172.21.15.88:0/2997816105 wait complete. 2023-09-30T01:33:07.453 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.442+0000 7f7fc93f4700 10 monclient: build_initial_monmap 2023-09-30T01:33:07.453 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.442+0000 7f7fc93f4700 10 monclient: monmap: 2023-09-30T01:33:07.453 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2023-09-30T01:33:07.454 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:33:07.454 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 0.000000 2023-09-30T01:33:07.454 INFO:teuthology.orchestra.run.smithi088.stderr:created 0.000000 2023-09-30T01:33:07.454 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2023-09-30T01:33:07.454 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 1 2023-09-30T01:33:07.454 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.noname-a 2023-09-30T01:33:07.454 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.noname-f 2023-09-30T01:33:07.454 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.noname-b 2023-09-30T01:33:07.454 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.noname-g 2023-09-30T01:33:07.454 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.noname-c 2023-09-30T01:33:07.454 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.noname-h 2023-09-30T01:33:07.454 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.noname-d 2023-09-30T01:33:07.454 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.noname-i 2023-09-30T01:33:07.455 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.noname-e 2023-09-30T01:33:07.455 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:33:07.458 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.442+0000 7f7fc93f4700 1 Processor -- start 2023-09-30T01:33:07.458 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.442+0000 7f7fc93f4700 1 -- start start 2023-09-30T01:33:07.458 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.442+0000 7f7fc93f4700 10 monclient: init 2023-09-30T01:33:07.459 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.442+0000 7f7fc93f4700 10 monclient: _reopen_session rank -1 2023-09-30T01:33:07.459 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.442+0000 7f7fc93f4700 10 monclient: _add_conns ranks=[8,5,4,2,3,1,0,6,7] 2023-09-30T01:33:07.459 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.442+0000 7f7fc93f4700 1 --2- >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f7fc4097a40 0x7f7fc411cf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:33:07.459 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.442+0000 7f7fc93f4700 10 monclient(hunting): picked mon.noname-e con 0x7f7fc4097a40 addr [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] 2023-09-30T01:33:07.459 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.442+0000 7f7fc93f4700 1 --2- >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7f7fc41311c0 0x7f7fc411d460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:33:07.459 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.442+0000 7f7fc93f4700 10 monclient(hunting): picked mon.noname-h con 0x7f7fc41311c0 addr [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] 2023-09-30T01:33:07.459 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.442+0000 7f7fc93f4700 1 --2- >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f7fc4132e10 0x7f7fc411d9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:33:07.459 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.442+0000 7f7fc93f4700 10 monclient(hunting): picked mon.noname-c con 0x7f7fc4132e10 addr [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] 2023-09-30T01:33:07.459 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.442+0000 7f7fc93f4700 10 monclient(hunting): start opening mon connection 2023-09-30T01:33:07.459 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.442+0000 7f7fc93f4700 1 -- --> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] -- mon_getmap magic: 0 v1 -- 0x7f7fc4099c30 con 0x7f7fc4132e10 2023-09-30T01:33:07.459 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.442+0000 7f7fc93f4700 10 monclient(hunting): start opening mon connection 2023-09-30T01:33:07.460 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.442+0000 7f7fc93f4700 1 -- --> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] -- mon_getmap magic: 0 v1 -- 0x7f7fc40996b0 con 0x7f7fc41311c0 2023-09-30T01:33:07.460 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.442+0000 7f7fc93f4700 10 monclient(hunting): start opening mon connection 2023-09-30T01:33:07.460 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.442+0000 7f7fc93f4700 1 -- --> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] -- mon_getmap magic: 0 v1 -- 0x7f7fc411efd0 con 0x7f7fc4097a40 2023-09-30T01:33:07.460 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.442+0000 7f7fc93f4700 10 monclient(hunting): _renew_subs 2023-09-30T01:33:07.460 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.442+0000 7f7fc93f4700 10 monclient(hunting): authenticate will time out at 859.483085s 2023-09-30T01:33:07.460 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.442+0000 7f7fc3fff700 1 --2- >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f7fc4097a40 0x7f7fc411cf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:33:07.460 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.442+0000 7f7fc8bf3700 1 --2- >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f7fc4132e10 0x7f7fc411d9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:33:07.460 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.442+0000 7f7fc8bf3700 1 --2- >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f7fc4132e10 0x7f7fc411d9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3302/0 says I am v2:172.21.15.88:50466/0 (socket says 172.21.15.88:50466) 2023-09-30T01:33:07.460 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.442+0000 7f7fc3fff700 1 --2- >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f7fc4097a40 0x7f7fc411cf20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3304/0 says I am v2:172.21.15.88:36818/0 (socket says 172.21.15.88:36818) 2023-09-30T01:33:07.460 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.442+0000 7f7fc8bf3700 1 -- 172.21.15.88:0/690437577 learned_addr learned my addr 172.21.15.88:0/690437577 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-09-30T01:33:07.460 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.442+0000 7f7fc8bf3700 10 monclient(hunting): get_auth_request con 0x7f7fc4132e10 auth_method 0 2023-09-30T01:33:07.460 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.442+0000 7f7fc8bf3700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:33:07.460 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.442+0000 7f7fc8bf3700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:33:07.461 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.442+0000 7f7fc8bf3700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:33:07.461 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.442+0000 7f7fc3fff700 10 monclient(hunting): get_auth_request con 0x7f7fc4097a40 auth_method 0 2023-09-30T01:33:07.461 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.442+0000 7f7fc3fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:33:07.461 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.442+0000 7f7fc3fff700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:33:07.461 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.442+0000 7f7fc3fff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:33:07.461 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.442+0000 7f7fc8bf3700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:33:07.461 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.442+0000 7f7fc8bf3700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:33:07.461 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.442+0000 7f7fc37fe700 1 --2- 172.21.15.88:0/690437577 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7f7fc41311c0 0x7f7fc411d460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:33:07.461 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.442+0000 7f7fc8bf3700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:33:07.461 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.442+0000 7f7fc3fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:33:07.461 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.442+0000 7f7fc3fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:33:07.461 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.442+0000 7f7fc3fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:33:07.462 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.442+0000 7f7fc37fe700 10 monclient(hunting): get_auth_request con 0x7f7fc41311c0 auth_method 0 2023-09-30T01:33:07.462 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.442+0000 7f7fc37fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:33:07.462 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.442+0000 7f7fc37fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:33:07.462 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.442+0000 7f7fc37fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:33:07.462 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.442+0000 7f7fc3fff700 10 monclient(hunting): handle_auth_done global_id 4400 payload 824 2023-09-30T01:33:07.462 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.442+0000 7f7fc3fff700 1 -- 172.21.15.88:0/690437577 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7f7fc41311c0 msgr2=0x7f7fc411d460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:33:07.462 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.442+0000 7f7fc3fff700 1 --2- 172.21.15.88:0/690437577 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7f7fc41311c0 0x7f7fc411d460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:07.462 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.442+0000 7f7fc3fff700 1 -- 172.21.15.88:0/690437577 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f7fc4132e10 msgr2=0x7f7fc411d9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:33:07.462 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.442+0000 7f7fc3fff700 1 --2- 172.21.15.88:0/690437577 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f7fc4132e10 0x7f7fc411d9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:07.462 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.442+0000 7f7fc3fff700 10 monclient: _finish_hunting 0 2023-09-30T01:33:07.462 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.442+0000 7f7fc3fff700 1 monclient: found mon.noname-e 2023-09-30T01:33:07.462 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.442+0000 7f7fc3fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:33:07.462 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.442+0000 7f7fc3fff700 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.88:3304/0 2023-09-30T01:33:07.463 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.442+0000 7f7fc3fff700 1 -- 172.21.15.88:0/690437577 --> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7fb4009a20 con 0x7f7fc4097a40 2023-09-30T01:33:07.463 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.442+0000 7f7fc37fe700 1 --2- 172.21.15.88:0/690437577 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7f7fc41311c0 0x7f7fc411d460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-30T01:33:07.463 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.442+0000 7f7fc8bf3700 1 --2- 172.21.15.88:0/690437577 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f7fc4132e10 0x7f7fc411d9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:33:07.463 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.446+0000 7f7fc3fff700 1 --2- 172.21.15.88:0/690437577 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f7fc4097a40 0x7f7fc411cf20 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f7fbc0058d0 tx=0x7f7fbc007d10 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:33:07.463 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.446+0000 7f7fbbfff700 1 -- 172.21.15.88:0/690437577 <== mon.8 v2:172.21.15.88:3304/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f7fbc0070d0 con 0x7f7fc4097a40 2023-09-30T01:33:07.463 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.446+0000 7f7fbbfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:33:07.463 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.446+0000 7f7fbbfff700 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2023-09-30T01:33:07.463 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.446+0000 7f7fbbfff700 10 monclient: dump: 2023-09-30T01:33:07.463 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:33:07.463 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:33:07.463 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:07.463 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:07.464 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:33:07.464 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:33:07.464 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:33:07.464 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:33:07.464 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:33:07.464 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:33:07.464 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:33:07.464 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:33:07.464 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:33:07.464 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:33:07.464 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:33:07.464 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:33:07.465 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.446+0000 7f7fbbfff700 10 monclient: _finish_auth 0 2023-09-30T01:33:07.465 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.446+0000 7f7fbbfff700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:33:07.465 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.446+0000 7f7fc93f4700 5 monclient: authenticate success, global_id 4400 2023-09-30T01:33:07.465 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.446+0000 7f7fbbfff700 1 -- 172.21.15.88:0/690437577 <== mon.8 v2:172.21.15.88:3304/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7fbc007250 con 0x7f7fc4097a40 2023-09-30T01:33:07.465 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.446+0000 7f7fc93f4700 10 monclient: _renew_subs 2023-09-30T01:33:07.465 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.446+0000 7f7fc93f4700 10 monclient: _send_mon_message to mon.e at v2:172.21.15.88:3304/0 2023-09-30T01:33:07.465 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.446+0000 7f7fc93f4700 1 -- 172.21.15.88:0/690437577 --> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7fc411eb50 con 0x7f7fc4097a40 2023-09-30T01:33:07.465 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.446+0000 7f7fbbfff700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:33:07.465 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.446+0000 7f7fbbfff700 1 -- 172.21.15.88:0/690437577 <== mon.8 v2:172.21.15.88:3304/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f7fbc011360 con 0x7f7fc4097a40 2023-09-30T01:33:07.465 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.446+0000 7f7fbbfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:33:07.465 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.446+0000 7f7fbbfff700 10 monclient: got monmap 1 from mon.e (according to old e1) 2023-09-30T01:33:07.465 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.446+0000 7f7fc27fc700 4 set_mon_vals no callback set 2023-09-30T01:33:07.465 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.446+0000 7f7fbbfff700 10 monclient: dump: 2023-09-30T01:33:07.466 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:33:07.466 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:33:07.466 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:07.466 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:07.466 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:33:07.466 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:33:07.466 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:33:07.466 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:33:07.466 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:33:07.466 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:33:07.466 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:33:07.466 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:33:07.467 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:33:07.467 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:33:07.467 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:33:07.467 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:33:07.467 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.446+0000 7f7fc93f4700 10 monclient: _renew_subs 2023-09-30T01:33:07.467 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.446+0000 7f7fc93f4700 10 monclient: _send_mon_message to mon.e at v2:172.21.15.88:3304/0 2023-09-30T01:33:07.467 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.446+0000 7f7fc93f4700 1 -- 172.21.15.88:0/690437577 --> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7fc411e3f0 con 0x7f7fc4097a40 2023-09-30T01:33:07.467 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.446+0000 7f7fc93f4700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:33:07.467 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.446+0000 7f7fc93f4700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:33:07.467 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.446+0000 7f7fc93f4700 10 monclient: _send_mon_message to mon.e at v2:172.21.15.88:3304/0 2023-09-30T01:33:07.467 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.446+0000 7f7fbbfff700 1 -- 172.21.15.88:0/690437577 <== mon.8 v2:172.21.15.88:3304/0 4 ==== mgrmap(e 5) v1 ==== 40144+0+0 (secure 0 0 0) 0x7f7fbc005b30 con 0x7f7fc4097a40 2023-09-30T01:33:07.467 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.446+0000 7f7fc93f4700 1 -- 172.21.15.88:0/690437577 --> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7fc409e720 con 0x7f7fc4097a40 2023-09-30T01:33:07.467 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.446+0000 7f7fbbfff700 1 --2- 172.21.15.88:0/690437577 >> v2:172.21.15.107:6812/29926 conn(0x7f7fa4033360 0x7f7fa4035810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:33:07.468 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.446+0000 7f7fbbfff700 1 -- 172.21.15.88:0/690437577 <== mon.8 v2:172.21.15.88:3304/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 3502+0+0 (secure 0 0 0) 0x7f7fbc042430 con 0x7f7fc4097a40 2023-09-30T01:33:07.468 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.446+0000 7f7fc37fe700 1 --2- 172.21.15.88:0/690437577 >> v2:172.21.15.107:6812/29926 conn(0x7f7fa4033360 0x7f7fa4035810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:33:07.468 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.446+0000 7f7fc37fe700 10 monclient: get_auth_request con 0x7f7fa4033360 auth_method 0 2023-09-30T01:33:07.468 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.446+0000 7f7fc37fe700 1 --2- 172.21.15.88:0/690437577 >> v2:172.21.15.107:6812/29926 conn(0x7f7fa4033360 0x7f7fa4035810 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f7fac009ce0 tx=0x7f7fac008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:33:07.468 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.450+0000 7f7fbbfff700 1 -- 172.21.15.88:0/690437577 <== mon.8 v2:172.21.15.88:3304/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f7fbc018680 con 0x7f7fc4097a40 2023-09-30T01:33:07.468 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.450+0000 7f7fbbfff700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:33:07.468 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.450+0000 7f7fbbfff700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:33:07.611 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.606+0000 7f7fc93f4700 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 5}] 2023-09-30T01:33:07.611 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.606+0000 7f7fc93f4700 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2023-09-30T01:33:07.611 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.606+0000 7f7fc93f4700 10 monclient: _send_mon_message to mon.e at v2:172.21.15.88:3304/0 2023-09-30T01:33:07.611 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.606+0000 7f7fc93f4700 1 -- 172.21.15.88:0/690437577 --> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7f7fc4071960 con 0x7f7fc4097a40 2023-09-30T01:33:07.612 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.610+0000 7f7fbbfff700 1 -- 172.21.15.88:0/690437577 <== mon.8 v2:172.21.15.88:3304/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f7fbc017030 con 0x7f7fc4097a40 2023-09-30T01:33:07.612 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.610+0000 7f7fbbfff700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2023-09-30T01:33:07.612 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.610+0000 7f7fbbfff700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:33:07.612 INFO:teuthology.orchestra.run.smithi088.stdout:47244640258 2023-09-30T01:33:07.616 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.614+0000 7f7fc93f4700 1 -- 172.21.15.88:0/690437577 >> v2:172.21.15.107:6812/29926 conn(0x7f7fa4033360 msgr2=0x7f7fa4035810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:33:07.616 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.614+0000 7f7fc93f4700 1 --2- 172.21.15.88:0/690437577 >> v2:172.21.15.107:6812/29926 conn(0x7f7fa4033360 0x7f7fa4035810 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f7fac009ce0 tx=0x7f7fac008040 comp rx=0 tx=0).stop 2023-09-30T01:33:07.616 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.614+0000 7f7fc93f4700 10 monclient: shutdown 2023-09-30T01:33:07.616 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.614+0000 7f7fc93f4700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:33:07.616 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.614+0000 7f7fc93f4700 1 -- 172.21.15.88:0/690437577 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f7fc4097a40 msgr2=0x7f7fc411cf20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:33:07.616 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.614+0000 7f7fc93f4700 1 --2- 172.21.15.88:0/690437577 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f7fc4097a40 0x7f7fc411cf20 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f7fbc0058d0 tx=0x7f7fbc007d10 comp rx=0 tx=0).stop 2023-09-30T01:33:07.616 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.614+0000 7f7fc93f4700 1 -- 172.21.15.88:0/690437577 shutdown_connections 2023-09-30T01:33:07.616 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.614+0000 7f7fc93f4700 1 --2- 172.21.15.88:0/690437577 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7f7fc41311c0 0x7f7fc411d460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:07.617 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.614+0000 7f7fc93f4700 1 --2- 172.21.15.88:0/690437577 >> v2:172.21.15.107:6812/29926 conn(0x7f7fa4033360 0x7f7fa4035810 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:07.617 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.614+0000 7f7fc93f4700 1 --2- 172.21.15.88:0/690437577 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f7fc4132e10 0x7f7fc411d9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:07.617 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.614+0000 7f7fc93f4700 1 --2- 172.21.15.88:0/690437577 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f7fc4097a40 0x7f7fc411cf20 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:07.617 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.614+0000 7f7fc93f4700 1 -- 172.21.15.88:0/690437577 >> 172.21.15.88:0/690437577 conn(0x7f7fc406e5e0 msgr2=0x7f7fc40a6ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:33:07.617 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.614+0000 7f7fc93f4700 1 -- 172.21.15.88:0/690437577 shutdown_connections 2023-09-30T01:33:07.617 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.614+0000 7f7fc93f4700 1 -- 172.21.15.88:0/690437577 wait complete. 2023-09-30T01:33:07.630 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640258 got 47244640258 for osd.5 2023-09-30T01:33:07.630 DEBUG:teuthology.parallel:result is None 2023-09-30T01:33:07.630 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2023-09-30T01:33:07.630 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-09-30T01:33:07.732 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.730+0000 7fcb7933c700 10 monclient: get_monmap_and_config 2023-09-30T01:33:07.732 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.730+0000 7fcb7933c700 10 monclient: build_initial_monmap 2023-09-30T01:33:07.733 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.730+0000 7fcb7933c700 10 monclient: monmap: 2023-09-30T01:33:07.733 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2023-09-30T01:33:07.733 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:33:07.733 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:33:07.733242+0000 2023-09-30T01:33:07.733 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:33:07.733242+0000 2023-09-30T01:33:07.733 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2023-09-30T01:33:07.733 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:33:07.733 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a-legacy 2023-09-30T01:33:07.733 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.107:6789/0 mon.noname-f-legacy 2023-09-30T01:33:07.733 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.noname-b 2023-09-30T01:33:07.734 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.noname-g 2023-09-30T01:33:07.734 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.noname-c 2023-09-30T01:33:07.734 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.noname-h 2023-09-30T01:33:07.734 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.noname-d 2023-09-30T01:33:07.734 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.noname-i 2023-09-30T01:33:07.734 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.noname-e 2023-09-30T01:33:07.734 INFO:teuthology.orchestra.run.smithi088.stderr:9: v2:172.21.15.88:3300/0 mon.noname-a 2023-09-30T01:33:07.734 INFO:teuthology.orchestra.run.smithi088.stderr:10: v2:172.21.15.107:3300/0 mon.noname-f 2023-09-30T01:33:07.734 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:33:07.742 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.730+0000 7fcb7933c700 1 Processor -- start 2023-09-30T01:33:07.742 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.730+0000 7fcb7933c700 1 -- start start 2023-09-30T01:33:07.742 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.730+0000 7fcb7933c700 10 monclient: init 2023-09-30T01:33:07.742 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.730+0000 7fcb7933c700 10 monclient: _reopen_session rank -1 2023-09-30T01:33:07.742 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.730+0000 7fcb7933c700 10 monclient: _add_conns ranks=[6,0,1,4,3,2,8,10,9,7,5] 2023-09-30T01:33:07.742 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.730+0000 7fcb7933c700 1 --2- >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7fcb740a3380 0x7fcb740a48b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:33:07.742 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.730+0000 7fcb7933c700 10 monclient(hunting): picked mon.noname-d con 0x7fcb740a3380 addr [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] 2023-09-30T01:33:07.742 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.730+0000 7fcb7933c700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fcb7409ffd0 addr v1:172.21.15.88:6789/0 2023-09-30T01:33:07.742 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.730+0000 7fcb7933c700 10 monclient(hunting): picked mon.noname-f-legacy con 0x7fcb740a03a0 addr v1:172.21.15.107:6789/0 2023-09-30T01:33:07.743 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.730+0000 7fcb7933c700 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb7404bb20 con 0x7fcb7409ffd0 2023-09-30T01:33:07.743 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.730+0000 7fcb7933c700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb74098480 con 0x7fcb740a03a0 2023-09-30T01:33:07.743 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.730+0000 7fcb7933c700 10 monclient(hunting): start opening mon connection 2023-09-30T01:33:07.743 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.730+0000 7fcb7933c700 1 -- --> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] -- mon_getmap magic: 0 v1 -- 0x7fcb74097ab0 con 0x7fcb740a3380 2023-09-30T01:33:07.743 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.730+0000 7fcb7933c700 10 monclient(hunting): _renew_subs 2023-09-30T01:33:07.743 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.730+0000 7fcb7933c700 10 monclient(hunting): authenticate will time out at 859.770682s 2023-09-30T01:33:07.743 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.730+0000 7fcb73fff700 1 --2- >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7fcb740a3380 0x7fcb740a48b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:33:07.743 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.730+0000 7fcb737fe700 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7fcb7409ffd0 0x7fcb741333b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:54148/0 (socket says 172.21.15.88:54148) 2023-09-30T01:33:07.743 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.730+0000 7fcb737fe700 1 -- 172.21.15.88:0/2348249030 learned_addr learned my addr 172.21.15.88:0/2348249030 (peer_addr_for_me v1:172.21.15.88:0/0) 2023-09-30T01:33:07.743 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.730+0000 7fcb73fff700 10 monclient(hunting): get_auth_request con 0x7fcb740a3380 auth_method 0 2023-09-30T01:33:07.743 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.730+0000 7fcb73fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:33:07.744 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.730+0000 7fcb73fff700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:33:07.744 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.730+0000 7fcb73fff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:33:07.744 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.730+0000 7fcb73fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:33:07.744 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.730+0000 7fcb73fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:33:07.744 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.730+0000 7fcb73fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:33:07.744 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.730+0000 7fcb72ffd700 1 -- 172.21.15.88:0/2348249030 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3570744745 0 0) 0x7fcb7404bb20 con 0x7fcb7409ffd0 2023-09-30T01:33:07.744 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.730+0000 7fcb72ffd700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:33:07.744 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.730+0000 7fcb72ffd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:33:07.744 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.730+0000 7fcb72ffd700 10 monclient(hunting): my global_id is 4419 2023-09-30T01:33:07.744 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.730+0000 7fcb72ffd700 1 -- 172.21.15.88:0/2348249030 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcb54003570 con 0x7fcb7409ffd0 2023-09-30T01:33:07.744 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.730+0000 7fcb73fff700 10 monclient(hunting): handle_auth_done global_id 4299 payload 274 2023-09-30T01:33:07.744 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.730+0000 7fcb72ffd700 1 -- 172.21.15.88:0/2348249030 <== mon.1 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2414624835 0 0) 0x7fcb74098480 con 0x7fcb740a03a0 2023-09-30T01:33:07.745 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.730+0000 7fcb73fff700 1 -- 172.21.15.88:0/2348249030 >> v1:172.21.15.107:6789/0 conn(0x7fcb740a03a0 legacy=0x7fcb74133ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:33:07.745 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.730+0000 7fcb73fff700 1 -- 172.21.15.88:0/2348249030 >> v1:172.21.15.88:6789/0 conn(0x7fcb7409ffd0 legacy=0x7fcb741333b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:33:07.745 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.730+0000 7fcb73fff700 10 monclient: _finish_hunting 0 2023-09-30T01:33:07.745 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.730+0000 7fcb73fff700 1 monclient: found mon.noname-d 2023-09-30T01:33:07.745 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.730+0000 7fcb73fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:33:07.745 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.730+0000 7fcb73fff700 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.88:3303/0 2023-09-30T01:33:07.745 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.730+0000 7fcb73fff700 1 -- 172.21.15.88:0/2348249030 --> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb740a3750 con 0x7fcb740a3380 2023-09-30T01:33:07.745 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.730+0000 7fcb72ffd700 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2023-09-30T01:33:07.745 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.734+0000 7fcb73fff700 1 --2- 172.21.15.88:0/2348249030 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7fcb740a3380 0x7fcb740a48b0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fcb6c009910 tx=0x7fcb6c0091a0 comp rx=0 tx=0).ready entity=mon.6 client_cookie=434af3364e203b3b server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:33:07.745 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.734+0000 7fcb72ffd700 1 -- 172.21.15.88:0/2348249030 <== mon.6 v2:172.21.15.88:3303/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7fcb6c00bb00 con 0x7fcb740a3380 2023-09-30T01:33:07.745 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.734+0000 7fcb72ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:33:07.745 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.734+0000 7fcb72ffd700 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2023-09-30T01:33:07.746 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.734+0000 7fcb72ffd700 10 monclient: dump: 2023-09-30T01:33:07.746 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:33:07.746 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:33:07.746 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:07.746 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:07.746 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:33:07.746 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:33:07.746 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:33:07.746 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:33:07.746 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:33:07.746 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:33:07.746 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:33:07.747 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:33:07.747 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:33:07.747 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:33:07.747 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:33:07.747 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:33:07.747 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.734+0000 7fcb72ffd700 10 monclient: _finish_auth 0 2023-09-30T01:33:07.747 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.734+0000 7fcb72ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:33:07.747 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.734+0000 7fcb72ffd700 1 -- 172.21.15.88:0/2348249030 <== mon.6 v2:172.21.15.88:3303/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb6c00bc80 con 0x7fcb740a3380 2023-09-30T01:33:07.747 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.734+0000 7fcb7933c700 5 monclient: authenticate success, global_id 4299 2023-09-30T01:33:07.747 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.734+0000 7fcb7933c700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:33:07.747 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.734+0000 7fcb72ffd700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:33:07.748 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.734+0000 7fcb7933c700 10 monclient: get_monmap_and_config success 2023-09-30T01:33:07.748 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.734+0000 7fcb72ffd700 1 -- 172.21.15.88:0/2348249030 <== mon.6 v2:172.21.15.88:3303/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7fcb6c00bdf0 con 0x7fcb740a3380 2023-09-30T01:33:07.748 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.734+0000 7fcb72ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:33:07.748 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.734+0000 7fcb7933c700 4 set_mon_vals no callback set 2023-09-30T01:33:07.748 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.734+0000 7fcb72ffd700 10 monclient: got monmap 1 from mon.d (according to old e1) 2023-09-30T01:33:07.748 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.734+0000 7fcb72ffd700 10 monclient: dump: 2023-09-30T01:33:07.748 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:33:07.749 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:33:07.749 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:07.749 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:07.749 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:33:07.749 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:33:07.749 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:33:07.749 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:33:07.749 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:33:07.749 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:33:07.749 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:33:07.750 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:33:07.750 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:33:07.750 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:33:07.750 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:33:07.750 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:33:07.750 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.734+0000 7fcb7933c700 10 monclient: shutdown 2023-09-30T01:33:07.750 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.734+0000 7fcb7933c700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:33:07.750 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.734+0000 7fcb7933c700 1 -- 172.21.15.88:0/2348249030 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7fcb740a3380 msgr2=0x7fcb740a48b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:33:07.750 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.734+0000 7fcb7933c700 1 --2- 172.21.15.88:0/2348249030 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7fcb740a3380 0x7fcb740a48b0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fcb6c009910 tx=0x7fcb6c0091a0 comp rx=0 tx=0).stop 2023-09-30T01:33:07.751 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.734+0000 7fcb7933c700 1 -- 172.21.15.88:0/2348249030 shutdown_connections 2023-09-30T01:33:07.751 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.734+0000 7fcb7933c700 1 --2- 172.21.15.88:0/2348249030 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7fcb740a3380 0x7fcb740a48b0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:07.751 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.734+0000 7fcb7933c700 1 -- 172.21.15.88:0/2348249030 >> 172.21.15.88:0/2348249030 conn(0x7fcb7406e5e0 msgr2=0x7fcb7406e9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:33:07.751 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.734+0000 7fcb7933c700 1 -- 172.21.15.88:0/2348249030 shutdown_connections 2023-09-30T01:33:07.751 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.734+0000 7fcb7933c700 1 -- 172.21.15.88:0/2348249030 wait complete. 2023-09-30T01:33:07.751 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.734+0000 7fcb7933c700 10 monclient: build_initial_monmap 2023-09-30T01:33:07.751 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.734+0000 7fcb7933c700 10 monclient: monmap: 2023-09-30T01:33:07.751 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2023-09-30T01:33:07.751 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:33:07.751 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 0.000000 2023-09-30T01:33:07.752 INFO:teuthology.orchestra.run.smithi088.stderr:created 0.000000 2023-09-30T01:33:07.752 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2023-09-30T01:33:07.752 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 1 2023-09-30T01:33:07.752 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.noname-a 2023-09-30T01:33:07.752 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.noname-f 2023-09-30T01:33:07.752 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.noname-b 2023-09-30T01:33:07.752 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.noname-g 2023-09-30T01:33:07.752 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.noname-c 2023-09-30T01:33:07.752 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.noname-h 2023-09-30T01:33:07.752 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.noname-d 2023-09-30T01:33:07.753 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.noname-i 2023-09-30T01:33:07.753 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.noname-e 2023-09-30T01:33:07.753 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:33:07.753 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.734+0000 7fcb7933c700 1 Processor -- start 2023-09-30T01:33:07.753 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.734+0000 7fcb7933c700 1 -- start start 2023-09-30T01:33:07.753 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.734+0000 7fcb7933c700 10 monclient: init 2023-09-30T01:33:07.753 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.738+0000 7fcb7933c700 10 monclient: _reopen_session rank -1 2023-09-30T01:33:07.753 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.738+0000 7fcb7933c700 10 monclient: _add_conns ranks=[4,1,2,7,6,5,0,3,8] 2023-09-30T01:33:07.753 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.738+0000 7fcb7933c700 1 --2- >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7fcb7409ffd0 0x7fcb7413ee80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:33:07.753 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.738+0000 7fcb7933c700 10 monclient(hunting): picked mon.noname-c con 0x7fcb7409ffd0 addr [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] 2023-09-30T01:33:07.753 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.738+0000 7fcb7933c700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcb740a03a0 0x7fcb7413e530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:33:07.754 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.738+0000 7fcb7933c700 10 monclient(hunting): picked mon.noname-f con 0x7fcb740a03a0 addr [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] 2023-09-30T01:33:07.754 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.738+0000 7fcb7933c700 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fcb74122700 0x7fcb741423f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:33:07.754 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.738+0000 7fcb7933c700 10 monclient(hunting): picked mon.noname-b con 0x7fcb74122700 addr [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] 2023-09-30T01:33:07.754 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.738+0000 7fcb7933c700 10 monclient(hunting): start opening mon connection 2023-09-30T01:33:07.754 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.738+0000 7fcb7933c700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb74123f00 con 0x7fcb740a03a0 2023-09-30T01:33:07.754 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.738+0000 7fcb7933c700 10 monclient(hunting): start opening mon connection 2023-09-30T01:33:07.754 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.738+0000 7fcb7933c700 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb741235d0 con 0x7fcb74122700 2023-09-30T01:33:07.754 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.738+0000 7fcb7933c700 10 monclient(hunting): start opening mon connection 2023-09-30T01:33:07.754 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.738+0000 7fcb7933c700 1 -- --> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] -- mon_getmap magic: 0 v1 -- 0x7fcb741238b0 con 0x7fcb7409ffd0 2023-09-30T01:33:07.754 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.738+0000 7fcb7933c700 10 monclient(hunting): _renew_subs 2023-09-30T01:33:07.755 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.738+0000 7fcb7933c700 10 monclient(hunting): authenticate will time out at 859.776350s 2023-09-30T01:33:07.755 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.738+0000 7fcb73fff700 1 --2- >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7fcb7409ffd0 0x7fcb7413ee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:33:07.755 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.738+0000 7fcb73fff700 1 --2- >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7fcb7409ffd0 0x7fcb7413ee80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3302/0 says I am v2:172.21.15.88:50472/0 (socket says 172.21.15.88:50472) 2023-09-30T01:33:07.755 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.738+0000 7fcb73fff700 1 -- 172.21.15.88:0/1279182356 learned_addr learned my addr 172.21.15.88:0/1279182356 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-09-30T01:33:07.755 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.738+0000 7fcb73fff700 10 monclient(hunting): get_auth_request con 0x7fcb7409ffd0 auth_method 0 2023-09-30T01:33:07.755 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.738+0000 7fcb73fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:33:07.755 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.738+0000 7fcb73fff700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:33:07.755 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.738+0000 7fcb73fff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:33:07.755 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.738+0000 7fcb73fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:33:07.756 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.738+0000 7fcb73fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:33:07.756 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.738+0000 7fcb78b3b700 1 --2- 172.21.15.88:0/1279182356 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fcb74122700 0x7fcb741423f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:33:07.756 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.738+0000 7fcb73fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:33:07.756 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.738+0000 7fcb78b3b700 10 monclient(hunting): get_auth_request con 0x7fcb74122700 auth_method 0 2023-09-30T01:33:07.756 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.738+0000 7fcb78b3b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:33:07.756 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.738+0000 7fcb78b3b700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:33:07.756 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.738+0000 7fcb78b3b700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:33:07.756 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.738+0000 7fcb737fe700 1 --2- 172.21.15.88:0/1279182356 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcb740a03a0 0x7fcb7413e530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:33:07.756 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.738+0000 7fcb737fe700 10 monclient(hunting): get_auth_request con 0x7fcb740a03a0 auth_method 0 2023-09-30T01:33:07.756 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.738+0000 7fcb737fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:33:07.756 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.738+0000 7fcb737fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:33:07.757 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.738+0000 7fcb737fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:33:07.757 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.738+0000 7fcb78b3b700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:33:07.757 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.738+0000 7fcb78b3b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:33:07.757 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.738+0000 7fcb78b3b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:33:07.757 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.738+0000 7fcb73fff700 10 monclient(hunting): handle_auth_done global_id 4342 payload 824 2023-09-30T01:33:07.757 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.738+0000 7fcb73fff700 1 -- 172.21.15.88:0/1279182356 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fcb74122700 msgr2=0x7fcb741423f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:33:07.757 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.738+0000 7fcb73fff700 1 --2- 172.21.15.88:0/1279182356 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fcb74122700 0x7fcb741423f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:07.757 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.738+0000 7fcb73fff700 1 -- 172.21.15.88:0/1279182356 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcb740a03a0 msgr2=0x7fcb7413e530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:33:07.757 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.738+0000 7fcb73fff700 1 --2- 172.21.15.88:0/1279182356 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcb740a03a0 0x7fcb7413e530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:07.757 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.738+0000 7fcb73fff700 10 monclient: _finish_hunting 0 2023-09-30T01:33:07.758 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.738+0000 7fcb73fff700 1 monclient: found mon.noname-c 2023-09-30T01:33:07.758 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.738+0000 7fcb73fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:33:07.758 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.738+0000 7fcb73fff700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.88:3302/0 2023-09-30T01:33:07.758 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.738+0000 7fcb73fff700 1 -- 172.21.15.88:0/1279182356 --> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb6c009420 con 0x7fcb7409ffd0 2023-09-30T01:33:07.758 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.738+0000 7fcb73fff700 1 --2- 172.21.15.88:0/1279182356 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7fcb7409ffd0 0x7fcb7413ee80 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fcb74098610 tx=0x7fcb6c017000 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:33:07.758 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.738+0000 7fcb6bfff700 1 -- 172.21.15.88:0/1279182356 <== mon.4 v2:172.21.15.88:3302/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7fcb6c00bdf0 con 0x7fcb7409ffd0 2023-09-30T01:33:07.758 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.738+0000 7fcb6bfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:33:07.759 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.738+0000 7fcb6bfff700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T01:33:07.759 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.738+0000 7fcb6bfff700 10 monclient: dump: 2023-09-30T01:33:07.759 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:33:07.759 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:33:07.759 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:07.759 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:07.759 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:33:07.760 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:33:07.760 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:33:07.760 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:33:07.760 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:33:07.760 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:33:07.760 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:33:07.760 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:33:07.760 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:33:07.760 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:33:07.760 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:33:07.760 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:33:07.760 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.738+0000 7fcb6bfff700 10 monclient: _finish_auth 0 2023-09-30T01:33:07.761 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.738+0000 7fcb6bfff700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:33:07.761 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.738+0000 7fcb6bfff700 1 -- 172.21.15.88:0/1279182356 <== mon.4 v2:172.21.15.88:3302/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb6c025900 con 0x7fcb7409ffd0 2023-09-30T01:33:07.761 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.738+0000 7fcb7933c700 5 monclient: authenticate success, global_id 4342 2023-09-30T01:33:07.761 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.738+0000 7fcb7933c700 10 monclient: _renew_subs 2023-09-30T01:33:07.761 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.738+0000 7fcb7933c700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.88:3302/0 2023-09-30T01:33:07.761 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.738+0000 7fcb7933c700 1 -- 172.21.15.88:0/1279182356 --> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb74123b90 con 0x7fcb7409ffd0 2023-09-30T01:33:07.761 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.738+0000 7fcb6bfff700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:33:07.761 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.738+0000 7fcb6bfff700 1 -- 172.21.15.88:0/1279182356 <== mon.4 v2:172.21.15.88:3302/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7fcb6c01a450 con 0x7fcb7409ffd0 2023-09-30T01:33:07.761 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.738+0000 7fcb6bfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:33:07.761 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.738+0000 7fcb6bfff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T01:33:07.761 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.738+0000 7fcb6bfff700 10 monclient: dump: 2023-09-30T01:33:07.762 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:33:07.762 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:33:07.762 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:07.762 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:07.762 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:33:07.762 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:33:07.762 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:33:07.762 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:33:07.762 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:33:07.762 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:33:07.762 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:33:07.763 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:33:07.763 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:33:07.763 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:33:07.763 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:33:07.763 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:33:07.763 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.738+0000 7fcb71ffb700 4 set_mon_vals no callback set 2023-09-30T01:33:07.763 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.738+0000 7fcb7933c700 10 monclient: _renew_subs 2023-09-30T01:33:07.763 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.738+0000 7fcb7933c700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.88:3302/0 2023-09-30T01:33:07.763 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.738+0000 7fcb7933c700 1 -- 172.21.15.88:0/1279182356 --> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb7413ea70 con 0x7fcb7409ffd0 2023-09-30T01:33:07.763 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.738+0000 7fcb7933c700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:33:07.763 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.738+0000 7fcb7933c700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:33:07.763 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.738+0000 7fcb7933c700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.88:3302/0 2023-09-30T01:33:07.763 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.738+0000 7fcb7933c700 1 -- 172.21.15.88:0/1279182356 --> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcb74121420 con 0x7fcb7409ffd0 2023-09-30T01:33:07.764 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.738+0000 7fcb6bfff700 1 -- 172.21.15.88:0/1279182356 <== mon.4 v2:172.21.15.88:3302/0 4 ==== mgrmap(e 5) v1 ==== 40144+0+0 (secure 0 0 0) 0x7fcb6c025480 con 0x7fcb7409ffd0 2023-09-30T01:33:07.765 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.738+0000 7fcb6bfff700 1 --2- 172.21.15.88:0/1279182356 >> v2:172.21.15.107:6812/29926 conn(0x7fcb58033290 0x7fcb58035740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:33:07.765 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.738+0000 7fcb6bfff700 1 -- 172.21.15.88:0/1279182356 <== mon.4 v2:172.21.15.88:3302/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 3502+0+0 (secure 0 0 0) 0x7fcb6c044fa0 con 0x7fcb7409ffd0 2023-09-30T01:33:07.765 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.742+0000 7fcb737fe700 1 --2- 172.21.15.88:0/1279182356 >> v2:172.21.15.107:6812/29926 conn(0x7fcb58033290 0x7fcb58035740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:33:07.765 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.742+0000 7fcb737fe700 10 monclient: get_auth_request con 0x7fcb58033290 auth_method 0 2023-09-30T01:33:07.765 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.742+0000 7fcb737fe700 1 --2- 172.21.15.88:0/1279182356 >> v2:172.21.15.107:6812/29926 conn(0x7fcb58033290 0x7fcb58035740 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fcb7411fe70 tx=0x7fcb5c009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:33:07.765 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.746+0000 7fcb6bfff700 1 -- 172.21.15.88:0/1279182356 <== mon.4 v2:172.21.15.88:3302/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fcb6c01a5d0 con 0x7fcb7409ffd0 2023-09-30T01:33:07.765 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.746+0000 7fcb6bfff700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:33:07.765 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.746+0000 7fcb6bfff700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:33:07.896 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.894+0000 7fcb7933c700 1 -- 172.21.15.88:0/1279182356 --> v2:172.21.15.107:6812/29926 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fcb7413ec60 con 0x7fcb58033290 2023-09-30T01:33:07.899 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.898+0000 7fcb6bfff700 1 -- 172.21.15.88:0/1279182356 <== mgr.4108 v2:172.21.15.107:6812/29926 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+40390 (secure 0 0 0) 0x7fcb7413ec60 con 0x7fcb58033290 2023-09-30T01:33:07.901 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-09-30T01:33:07.902 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.898+0000 7fcb7933c700 1 -- 172.21.15.88:0/1279182356 >> v2:172.21.15.107:6812/29926 conn(0x7fcb58033290 msgr2=0x7fcb58035740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:33:07.902 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.898+0000 7fcb7933c700 1 --2- 172.21.15.88:0/1279182356 >> v2:172.21.15.107:6812/29926 conn(0x7fcb58033290 0x7fcb58035740 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fcb7411fe70 tx=0x7fcb5c009040 comp rx=0 tx=0).stop 2023-09-30T01:33:07.903 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.898+0000 7fcb7933c700 10 monclient: shutdown 2023-09-30T01:33:07.903 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.898+0000 7fcb7933c700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:33:07.903 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.898+0000 7fcb7933c700 1 -- 172.21.15.88:0/1279182356 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7fcb7409ffd0 msgr2=0x7fcb7413ee80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:33:07.903 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.898+0000 7fcb7933c700 1 --2- 172.21.15.88:0/1279182356 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7fcb7409ffd0 0x7fcb7413ee80 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fcb74098610 tx=0x7fcb6c017000 comp rx=0 tx=0).stop 2023-09-30T01:33:07.903 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.898+0000 7fcb7933c700 1 -- 172.21.15.88:0/1279182356 shutdown_connections 2023-09-30T01:33:07.904 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.898+0000 7fcb7933c700 1 --2- 172.21.15.88:0/1279182356 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fcb74122700 0x7fcb741423f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:07.904 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.898+0000 7fcb7933c700 1 --2- 172.21.15.88:0/1279182356 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcb740a03a0 0x7fcb7413e530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:07.904 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.898+0000 7fcb7933c700 1 --2- 172.21.15.88:0/1279182356 >> v2:172.21.15.107:6812/29926 conn(0x7fcb58033290 0x7fcb58035740 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:07.904 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.898+0000 7fcb7933c700 1 --2- 172.21.15.88:0/1279182356 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7fcb7409ffd0 0x7fcb7413ee80 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:07.904 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.898+0000 7fcb7933c700 1 -- 172.21.15.88:0/1279182356 >> 172.21.15.88:0/1279182356 conn(0x7fcb7406e5e0 msgr2=0x7fcb740a28d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:33:07.905 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.898+0000 7fcb7933c700 1 -- 172.21.15.88:0/1279182356 shutdown_connections 2023-09-30T01:33:07.905 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:07.898+0000 7fcb7933c700 1 -- 172.21.15.88:0/1279182356 wait complete. 2023-09-30T01:33:07.905 INFO:teuthology.orchestra.run.smithi088.stderr:dumped all 2023-09-30T01:33:07.913 INFO:teuthology.orchestra.run.smithi088.stdout:{"pg_ready":true,"pg_map":{"version":14,"stamp":"2023-09-30T01:33:07.653762+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":11,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":32896,"kb_used_data":1748,"kb_used_omap":0,"kb_used_meta":30784,"kb_avail":566198144,"statfs":{"total":579820584960,"available":579786899456,"internally_reserved":0,"allocated":1789952,"data_stored":1223102,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":31522816},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":9,"num_write":14,"num_write_kb":262,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"2.000676"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-09-30T01:33:03.296691+0000","last_change":"2023-09-30T01:33:03.292315+0000","last_active":"2023-09-30T01:33:03.296691+0000","last_peered":"2023-09-30T01:33:03.296691+0000","last_clean":"2023-09-30T01:33:03.296691+0000","last_became_active":"2023-09-30T01:33:03.291889+0000","last_became_peered":"2023-09-30T01:33:03.291889+0000","last_unstale":"2023-09-30T01:33:03.296691+0000","last_undegraded":"2023-09-30T01:33:03.296691+0000","last_fullsized":"2023-09-30T01:33:03.296691+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T01:33:02.252599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T01:33:02.252599+0000","last_clean_scrub_stamp":"2023-09-30T01:33:02.252599+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T11:44:00.574379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-09-30T01:33:03.297822+0000","last_change":"2023-09-30T01:33:03.295081+0000","last_active":"2023-09-30T01:33:03.297822+0000","last_peered":"2023-09-30T01:33:03.297822+0000","last_clean":"2023-09-30T01:33:03.297822+0000","last_became_active":"2023-09-30T01:33:03.294535+0000","last_became_peered":"2023-09-30T01:33:03.294535+0000","last_unstale":"2023-09-30T01:33:03.297822+0000","last_undegraded":"2023-09-30T01:33:03.297822+0000","last_fullsized":"2023-09-30T01:33:03.297822+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T01:33:02.252599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T01:33:02.252599+0000","last_clean_scrub_stamp":"2023-09-30T01:33:02.252599+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T05:49:49.642723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-09-30T01:33:03.300117+0000","last_change":"2023-09-30T01:33:03.297392+0000","last_active":"2023-09-30T01:33:03.300117+0000","last_peered":"2023-09-30T01:33:03.300117+0000","last_clean":"2023-09-30T01:33:03.300117+0000","last_became_active":"2023-09-30T01:33:03.296959+0000","last_became_peered":"2023-09-30T01:33:03.296959+0000","last_unstale":"2023-09-30T01:33:03.300117+0000","last_undegraded":"2023-09-30T01:33:03.300117+0000","last_fullsized":"2023-09-30T01:33:03.300117+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T01:33:02.252599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T01:33:02.252599+0000","last_clean_scrub_stamp":"2023-09-30T01:33:02.252599+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T11:57:13.112607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-09-30T01:33:03.296932+0000","last_change":"2023-09-30T01:33:03.295478+0000","last_active":"2023-09-30T01:33:03.296932+0000","last_peered":"2023-09-30T01:33:03.296932+0000","last_clean":"2023-09-30T01:33:03.296932+0000","last_became_active":"2023-09-30T01:33:03.295106+0000","last_became_peered":"2023-09-30T01:33:03.295106+0000","last_unstale":"2023-09-30T01:33:03.296932+0000","last_undegraded":"2023-09-30T01:33:03.296932+0000","last_fullsized":"2023-09-30T01:33:03.296932+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T01:33:02.252599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T01:33:02.252599+0000","last_clean_scrub_stamp":"2023-09-30T01:33:02.252599+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T07:34:36.123333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"15'117","reported_seq":159,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-09-30T01:33:05.391246+0000","last_change":"2023-09-30T01:33:03.296872+0000","last_active":"2023-09-30T01:33:05.391246+0000","last_peered":"2023-09-30T01:33:05.391246+0000","last_clean":"2023-09-30T01:33:05.391246+0000","last_became_active":"2023-09-30T01:33:03.295200+0000","last_became_peered":"2023-09-30T01:33:03.295200+0000","last_unstale":"2023-09-30T01:33:05.391246+0000","last_undegraded":"2023-09-30T01:33:05.391246+0000","last_fullsized":"2023-09-30T01:33:05.391246+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T01:33:02.252599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T01:33:02.252599+0000","last_clean_scrub_stamp":"2023-09-30T01:33:02.252599+0000","objects_scrubbed":0,"log_size":117,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T01:51:03.655484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-09-30T01:33:03.296258+0000","last_change":"2023-09-30T01:33:03.293385+0000","last_active":"2023-09-30T01:33:03.296258+0000","last_peered":"2023-09-30T01:33:03.296258+0000","last_clean":"2023-09-30T01:33:03.296258+0000","last_became_active":"2023-09-30T01:33:03.293004+0000","last_became_peered":"2023-09-30T01:33:03.293004+0000","last_unstale":"2023-09-30T01:33:03.296258+0000","last_undegraded":"2023-09-30T01:33:03.296258+0000","last_fullsized":"2023-09-30T01:33:03.296258+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T01:33:02.252599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T01:33:02.252599+0000","last_clean_scrub_stamp":"2023-09-30T01:33:02.252599+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T03:32:32.404638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-09-30T01:33:03.299431+0000","last_change":"2023-09-30T01:33:03.296945+0000","last_active":"2023-09-30T01:33:03.299431+0000","last_peered":"2023-09-30T01:33:03.299431+0000","last_clean":"2023-09-30T01:33:03.299431+0000","last_became_active":"2023-09-30T01:33:03.296465+0000","last_became_peered":"2023-09-30T01:33:03.296465+0000","last_unstale":"2023-09-30T01:33:03.299431+0000","last_undegraded":"2023-09-30T01:33:03.299431+0000","last_fullsized":"2023-09-30T01:33:03.299431+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T01:33:02.252599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T01:33:02.252599+0000","last_clean_scrub_stamp":"2023-09-30T01:33:02.252599+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T05:32:13.855128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-09-30T01:33:03.296569+0000","last_change":"2023-09-30T01:33:03.291527+0000","last_active":"2023-09-30T01:33:03.296569+0000","last_peered":"2023-09-30T01:33:03.296569+0000","last_clean":"2023-09-30T01:33:03.296569+0000","last_became_active":"2023-09-30T01:33:03.291084+0000","last_became_peered":"2023-09-30T01:33:03.291084+0000","last_unstale":"2023-09-30T01:33:03.296569+0000","last_undegraded":"2023-09-30T01:33:03.296569+0000","last_fullsized":"2023-09-30T01:33:03.296569+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T01:33:02.252599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T01:33:02.252599+0000","last_clean_scrub_stamp":"2023-09-30T01:33:02.252599+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T12:10:57.124090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-09-30T01:33:03.299303+0000","last_change":"2023-09-30T01:33:03.296313+0000","last_active":"2023-09-30T01:33:03.299303+0000","last_peered":"2023-09-30T01:33:03.299303+0000","last_clean":"2023-09-30T01:33:03.299303+0000","last_became_active":"2023-09-30T01:33:03.294604+0000","last_became_peered":"2023-09-30T01:33:03.294604+0000","last_unstale":"2023-09-30T01:33:03.299303+0000","last_undegraded":"2023-09-30T01:33:03.299303+0000","last_fullsized":"2023-09-30T01:33:03.299303+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T01:33:02.252599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T01:33:02.252599+0000","last_clean_scrub_stamp":"2023-09-30T01:33:02.252599+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T06:00:29.060679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":1057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":6}],"osd_stats":[{"osd":5,"up_from":11,"seq":47244640259,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5128,"kb_used_data":128,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":94366712,"statfs":{"total":96636764160,"available":96631513088,"internally_reserved":0,"allocated":131072,"data_stored":37781,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60399999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.90800000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82599999999999996}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.446}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0389999999999999}]}]},{"osd":4,"up_from":11,"seq":47244640259,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5128,"kb_used_data":128,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":94366712,"statfs":{"total":96636764160,"available":96631513088,"internally_reserved":0,"allocated":131072,"data_stored":37781,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69299999999999995}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82399999999999995}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1950000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.754}]}]},{"osd":3,"up_from":11,"seq":47244640260,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6096,"kb_used_data":648,"kb_used_omap":0,"kb_used_meta":5440,"kb_avail":94365744,"statfs":{"total":96636764160,"available":96630521856,"internally_reserved":0,"allocated":663552,"data_stored":566709,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5570560},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54300000000000004}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83099999999999996}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.256}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51300000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.85199999999999998}]}]},{"osd":2,"up_from":11,"seq":47244640261,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5128,"kb_used_data":128,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":94366712,"statfs":{"total":96636764160,"available":96631513088,"internally_reserved":0,"allocated":131072,"data_stored":37781,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52500000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86299999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.89900000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73299999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76200000000000001}]}]},{"osd":1,"up_from":11,"seq":47244640260,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6288,"kb_used_data":588,"kb_used_omap":0,"kb_used_meta":5376,"kb_avail":94365552,"statfs":{"total":96636764160,"available":96630325248,"internally_reserved":0,"allocated":602112,"data_stored":505269,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5505024},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45300000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55000000000000004}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78000000000000003}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.98199999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0880000000000001}]}]},{"osd":0,"up_from":11,"seq":47244640261,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5128,"kb_used_data":128,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":94366712,"statfs":{"total":96636764160,"available":96631513088,"internally_reserved":0,"allocated":131072,"data_stored":37781,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77700000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60399999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1319999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2609999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3200000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-09-30T01:33:07.914 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-09-30T01:33:08.012 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.009+0000 7fd583b49700 10 monclient: get_monmap_and_config 2023-09-30T01:33:08.012 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.009+0000 7fd583b49700 10 monclient: build_initial_monmap 2023-09-30T01:33:08.012 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.009+0000 7fd583b49700 10 monclient: monmap: 2023-09-30T01:33:08.012 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2023-09-30T01:33:08.012 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:33:08.012 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:33:08.012373+0000 2023-09-30T01:33:08.012 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:33:08.012373+0000 2023-09-30T01:33:08.012 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2023-09-30T01:33:08.013 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:33:08.013 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a-legacy 2023-09-30T01:33:08.013 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.107:6789/0 mon.noname-f-legacy 2023-09-30T01:33:08.013 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.noname-b 2023-09-30T01:33:08.013 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.noname-g 2023-09-30T01:33:08.013 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.noname-c 2023-09-30T01:33:08.013 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.noname-h 2023-09-30T01:33:08.013 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.noname-d 2023-09-30T01:33:08.013 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.noname-i 2023-09-30T01:33:08.013 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.noname-e 2023-09-30T01:33:08.013 INFO:teuthology.orchestra.run.smithi088.stderr:9: v2:172.21.15.88:3300/0 mon.noname-a 2023-09-30T01:33:08.013 INFO:teuthology.orchestra.run.smithi088.stderr:10: v2:172.21.15.107:3300/0 mon.noname-f 2023-09-30T01:33:08.014 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:33:08.021 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.009+0000 7fd583b49700 1 Processor -- start 2023-09-30T01:33:08.021 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.009+0000 7fd583b49700 1 -- start start 2023-09-30T01:33:08.022 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.009+0000 7fd583b49700 10 monclient: init 2023-09-30T01:33:08.022 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.009+0000 7fd583b49700 10 monclient: _reopen_session rank -1 2023-09-30T01:33:08.022 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.009+0000 7fd583b49700 10 monclient: _add_conns ranks=[0,1,7,8,6,5,4,3,9,2,10] 2023-09-30T01:33:08.022 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.009+0000 7fd583b49700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fd57c0a3270 addr v1:172.21.15.88:6789/0 2023-09-30T01:33:08.022 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.009+0000 7fd583b49700 10 monclient(hunting): picked mon.noname-f-legacy con 0x7fd57c0a4b00 addr v1:172.21.15.107:6789/0 2023-09-30T01:33:08.022 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.009+0000 7fd583b49700 1 --2- >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7fd57c0a0870 0x7fd57c13be80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:33:08.022 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.009+0000 7fd583b49700 10 monclient(hunting): picked mon.noname-i con 0x7fd57c0a0870 addr [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] 2023-09-30T01:33:08.022 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.009+0000 7fd583b49700 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd57c05ecb0 con 0x7fd57c0a3270 2023-09-30T01:33:08.022 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.009+0000 7fd583b49700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd57c098610 con 0x7fd57c0a4b00 2023-09-30T01:33:08.022 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.009+0000 7fd583b49700 10 monclient(hunting): start opening mon connection 2023-09-30T01:33:08.022 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.009+0000 7fd583b49700 1 -- --> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] -- mon_getmap magic: 0 v1 -- 0x7fd57c0a4ed0 con 0x7fd57c0a0870 2023-09-30T01:33:08.022 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.009+0000 7fd583b49700 10 monclient(hunting): _renew_subs 2023-09-30T01:33:08.023 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.009+0000 7fd583b49700 10 monclient(hunting): authenticate will time out at 860.050141s 2023-09-30T01:33:08.023 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.009+0000 7fd5818e5700 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7fd57c0a3270 0x7fd57c0a0160 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:54160/0 (socket says 172.21.15.88:54160) 2023-09-30T01:33:08.023 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.009+0000 7fd5818e5700 1 -- 172.21.15.88:0/1237071443 learned_addr learned my addr 172.21.15.88:0/1237071443 (peer_addr_for_me v1:172.21.15.88:0/0) 2023-09-30T01:33:08.023 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.009+0000 7fd5820e6700 1 --2- 172.21.15.88:0/1237071443 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7fd57c0a0870 0x7fd57c13be80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:33:08.023 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.009+0000 7fd5820e6700 10 monclient(hunting): get_auth_request con 0x7fd57c0a0870 auth_method 0 2023-09-30T01:33:08.023 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.009+0000 7fd5820e6700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:33:08.023 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.009+0000 7fd5820e6700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:33:08.023 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.009+0000 7fd5820e6700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:33:08.023 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.009+0000 7fd5808e3700 1 -- 172.21.15.88:0/1237071443 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2154001735 0 0) 0x7fd57c05ecb0 con 0x7fd57c0a3270 2023-09-30T01:33:08.023 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.009+0000 7fd5808e3700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:33:08.023 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.009+0000 7fd5808e3700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:33:08.023 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.009+0000 7fd5808e3700 10 monclient(hunting): my global_id is 4428 2023-09-30T01:33:08.024 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.009+0000 7fd5808e3700 1 -- 172.21.15.88:0/1237071443 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd568003570 con 0x7fd57c0a3270 2023-09-30T01:33:08.024 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.009+0000 7fd5820e6700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:33:08.024 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.009+0000 7fd5820e6700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:33:08.024 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.009+0000 7fd5820e6700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:33:08.024 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.009+0000 7fd5808e3700 1 -- 172.21.15.88:0/1237071443 <== mon.1 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3221593837 0 0) 0x7fd57c098610 con 0x7fd57c0a4b00 2023-09-30T01:33:08.024 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.009+0000 7fd5808e3700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:33:08.024 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.009+0000 7fd5808e3700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:33:08.024 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.009+0000 7fd5808e3700 10 monclient(hunting): my global_id is 4519 2023-09-30T01:33:08.024 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.009+0000 7fd5808e3700 1 -- 172.21.15.88:0/1237071443 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd57c05ecb0 con 0x7fd57c0a4b00 2023-09-30T01:33:08.024 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.009+0000 7fd5808e3700 1 -- 172.21.15.88:0/1237071443 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1069269600 0 0) 0x7fd568003570 con 0x7fd57c0a3270 2023-09-30T01:33:08.024 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.009+0000 7fd5808e3700 1 -- 172.21.15.88:0/1237071443 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7fd57c0a0870 msgr2=0x7fd57c13be80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:33:08.024 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.009+0000 7fd5808e3700 1 --2- 172.21.15.88:0/1237071443 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7fd57c0a0870 0x7fd57c13be80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:08.025 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.009+0000 7fd5808e3700 1 -- 172.21.15.88:0/1237071443 >> v1:172.21.15.107:6789/0 conn(0x7fd57c0a4b00 legacy=0x7fd57c138750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:33:08.025 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.009+0000 7fd5808e3700 10 monclient: _finish_hunting 0 2023-09-30T01:33:08.025 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.009+0000 7fd5808e3700 1 monclient: found mon.noname-a-legacy 2023-09-30T01:33:08.025 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.009+0000 7fd5808e3700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:33:08.025 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.009+0000 7fd5808e3700 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.88:6789/0 2023-09-30T01:33:08.025 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.009+0000 7fd5808e3700 1 -- 172.21.15.88:0/1237071443 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd57c0a0cb0 con 0x7fd57c0a3270 2023-09-30T01:33:08.025 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.009+0000 7fd5808e3700 10 monclient: _finish_auth 0 2023-09-30T01:33:08.025 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.009+0000 7fd5808e3700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:33:08.025 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.009+0000 7fd583b49700 5 monclient: authenticate success, global_id 4428 2023-09-30T01:33:08.025 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.009+0000 7fd583b49700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:33:08.025 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.009+0000 7fd5808e3700 1 -- 172.21.15.88:0/1237071443 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (unknown 176869241 0 0) 0x7fd5780018f0 con 0x7fd57c0a3270 2023-09-30T01:33:08.026 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.009+0000 7fd5808e3700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:33:08.026 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.009+0000 7fd5808e3700 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2023-09-30T01:33:08.026 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.009+0000 7fd5808e3700 10 monclient: dump: 2023-09-30T01:33:08.026 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:33:08.026 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:33:08.026 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:08.026 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:08.026 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:33:08.026 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:33:08.026 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:33:08.027 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:33:08.027 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:33:08.027 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:33:08.027 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:33:08.027 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:33:08.027 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:33:08.027 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:33:08.027 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:33:08.027 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:33:08.028 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.009+0000 7fd5808e3700 1 monclient: mon.a has (v2) addrs [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] but i'm connected to v1:172.21.15.88:6789/0, reconnecting 2023-09-30T01:33:08.028 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.009+0000 7fd5808e3700 10 monclient: _reopen_session rank -1 2023-09-30T01:33:08.028 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.009+0000 7fd5808e3700 1 -- 172.21.15.88:0/1237071443 >> v1:172.21.15.88:6789/0 conn(0x7fd57c0a3270 legacy=0x7fd57c0a0160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:33:08.028 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.009+0000 7fd5808e3700 10 monclient: _add_conns ranks=[0,5,6,8,2,1,3,7,4] 2023-09-30T01:33:08.028 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.009+0000 7fd5808e3700 1 --2- 172.21.15.88:0/1237071443 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd568004810 0x7fd568004be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:33:08.028 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.009+0000 7fd5808e3700 10 monclient(hunting): picked mon.a con 0x7fd568004810 addr [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] 2023-09-30T01:33:08.028 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.009+0000 7fd5808e3700 1 --2- 172.21.15.88:0/1237071443 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7fd568005120 0x7fd568009a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:33:08.029 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.009+0000 7fd5808e3700 10 monclient(hunting): picked mon.h con 0x7fd568005120 addr [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] 2023-09-30T01:33:08.029 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.009+0000 7fd5808e3700 1 --2- 172.21.15.88:0/1237071443 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7fd568009f90 0x7fd56800c370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:33:08.029 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.009+0000 7fd5808e3700 10 monclient(hunting): picked mon.d con 0x7fd568009f90 addr [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] 2023-09-30T01:33:08.029 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.009+0000 7fd5808e3700 10 monclient(hunting): start opening mon connection 2023-09-30T01:33:08.029 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.009+0000 7fd5808e3700 1 -- 172.21.15.88:0/1237071443 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd57c0a4ed0 con 0x7fd568004810 2023-09-30T01:33:08.030 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.009+0000 7fd5818e5700 1 --2- 172.21.15.88:0/1237071443 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd568004810 0x7fd568004be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:33:08.030 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.009+0000 7fd5808e3700 10 monclient(hunting): start opening mon connection 2023-09-30T01:33:08.030 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.009+0000 7fd5808e3700 1 -- 172.21.15.88:0/1237071443 --> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] -- mon_getmap magic: 0 v1 -- 0x7fd56800c9c0 con 0x7fd568005120 2023-09-30T01:33:08.030 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.009+0000 7fd5808e3700 10 monclient(hunting): start opening mon connection 2023-09-30T01:33:08.030 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.009+0000 7fd5808e3700 1 -- 172.21.15.88:0/1237071443 --> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] -- mon_getmap magic: 0 v1 -- 0x7fd56800cb30 con 0x7fd568009f90 2023-09-30T01:33:08.030 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.009+0000 7fd5808e3700 10 monclient(hunting): _renew_subs 2023-09-30T01:33:08.030 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.009+0000 7fd5808e3700 10 monclient(hunting): _finish_auth 0 2023-09-30T01:33:08.030 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.009+0000 7fd5818e5700 10 monclient(hunting): get_auth_request con 0x7fd568004810 auth_method 0 2023-09-30T01:33:08.030 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.009+0000 7fd5820e6700 1 --2- 172.21.15.88:0/1237071443 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7fd57c0a0870 0x7fd57c13be80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:33:08.030 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.009+0000 7fd5818e5700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:33:08.031 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.009+0000 7fd5818e5700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:33:08.031 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.009+0000 7fd5818e5700 10 monclient(hunting): _init_auth already have auth, reseting 2023-09-30T01:33:08.031 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.009+0000 7fd583b49700 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2023-09-30T01:33:08.031 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.009+0000 7fd5818e5700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:33:08.031 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.009+0000 7fd5818e5700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:33:08.031 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.013+0000 7fd5818e5700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-09-30T01:33:08.031 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.013+0000 7fd5810e4700 1 --2- 172.21.15.88:0/1237071443 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7fd568005120 0x7fd568009a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:33:08.031 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.013+0000 7fd5810e4700 10 monclient(hunting): get_auth_request con 0x7fd568005120 auth_method 0 2023-09-30T01:33:08.031 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.013+0000 7fd5810e4700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:33:08.032 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.013+0000 7fd5810e4700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:33:08.032 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.013+0000 7fd5810e4700 10 monclient(hunting): _init_auth already have auth, reseting 2023-09-30T01:33:08.032 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.013+0000 7fd5820e6700 1 --2- 172.21.15.88:0/1237071443 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7fd568009f90 0x7fd56800c370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:33:08.032 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.013+0000 7fd5818e5700 10 monclient(hunting): handle_auth_done global_id 4428 payload 293 2023-09-30T01:33:08.032 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.013+0000 7fd5818e5700 1 -- 172.21.15.88:0/1237071443 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7fd568009f90 msgr2=0x7fd56800c370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:33:08.032 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.013+0000 7fd5818e5700 1 --2- 172.21.15.88:0/1237071443 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7fd568009f90 0x7fd56800c370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:08.032 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.013+0000 7fd5818e5700 1 -- 172.21.15.88:0/1237071443 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7fd568005120 msgr2=0x7fd568009a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:33:08.033 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.013+0000 7fd5818e5700 1 --2- 172.21.15.88:0/1237071443 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7fd568005120 0x7fd568009a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:08.033 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.013+0000 7fd5818e5700 10 monclient: _finish_hunting 0 2023-09-30T01:33:08.033 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.013+0000 7fd5818e5700 1 monclient: found mon.a 2023-09-30T01:33:08.033 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.013+0000 7fd5818e5700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:33:08.033 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.013+0000 7fd5818e5700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.88:3300/0 2023-09-30T01:33:08.033 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.013+0000 7fd5818e5700 1 -- 172.21.15.88:0/1237071443 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd568003570 con 0x7fd568004810 2023-09-30T01:33:08.033 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.013+0000 7fd5818e5700 10 monclient: _finish_auth 0 2023-09-30T01:33:08.033 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.013+0000 7fd5818e5700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:33:08.033 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.013+0000 7fd5820e6700 10 monclient: get_auth_request con 0x7fd568009f90 auth_method 0 2023-09-30T01:33:08.033 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.013+0000 7fd5820e6700 1 --2- 172.21.15.88:0/1237071443 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7fd568009f90 0x7fd56800c370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-09-30T01:33:08.033 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.013+0000 7fd5810e4700 1 --2- 172.21.15.88:0/1237071443 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7fd568005120 0x7fd568009a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-30T01:33:08.034 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.013+0000 7fd5820e6700 1 -- 172.21.15.88:0/1237071443 reap_dead start 2023-09-30T01:33:08.034 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.013+0000 7fd5818e5700 1 --2- 172.21.15.88:0/1237071443 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd568004810 0x7fd568004be0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fd578008e80 tx=0x7fd578001470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6db2333b65bbfa77 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:33:08.034 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.013+0000 7fd5808e3700 1 -- 172.21.15.88:0/1237071443 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7fd578005da0 con 0x7fd568004810 2023-09-30T01:33:08.034 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.013+0000 7fd5808e3700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:33:08.034 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.013+0000 7fd5808e3700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:33:08.034 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.013+0000 7fd5808e3700 10 monclient: dump: 2023-09-30T01:33:08.034 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:33:08.034 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:33:08.034 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:08.035 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:08.035 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:33:08.035 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:33:08.035 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:33:08.035 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:33:08.035 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:33:08.035 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:33:08.035 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:33:08.035 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:33:08.036 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:33:08.036 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:33:08.036 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:33:08.036 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:33:08.036 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.013+0000 7fd583b49700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:33:08.036 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.013+0000 7fd5808e3700 1 -- 172.21.15.88:0/1237071443 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd578005620 con 0x7fd568004810 2023-09-30T01:33:08.036 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.013+0000 7fd5808e3700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:33:08.036 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.013+0000 7fd583b49700 10 monclient: get_monmap_and_config success 2023-09-30T01:33:08.036 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.013+0000 7fd583b49700 4 set_mon_vals no callback set 2023-09-30T01:33:08.037 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.013+0000 7fd5808e3700 1 -- 172.21.15.88:0/1237071443 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7fd578005790 con 0x7fd568004810 2023-09-30T01:33:08.037 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.013+0000 7fd5808e3700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:33:08.037 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.013+0000 7fd5808e3700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:33:08.037 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.013+0000 7fd5808e3700 10 monclient: dump: 2023-09-30T01:33:08.037 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:33:08.037 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:33:08.037 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:08.037 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:08.037 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:33:08.038 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:33:08.038 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:33:08.038 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:33:08.038 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:33:08.038 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:33:08.038 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:33:08.038 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:33:08.038 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:33:08.038 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:33:08.038 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:33:08.038 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:33:08.039 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.013+0000 7fd583b49700 10 monclient: shutdown 2023-09-30T01:33:08.039 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.013+0000 7fd583b49700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:33:08.039 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.013+0000 7fd583b49700 1 -- 172.21.15.88:0/1237071443 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd568004810 msgr2=0x7fd568004be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:33:08.039 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.013+0000 7fd583b49700 1 --2- 172.21.15.88:0/1237071443 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd568004810 0x7fd568004be0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fd578008e80 tx=0x7fd578001470 comp rx=0 tx=0).stop 2023-09-30T01:33:08.039 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.013+0000 7fd583b49700 1 -- 172.21.15.88:0/1237071443 shutdown_connections 2023-09-30T01:33:08.039 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.013+0000 7fd583b49700 1 --2- 172.21.15.88:0/1237071443 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd568004810 0x7fd568004be0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:08.039 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.013+0000 7fd583b49700 1 -- 172.21.15.88:0/1237071443 >> 172.21.15.88:0/1237071443 conn(0x7fd57c0a67f0 msgr2=0x7fd57c0a6bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:33:08.039 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.013+0000 7fd583b49700 1 -- 172.21.15.88:0/1237071443 shutdown_connections 2023-09-30T01:33:08.039 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.013+0000 7fd583b49700 1 -- 172.21.15.88:0/1237071443 wait complete. 2023-09-30T01:33:08.039 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.013+0000 7fd583b49700 10 monclient: build_initial_monmap 2023-09-30T01:33:08.039 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.013+0000 7fd583b49700 10 monclient: monmap: 2023-09-30T01:33:08.040 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2023-09-30T01:33:08.040 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:33:08.040 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 0.000000 2023-09-30T01:33:08.040 INFO:teuthology.orchestra.run.smithi088.stderr:created 0.000000 2023-09-30T01:33:08.040 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2023-09-30T01:33:08.040 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 1 2023-09-30T01:33:08.040 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.noname-a 2023-09-30T01:33:08.040 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.noname-f 2023-09-30T01:33:08.040 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.noname-b 2023-09-30T01:33:08.040 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.noname-g 2023-09-30T01:33:08.040 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.noname-c 2023-09-30T01:33:08.040 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.noname-h 2023-09-30T01:33:08.041 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.noname-d 2023-09-30T01:33:08.041 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.noname-i 2023-09-30T01:33:08.041 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.noname-e 2023-09-30T01:33:08.041 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:33:08.041 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.013+0000 7fd583b49700 1 Processor -- start 2023-09-30T01:33:08.041 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.013+0000 7fd583b49700 1 -- start start 2023-09-30T01:33:08.041 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.013+0000 7fd583b49700 10 monclient: init 2023-09-30T01:33:08.041 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.013+0000 7fd583b49700 10 monclient: _reopen_session rank -1 2023-09-30T01:33:08.041 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.013+0000 7fd583b49700 10 monclient: _add_conns ranks=[1,7,0,2,6,3,5,8,4] 2023-09-30T01:33:08.041 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.013+0000 7fd583b49700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd568004810 0x7fd57c12fba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:33:08.042 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.013+0000 7fd583b49700 10 monclient(hunting): picked mon.noname-f con 0x7fd568004810 addr [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] 2023-09-30T01:33:08.042 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.013+0000 7fd583b49700 1 --2- >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7fd57c1300e0 0x7fd57c09ca60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:33:08.042 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.013+0000 7fd583b49700 10 monclient(hunting): picked mon.noname-i con 0x7fd57c1300e0 addr [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] 2023-09-30T01:33:08.042 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.013+0000 7fd583b49700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd57c1304b0 0x7fd57c09cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:33:08.042 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.013+0000 7fd583b49700 10 monclient(hunting): picked mon.noname-a con 0x7fd57c1304b0 addr [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] 2023-09-30T01:33:08.042 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.013+0000 7fd583b49700 10 monclient(hunting): start opening mon connection 2023-09-30T01:33:08.042 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.013+0000 7fd583b49700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd57c11d680 con 0x7fd57c1304b0 2023-09-30T01:33:08.042 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.013+0000 7fd583b49700 10 monclient(hunting): start opening mon connection 2023-09-30T01:33:08.042 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.013+0000 7fd583b49700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd57c11d8d0 con 0x7fd568004810 2023-09-30T01:33:08.042 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.013+0000 7fd583b49700 10 monclient(hunting): start opening mon connection 2023-09-30T01:33:08.043 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.013+0000 7fd583b49700 1 -- --> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] -- mon_getmap magic: 0 v1 -- 0x7fd57c0a0e80 con 0x7fd57c1300e0 2023-09-30T01:33:08.043 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.013+0000 7fd583b49700 10 monclient(hunting): _renew_subs 2023-09-30T01:33:08.043 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.013+0000 7fd583b49700 10 monclient(hunting): authenticate will time out at 860.055435s 2023-09-30T01:33:08.043 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.013+0000 7fd5820e6700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd57c1304b0 0x7fd57c09cfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:33:08.043 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.013+0000 7fd5820e6700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd57c1304b0 0x7fd57c09cfa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:57926/0 (socket says 172.21.15.88:57926) 2023-09-30T01:33:08.043 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.013+0000 7fd5820e6700 1 -- 172.21.15.88:0/3769003540 learned_addr learned my addr 172.21.15.88:0/3769003540 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-09-30T01:33:08.044 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.013+0000 7fd5820e6700 10 monclient(hunting): get_auth_request con 0x7fd57c1304b0 auth_method 0 2023-09-30T01:33:08.044 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.013+0000 7fd5820e6700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:33:08.044 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.013+0000 7fd5820e6700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:33:08.044 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.013+0000 7fd5820e6700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:33:08.044 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.013+0000 7fd5820e6700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:33:08.044 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.013+0000 7fd5820e6700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:33:08.044 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.013+0000 7fd5820e6700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:33:08.044 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.013+0000 7fd5818e5700 1 --2- 172.21.15.88:0/3769003540 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd568004810 0x7fd57c12fba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:33:08.044 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.013+0000 7fd5820e6700 10 monclient(hunting): handle_auth_done global_id 4437 payload 824 2023-09-30T01:33:08.044 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.013+0000 7fd5810e4700 1 --2- 172.21.15.88:0/3769003540 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7fd57c1300e0 0x7fd57c09ca60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:33:08.044 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.013+0000 7fd5820e6700 1 -- 172.21.15.88:0/3769003540 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7fd57c1300e0 msgr2=0x7fd57c09ca60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:33:08.045 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.013+0000 7fd5820e6700 1 --2- 172.21.15.88:0/3769003540 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7fd57c1300e0 0x7fd57c09ca60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:08.045 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.013+0000 7fd5820e6700 1 -- 172.21.15.88:0/3769003540 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd568004810 msgr2=0x7fd57c12fba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:33:08.045 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.013+0000 7fd5820e6700 1 --2- 172.21.15.88:0/3769003540 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd568004810 0x7fd57c12fba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:08.045 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.013+0000 7fd5820e6700 10 monclient: _finish_hunting 0 2023-09-30T01:33:08.045 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.013+0000 7fd5820e6700 1 monclient: found mon.noname-a 2023-09-30T01:33:08.045 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.013+0000 7fd5820e6700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:33:08.045 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.013+0000 7fd5820e6700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.88:3300/0 2023-09-30T01:33:08.045 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.013+0000 7fd5820e6700 1 -- 172.21.15.88:0/3769003540 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd57c097c70 con 0x7fd57c1304b0 2023-09-30T01:33:08.045 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.013+0000 7fd5810e4700 10 monclient: get_auth_request con 0x7fd57c1300e0 auth_method 0 2023-09-30T01:33:08.045 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.013+0000 7fd5818e5700 10 monclient: get_auth_request con 0x7fd568004810 auth_method 0 2023-09-30T01:33:08.045 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.013+0000 7fd5810e4700 1 --2- 172.21.15.88:0/3769003540 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7fd57c1300e0 0x7fd57c09ca60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-09-30T01:33:08.045 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.013+0000 7fd5818e5700 1 --2- 172.21.15.88:0/3769003540 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd568004810 0x7fd57c12fba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-09-30T01:33:08.046 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.017+0000 7fd5820e6700 1 --2- 172.21.15.88:0/3769003540 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd57c1304b0 0x7fd57c09cfa0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fd57c0a3d20 tx=0x7fd57400b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:33:08.046 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.017+0000 7fd571ffb700 1 -- 172.21.15.88:0/3769003540 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7fd57401a070 con 0x7fd57c1304b0 2023-09-30T01:33:08.046 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.017+0000 7fd571ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:33:08.046 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.017+0000 7fd571ffb700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T01:33:08.046 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.017+0000 7fd571ffb700 10 monclient: dump: 2023-09-30T01:33:08.046 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:33:08.046 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:33:08.046 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:08.046 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:08.046 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:33:08.046 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:33:08.047 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:33:08.047 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:33:08.047 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:33:08.047 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:33:08.047 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:33:08.047 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:33:08.047 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:33:08.047 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:33:08.047 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:33:08.047 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:33:08.047 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.017+0000 7fd571ffb700 10 monclient: _finish_auth 0 2023-09-30T01:33:08.048 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.017+0000 7fd571ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:33:08.048 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.017+0000 7fd571ffb700 1 -- 172.21.15.88:0/3769003540 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd574007e70 con 0x7fd57c1304b0 2023-09-30T01:33:08.048 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.017+0000 7fd583b49700 5 monclient: authenticate success, global_id 4437 2023-09-30T01:33:08.048 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.017+0000 7fd571ffb700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:33:08.048 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.017+0000 7fd583b49700 10 monclient: _renew_subs 2023-09-30T01:33:08.048 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.017+0000 7fd571ffb700 1 -- 172.21.15.88:0/3769003540 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7fd574016d20 con 0x7fd57c1304b0 2023-09-30T01:33:08.048 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.017+0000 7fd583b49700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.88:3300/0 2023-09-30T01:33:08.048 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.017+0000 7fd583b49700 1 -- 172.21.15.88:0/3769003540 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd57c09eb80 con 0x7fd57c1304b0 2023-09-30T01:33:08.048 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.017+0000 7fd573fff700 4 set_mon_vals no callback set 2023-09-30T01:33:08.048 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.017+0000 7fd571ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:33:08.048 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.017+0000 7fd571ffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:33:08.049 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.017+0000 7fd571ffb700 10 monclient: dump: 2023-09-30T01:33:08.049 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:33:08.049 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:33:08.049 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:08.049 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:08.049 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:33:08.049 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:33:08.049 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:33:08.049 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:33:08.050 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:33:08.050 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:33:08.050 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:33:08.050 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:33:08.050 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:33:08.050 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:33:08.050 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:33:08.050 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:33:08.050 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.017+0000 7fd583b49700 10 monclient: _renew_subs 2023-09-30T01:33:08.050 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.017+0000 7fd583b49700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.88:3300/0 2023-09-30T01:33:08.051 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.017+0000 7fd583b49700 1 -- 172.21.15.88:0/3769003540 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd57c09ef80 con 0x7fd57c1304b0 2023-09-30T01:33:08.051 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.017+0000 7fd571ffb700 1 -- 172.21.15.88:0/3769003540 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 40144+0+0 (secure 0 0 0) 0x7fd574003020 con 0x7fd57c1304b0 2023-09-30T01:33:08.051 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.017+0000 7fd583b49700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:33:08.051 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.017+0000 7fd571ffb700 1 --2- 172.21.15.88:0/3769003540 >> v2:172.21.15.107:6812/29926 conn(0x7fd560033360 0x7fd560035810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:33:08.051 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.017+0000 7fd583b49700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:33:08.051 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.017+0000 7fd583b49700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.88:3300/0 2023-09-30T01:33:08.051 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.017+0000 7fd571ffb700 1 -- 172.21.15.88:0/3769003540 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 3502+0+0 (secure 0 0 0) 0x7fd574043dd0 con 0x7fd57c1304b0 2023-09-30T01:33:08.051 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.017+0000 7fd583b49700 1 -- 172.21.15.88:0/3769003540 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd57c071960 con 0x7fd57c1304b0 2023-09-30T01:33:08.052 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.017+0000 7fd5818e5700 1 --2- 172.21.15.88:0/3769003540 >> v2:172.21.15.107:6812/29926 conn(0x7fd560033360 0x7fd560035810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:33:08.052 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.017+0000 7fd5818e5700 10 monclient: get_auth_request con 0x7fd560033360 auth_method 0 2023-09-30T01:33:08.052 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.017+0000 7fd5818e5700 1 --2- 172.21.15.88:0/3769003540 >> v2:172.21.15.107:6812/29926 conn(0x7fd560033360 0x7fd560035810 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fd578008d50 tx=0x7fd578016000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:33:08.053 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.025+0000 7fd571ffb700 1 -- 172.21.15.88:0/3769003540 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fd574016600 con 0x7fd57c1304b0 2023-09-30T01:33:08.053 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.025+0000 7fd571ffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:33:08.053 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.025+0000 7fd571ffb700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:33:08.173 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.169+0000 7fd583b49700 1 -- 172.21.15.88:0/3769003540 --> v2:172.21.15.107:6812/29926 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fd57c11da10 con 0x7fd560033360 2023-09-30T01:33:08.176 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.173+0000 7fd571ffb700 1 -- 172.21.15.88:0/3769003540 <== mgr.4108 v2:172.21.15.107:6812/29926 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+40390 (secure 0 0 0) 0x7fd57c11da10 con 0x7fd560033360 2023-09-30T01:33:08.177 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-09-30T01:33:08.178 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.173+0000 7fd583b49700 1 -- 172.21.15.88:0/3769003540 >> v2:172.21.15.107:6812/29926 conn(0x7fd560033360 msgr2=0x7fd560035810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:33:08.178 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.173+0000 7fd583b49700 1 --2- 172.21.15.88:0/3769003540 >> v2:172.21.15.107:6812/29926 conn(0x7fd560033360 0x7fd560035810 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fd578008d50 tx=0x7fd578016000 comp rx=0 tx=0).stop 2023-09-30T01:33:08.178 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.173+0000 7fd583b49700 10 monclient: shutdown 2023-09-30T01:33:08.179 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.173+0000 7fd583b49700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:33:08.179 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.173+0000 7fd583b49700 1 -- 172.21.15.88:0/3769003540 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd57c1304b0 msgr2=0x7fd57c09cfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:33:08.179 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.173+0000 7fd583b49700 1 --2- 172.21.15.88:0/3769003540 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd57c1304b0 0x7fd57c09cfa0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fd57c0a3d20 tx=0x7fd57400b040 comp rx=0 tx=0).stop 2023-09-30T01:33:08.179 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.173+0000 7fd583b49700 1 -- 172.21.15.88:0/3769003540 shutdown_connections 2023-09-30T01:33:08.179 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.173+0000 7fd583b49700 1 --2- 172.21.15.88:0/3769003540 >> v2:172.21.15.107:6812/29926 conn(0x7fd560033360 0x7fd560035810 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:08.179 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.173+0000 7fd583b49700 1 --2- 172.21.15.88:0/3769003540 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd57c1304b0 0x7fd57c09cfa0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:08.179 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.173+0000 7fd583b49700 1 --2- 172.21.15.88:0/3769003540 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7fd57c1300e0 0x7fd57c09ca60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:08.179 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.173+0000 7fd583b49700 1 --2- 172.21.15.88:0/3769003540 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd568004810 0x7fd57c12fba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:08.179 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.173+0000 7fd583b49700 1 -- 172.21.15.88:0/3769003540 >> 172.21.15.88:0/3769003540 conn(0x7fd57c0a67f0 msgr2=0x7fd57c0990b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:33:08.179 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.173+0000 7fd583b49700 1 -- 172.21.15.88:0/3769003540 shutdown_connections 2023-09-30T01:33:08.179 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.173+0000 7fd583b49700 1 -- 172.21.15.88:0/3769003540 wait complete. 2023-09-30T01:33:08.180 INFO:teuthology.orchestra.run.smithi088.stderr:dumped all 2023-09-30T01:33:08.190 INFO:teuthology.orchestra.run.smithi088.stdout:{"pg_ready":true,"pg_map":{"version":14,"stamp":"2023-09-30T01:33:07.653762+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":11,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":32896,"kb_used_data":1748,"kb_used_omap":0,"kb_used_meta":30784,"kb_avail":566198144,"statfs":{"total":579820584960,"available":579786899456,"internally_reserved":0,"allocated":1789952,"data_stored":1223102,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":31522816},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":9,"num_write":14,"num_write_kb":262,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"2.000676"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-09-30T01:33:03.296691+0000","last_change":"2023-09-30T01:33:03.292315+0000","last_active":"2023-09-30T01:33:03.296691+0000","last_peered":"2023-09-30T01:33:03.296691+0000","last_clean":"2023-09-30T01:33:03.296691+0000","last_became_active":"2023-09-30T01:33:03.291889+0000","last_became_peered":"2023-09-30T01:33:03.291889+0000","last_unstale":"2023-09-30T01:33:03.296691+0000","last_undegraded":"2023-09-30T01:33:03.296691+0000","last_fullsized":"2023-09-30T01:33:03.296691+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T01:33:02.252599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T01:33:02.252599+0000","last_clean_scrub_stamp":"2023-09-30T01:33:02.252599+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T11:44:00.574379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-09-30T01:33:03.297822+0000","last_change":"2023-09-30T01:33:03.295081+0000","last_active":"2023-09-30T01:33:03.297822+0000","last_peered":"2023-09-30T01:33:03.297822+0000","last_clean":"2023-09-30T01:33:03.297822+0000","last_became_active":"2023-09-30T01:33:03.294535+0000","last_became_peered":"2023-09-30T01:33:03.294535+0000","last_unstale":"2023-09-30T01:33:03.297822+0000","last_undegraded":"2023-09-30T01:33:03.297822+0000","last_fullsized":"2023-09-30T01:33:03.297822+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T01:33:02.252599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T01:33:02.252599+0000","last_clean_scrub_stamp":"2023-09-30T01:33:02.252599+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T05:49:49.642723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-09-30T01:33:03.300117+0000","last_change":"2023-09-30T01:33:03.297392+0000","last_active":"2023-09-30T01:33:03.300117+0000","last_peered":"2023-09-30T01:33:03.300117+0000","last_clean":"2023-09-30T01:33:03.300117+0000","last_became_active":"2023-09-30T01:33:03.296959+0000","last_became_peered":"2023-09-30T01:33:03.296959+0000","last_unstale":"2023-09-30T01:33:03.300117+0000","last_undegraded":"2023-09-30T01:33:03.300117+0000","last_fullsized":"2023-09-30T01:33:03.300117+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T01:33:02.252599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T01:33:02.252599+0000","last_clean_scrub_stamp":"2023-09-30T01:33:02.252599+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T11:57:13.112607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-09-30T01:33:03.296932+0000","last_change":"2023-09-30T01:33:03.295478+0000","last_active":"2023-09-30T01:33:03.296932+0000","last_peered":"2023-09-30T01:33:03.296932+0000","last_clean":"2023-09-30T01:33:03.296932+0000","last_became_active":"2023-09-30T01:33:03.295106+0000","last_became_peered":"2023-09-30T01:33:03.295106+0000","last_unstale":"2023-09-30T01:33:03.296932+0000","last_undegraded":"2023-09-30T01:33:03.296932+0000","last_fullsized":"2023-09-30T01:33:03.296932+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T01:33:02.252599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T01:33:02.252599+0000","last_clean_scrub_stamp":"2023-09-30T01:33:02.252599+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T07:34:36.123333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"15'117","reported_seq":159,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-09-30T01:33:05.391246+0000","last_change":"2023-09-30T01:33:03.296872+0000","last_active":"2023-09-30T01:33:05.391246+0000","last_peered":"2023-09-30T01:33:05.391246+0000","last_clean":"2023-09-30T01:33:05.391246+0000","last_became_active":"2023-09-30T01:33:03.295200+0000","last_became_peered":"2023-09-30T01:33:03.295200+0000","last_unstale":"2023-09-30T01:33:05.391246+0000","last_undegraded":"2023-09-30T01:33:05.391246+0000","last_fullsized":"2023-09-30T01:33:05.391246+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T01:33:02.252599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T01:33:02.252599+0000","last_clean_scrub_stamp":"2023-09-30T01:33:02.252599+0000","objects_scrubbed":0,"log_size":117,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T01:51:03.655484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-09-30T01:33:03.296258+0000","last_change":"2023-09-30T01:33:03.293385+0000","last_active":"2023-09-30T01:33:03.296258+0000","last_peered":"2023-09-30T01:33:03.296258+0000","last_clean":"2023-09-30T01:33:03.296258+0000","last_became_active":"2023-09-30T01:33:03.293004+0000","last_became_peered":"2023-09-30T01:33:03.293004+0000","last_unstale":"2023-09-30T01:33:03.296258+0000","last_undegraded":"2023-09-30T01:33:03.296258+0000","last_fullsized":"2023-09-30T01:33:03.296258+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T01:33:02.252599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T01:33:02.252599+0000","last_clean_scrub_stamp":"2023-09-30T01:33:02.252599+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T03:32:32.404638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-09-30T01:33:03.299431+0000","last_change":"2023-09-30T01:33:03.296945+0000","last_active":"2023-09-30T01:33:03.299431+0000","last_peered":"2023-09-30T01:33:03.299431+0000","last_clean":"2023-09-30T01:33:03.299431+0000","last_became_active":"2023-09-30T01:33:03.296465+0000","last_became_peered":"2023-09-30T01:33:03.296465+0000","last_unstale":"2023-09-30T01:33:03.299431+0000","last_undegraded":"2023-09-30T01:33:03.299431+0000","last_fullsized":"2023-09-30T01:33:03.299431+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T01:33:02.252599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T01:33:02.252599+0000","last_clean_scrub_stamp":"2023-09-30T01:33:02.252599+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T05:32:13.855128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-09-30T01:33:03.296569+0000","last_change":"2023-09-30T01:33:03.291527+0000","last_active":"2023-09-30T01:33:03.296569+0000","last_peered":"2023-09-30T01:33:03.296569+0000","last_clean":"2023-09-30T01:33:03.296569+0000","last_became_active":"2023-09-30T01:33:03.291084+0000","last_became_peered":"2023-09-30T01:33:03.291084+0000","last_unstale":"2023-09-30T01:33:03.296569+0000","last_undegraded":"2023-09-30T01:33:03.296569+0000","last_fullsized":"2023-09-30T01:33:03.296569+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T01:33:02.252599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T01:33:02.252599+0000","last_clean_scrub_stamp":"2023-09-30T01:33:02.252599+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T12:10:57.124090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-09-30T01:33:03.299303+0000","last_change":"2023-09-30T01:33:03.296313+0000","last_active":"2023-09-30T01:33:03.299303+0000","last_peered":"2023-09-30T01:33:03.299303+0000","last_clean":"2023-09-30T01:33:03.299303+0000","last_became_active":"2023-09-30T01:33:03.294604+0000","last_became_peered":"2023-09-30T01:33:03.294604+0000","last_unstale":"2023-09-30T01:33:03.299303+0000","last_undegraded":"2023-09-30T01:33:03.299303+0000","last_fullsized":"2023-09-30T01:33:03.299303+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T01:33:02.252599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T01:33:02.252599+0000","last_clean_scrub_stamp":"2023-09-30T01:33:02.252599+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T06:00:29.060679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":1057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":6}],"osd_stats":[{"osd":5,"up_from":11,"seq":47244640259,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5128,"kb_used_data":128,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":94366712,"statfs":{"total":96636764160,"available":96631513088,"internally_reserved":0,"allocated":131072,"data_stored":37781,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60399999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.90800000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82599999999999996}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.446}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0389999999999999}]}]},{"osd":4,"up_from":11,"seq":47244640259,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5128,"kb_used_data":128,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":94366712,"statfs":{"total":96636764160,"available":96631513088,"internally_reserved":0,"allocated":131072,"data_stored":37781,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69299999999999995}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82399999999999995}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1950000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.754}]}]},{"osd":3,"up_from":11,"seq":47244640260,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6096,"kb_used_data":648,"kb_used_omap":0,"kb_used_meta":5440,"kb_avail":94365744,"statfs":{"total":96636764160,"available":96630521856,"internally_reserved":0,"allocated":663552,"data_stored":566709,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5570560},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54300000000000004}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83099999999999996}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.256}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51300000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.85199999999999998}]}]},{"osd":2,"up_from":11,"seq":47244640261,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5128,"kb_used_data":128,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":94366712,"statfs":{"total":96636764160,"available":96631513088,"internally_reserved":0,"allocated":131072,"data_stored":37781,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52500000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86299999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.89900000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73299999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76200000000000001}]}]},{"osd":1,"up_from":11,"seq":47244640260,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6288,"kb_used_data":588,"kb_used_omap":0,"kb_used_meta":5376,"kb_avail":94365552,"statfs":{"total":96636764160,"available":96630325248,"internally_reserved":0,"allocated":602112,"data_stored":505269,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5505024},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45300000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55000000000000004}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78000000000000003}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.98199999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0880000000000001}]}]},{"osd":0,"up_from":11,"seq":47244640261,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5128,"kb_used_data":128,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":94366712,"statfs":{"total":96636764160,"available":96631513088,"internally_reserved":0,"allocated":131072,"data_stored":37781,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77700000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60399999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1319999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2609999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3200000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-09-30T01:33:08.191 INFO:tasks.ceph.ceph_manager.ceph:clean! 2023-09-30T01:33:08.192 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2023-09-30T01:33:08.192 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2023-09-30T01:33:08.192 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:33:08.288 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.285+0000 7f72ed757700 10 monclient: get_monmap_and_config 2023-09-30T01:33:08.288 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.285+0000 7f72ed757700 10 monclient: build_initial_monmap 2023-09-30T01:33:08.289 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.285+0000 7f72ed757700 10 monclient: monmap: 2023-09-30T01:33:08.289 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2023-09-30T01:33:08.289 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:33:08.289 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:33:08.288868+0000 2023-09-30T01:33:08.289 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:33:08.288868+0000 2023-09-30T01:33:08.289 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2023-09-30T01:33:08.289 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:33:08.289 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a-legacy 2023-09-30T01:33:08.289 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.107:6789/0 mon.noname-f-legacy 2023-09-30T01:33:08.289 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.noname-b 2023-09-30T01:33:08.289 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.noname-g 2023-09-30T01:33:08.289 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.noname-c 2023-09-30T01:33:08.290 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.noname-h 2023-09-30T01:33:08.290 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.noname-d 2023-09-30T01:33:08.290 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.noname-i 2023-09-30T01:33:08.290 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.noname-e 2023-09-30T01:33:08.290 INFO:teuthology.orchestra.run.smithi088.stderr:9: v2:172.21.15.88:3300/0 mon.noname-a 2023-09-30T01:33:08.290 INFO:teuthology.orchestra.run.smithi088.stderr:10: v2:172.21.15.107:3300/0 mon.noname-f 2023-09-30T01:33:08.290 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:33:08.294 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.285+0000 7f72ed757700 1 Processor -- start 2023-09-30T01:33:08.294 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.285+0000 7f72ed757700 1 -- start start 2023-09-30T01:33:08.294 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.285+0000 7f72ed757700 10 monclient: init 2023-09-30T01:33:08.294 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.285+0000 7f72ed757700 10 monclient: _reopen_session rank -1 2023-09-30T01:33:08.294 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.285+0000 7f72ed757700 10 monclient: _add_conns ranks=[4,7,6,0,10,2,8,1,3,5,9] 2023-09-30T01:33:08.294 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.285+0000 7f72ed757700 1 --2- >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f72e8097a40 0x7f72e8132700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:33:08.294 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.285+0000 7f72ed757700 10 monclient(hunting): picked mon.noname-c con 0x7f72e8097a40 addr [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] 2023-09-30T01:33:08.294 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.285+0000 7f72ed757700 1 --2- >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7f72e81311c0 0x7f72e8132c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:33:08.294 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.285+0000 7f72ed757700 10 monclient(hunting): picked mon.noname-i con 0x7f72e81311c0 addr [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] 2023-09-30T01:33:08.294 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.285+0000 7f72ed757700 1 --2- >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7f72e81332e0 0x7f72e8133710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:33:08.295 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.285+0000 7f72ed757700 10 monclient(hunting): picked mon.noname-d con 0x7f72e81332e0 addr [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] 2023-09-30T01:33:08.295 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.285+0000 7f72ed757700 10 monclient(hunting): start opening mon connection 2023-09-30T01:33:08.295 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.285+0000 7f72ed757700 1 -- --> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] -- mon_getmap magic: 0 v1 -- 0x7f72e8131590 con 0x7f72e8097a40 2023-09-30T01:33:08.295 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.285+0000 7f72ed757700 10 monclient(hunting): start opening mon connection 2023-09-30T01:33:08.295 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.285+0000 7f72ed757700 1 -- --> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] -- mon_getmap magic: 0 v1 -- 0x7f72e809a5d0 con 0x7f72e81332e0 2023-09-30T01:33:08.295 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.285+0000 7f72ed757700 10 monclient(hunting): start opening mon connection 2023-09-30T01:33:08.295 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.285+0000 7f72ed757700 1 -- --> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] -- mon_getmap magic: 0 v1 -- 0x7f72e8097e10 con 0x7f72e81311c0 2023-09-30T01:33:08.295 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.285+0000 7f72ed757700 10 monclient(hunting): _renew_subs 2023-09-30T01:33:08.295 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.285+0000 7f72ed757700 10 monclient(hunting): authenticate will time out at 860.327217s 2023-09-30T01:33:08.295 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.285+0000 7f72ecf56700 1 --2- >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7f72e81332e0 0x7f72e8133710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:33:08.295 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.285+0000 7f72ecf56700 1 --2- >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7f72e81332e0 0x7f72e8133710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3303/0 says I am v2:172.21.15.88:53632/0 (socket says 172.21.15.88:53632) 2023-09-30T01:33:08.295 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.285+0000 7f72ecf56700 1 -- 172.21.15.88:0/1260168805 learned_addr learned my addr 172.21.15.88:0/1260168805 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-09-30T01:33:08.295 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.285+0000 7f72ecf56700 10 monclient(hunting): get_auth_request con 0x7f72e81332e0 auth_method 0 2023-09-30T01:33:08.296 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.285+0000 7f72ecf56700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:33:08.296 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.285+0000 7f72ecf56700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:33:08.296 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.285+0000 7f72ecf56700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:33:08.296 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.285+0000 7f72e7fff700 1 --2- 172.21.15.88:0/1260168805 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f72e8097a40 0x7f72e8132700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:33:08.296 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.285+0000 7f72ecf56700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:33:08.296 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.285+0000 7f72ecf56700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:33:08.296 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.285+0000 7f72ecf56700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:33:08.296 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.285+0000 7f72e7fff700 10 monclient(hunting): get_auth_request con 0x7f72e8097a40 auth_method 0 2023-09-30T01:33:08.296 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.285+0000 7f72e7fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:33:08.296 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.285+0000 7f72e7fff700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:33:08.296 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.285+0000 7f72e7fff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:33:08.296 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.285+0000 7f72dffff700 1 --2- 172.21.15.88:0/1260168805 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7f72e81311c0 0x7f72e8132c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:33:08.296 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.285+0000 7f72dffff700 10 monclient(hunting): get_auth_request con 0x7f72e81311c0 auth_method 0 2023-09-30T01:33:08.297 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.285+0000 7f72dffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:33:08.297 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.285+0000 7f72dffff700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:33:08.297 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.285+0000 7f72dffff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:33:08.297 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.285+0000 7f72ecf56700 10 monclient(hunting): handle_auth_done global_id 4308 payload 274 2023-09-30T01:33:08.297 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.285+0000 7f72ecf56700 1 -- 172.21.15.88:0/1260168805 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7f72e81311c0 msgr2=0x7f72e8132c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:33:08.297 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.285+0000 7f72ecf56700 1 --2- 172.21.15.88:0/1260168805 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7f72e81311c0 0x7f72e8132c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:08.297 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.285+0000 7f72ecf56700 1 -- 172.21.15.88:0/1260168805 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f72e8097a40 msgr2=0x7f72e8132700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:33:08.297 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.285+0000 7f72ecf56700 1 --2- 172.21.15.88:0/1260168805 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f72e8097a40 0x7f72e8132700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:08.297 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.285+0000 7f72ecf56700 10 monclient: _finish_hunting 0 2023-09-30T01:33:08.297 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.285+0000 7f72ecf56700 1 monclient: found mon.noname-d 2023-09-30T01:33:08.297 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.285+0000 7f72ecf56700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:33:08.297 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.285+0000 7f72ecf56700 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.88:3303/0 2023-09-30T01:33:08.298 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.285+0000 7f72ecf56700 1 -- 172.21.15.88:0/1260168805 --> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72e804bb20 con 0x7f72e81332e0 2023-09-30T01:33:08.298 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.285+0000 7f72e7fff700 1 --2- 172.21.15.88:0/1260168805 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f72e8097a40 0x7f72e8132700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-30T01:33:08.298 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.289+0000 7f72ecf56700 1 --2- 172.21.15.88:0/1260168805 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7f72e81332e0 0x7f72e8133710 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f72e0009f50 tx=0x7f72e0009840 comp rx=0 tx=0).ready entity=mon.6 client_cookie=ca0eba40f194798a server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:33:08.298 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.289+0000 7f72e77fe700 1 -- 172.21.15.88:0/1260168805 <== mon.6 v2:172.21.15.88:3303/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f72e000e070 con 0x7f72e81332e0 2023-09-30T01:33:08.298 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.289+0000 7f72e77fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:33:08.298 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.289+0000 7f72e77fe700 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2023-09-30T01:33:08.298 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.289+0000 7f72e77fe700 10 monclient: dump: 2023-09-30T01:33:08.298 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:33:08.298 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:33:08.298 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:08.298 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:08.298 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:33:08.299 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:33:08.299 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:33:08.299 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:33:08.299 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:33:08.299 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:33:08.299 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:33:08.299 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:33:08.299 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:33:08.299 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:33:08.299 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:33:08.299 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:33:08.299 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.289+0000 7f72e77fe700 10 monclient: _finish_auth 0 2023-09-30T01:33:08.300 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.289+0000 7f72e77fe700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:33:08.300 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.289+0000 7f72e77fe700 1 -- 172.21.15.88:0/1260168805 <== mon.6 v2:172.21.15.88:3303/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f72e0007400 con 0x7f72e81332e0 2023-09-30T01:33:08.300 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.289+0000 7f72ed757700 5 monclient: authenticate success, global_id 4308 2023-09-30T01:33:08.300 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.289+0000 7f72ed757700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:33:08.300 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.289+0000 7f72e77fe700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:33:08.300 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.289+0000 7f72ed757700 10 monclient: get_monmap_and_config success 2023-09-30T01:33:08.300 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.289+0000 7f72e77fe700 1 -- 172.21.15.88:0/1260168805 <== mon.6 v2:172.21.15.88:3303/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f72e00092e0 con 0x7f72e81332e0 2023-09-30T01:33:08.300 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.289+0000 7f72e77fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:33:08.300 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.289+0000 7f72ed757700 4 set_mon_vals no callback set 2023-09-30T01:33:08.300 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.289+0000 7f72e77fe700 10 monclient: got monmap 1 from mon.d (according to old e1) 2023-09-30T01:33:08.300 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.289+0000 7f72e77fe700 10 monclient: dump: 2023-09-30T01:33:08.300 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:33:08.301 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:33:08.301 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:08.301 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:08.301 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:33:08.301 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:33:08.301 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:33:08.301 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:33:08.301 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:33:08.301 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:33:08.301 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:33:08.301 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:33:08.301 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:33:08.302 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:33:08.302 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:33:08.302 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:33:08.302 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.289+0000 7f72ed757700 10 monclient: shutdown 2023-09-30T01:33:08.302 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.289+0000 7f72ed757700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:33:08.302 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.289+0000 7f72ed757700 1 -- 172.21.15.88:0/1260168805 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7f72e81332e0 msgr2=0x7f72e8133710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:33:08.302 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.289+0000 7f72ed757700 1 --2- 172.21.15.88:0/1260168805 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7f72e81332e0 0x7f72e8133710 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f72e0009f50 tx=0x7f72e0009840 comp rx=0 tx=0).stop 2023-09-30T01:33:08.302 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.289+0000 7f72ed757700 1 -- 172.21.15.88:0/1260168805 shutdown_connections 2023-09-30T01:33:08.302 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.289+0000 7f72ed757700 1 --2- 172.21.15.88:0/1260168805 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7f72e81332e0 0x7f72e8133710 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:08.302 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.289+0000 7f72ed757700 1 --2- 172.21.15.88:0/1260168805 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7f72e81311c0 0x7f72e8132c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:08.302 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.289+0000 7f72ed757700 1 --2- 172.21.15.88:0/1260168805 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f72e8097a40 0x7f72e8132700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:08.303 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.289+0000 7f72ed757700 1 -- 172.21.15.88:0/1260168805 >> 172.21.15.88:0/1260168805 conn(0x7f72e806e5e0 msgr2=0x7f72e806e9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:33:08.303 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.289+0000 7f72ed757700 1 -- 172.21.15.88:0/1260168805 shutdown_connections 2023-09-30T01:33:08.303 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.289+0000 7f72ed757700 1 -- 172.21.15.88:0/1260168805 wait complete. 2023-09-30T01:33:08.303 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.289+0000 7f72ed757700 10 monclient: build_initial_monmap 2023-09-30T01:33:08.303 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.289+0000 7f72ed757700 10 monclient: monmap: 2023-09-30T01:33:08.303 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2023-09-30T01:33:08.303 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:33:08.303 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 0.000000 2023-09-30T01:33:08.303 INFO:teuthology.orchestra.run.smithi088.stderr:created 0.000000 2023-09-30T01:33:08.303 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2023-09-30T01:33:08.303 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 1 2023-09-30T01:33:08.303 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.noname-a 2023-09-30T01:33:08.304 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.noname-f 2023-09-30T01:33:08.304 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.noname-b 2023-09-30T01:33:08.304 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.noname-g 2023-09-30T01:33:08.304 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.noname-c 2023-09-30T01:33:08.304 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.noname-h 2023-09-30T01:33:08.304 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.noname-d 2023-09-30T01:33:08.304 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.noname-i 2023-09-30T01:33:08.304 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.noname-e 2023-09-30T01:33:08.304 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:33:08.308 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.289+0000 7f72ed757700 1 Processor -- start 2023-09-30T01:33:08.308 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.289+0000 7f72ed757700 1 -- start start 2023-09-30T01:33:08.308 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.289+0000 7f72ed757700 10 monclient: init 2023-09-30T01:33:08.308 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.293+0000 7f72ed757700 10 monclient: _reopen_session rank -1 2023-09-30T01:33:08.308 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.293+0000 7f72ed757700 10 monclient: _add_conns ranks=[0,4,1,8,5,3,7,6,2] 2023-09-30T01:33:08.309 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.293+0000 7f72ed757700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f72e8097a40 0x7f72e811cd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:33:08.309 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.293+0000 7f72ed757700 10 monclient(hunting): picked mon.noname-a con 0x7f72e8097a40 addr [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] 2023-09-30T01:33:08.309 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.293+0000 7f72ed757700 1 --2- >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f72e81311c0 0x7f72e811d290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:33:08.309 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.293+0000 7f72ed757700 10 monclient(hunting): picked mon.noname-c con 0x7f72e81311c0 addr [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] 2023-09-30T01:33:08.309 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.293+0000 7f72ed757700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f72e81332e0 0x7f72e811d7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:33:08.309 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.293+0000 7f72ed757700 10 monclient(hunting): picked mon.noname-f con 0x7f72e81332e0 addr [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] 2023-09-30T01:33:08.309 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.293+0000 7f72ed757700 10 monclient(hunting): start opening mon connection 2023-09-30T01:33:08.309 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.293+0000 7f72ed757700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72e8099c30 con 0x7f72e8097a40 2023-09-30T01:33:08.309 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.293+0000 7f72ed757700 10 monclient(hunting): start opening mon connection 2023-09-30T01:33:08.309 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.293+0000 7f72ed757700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72e80996b0 con 0x7f72e81332e0 2023-09-30T01:33:08.309 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.293+0000 7f72ed757700 10 monclient(hunting): start opening mon connection 2023-09-30T01:33:08.309 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.293+0000 7f72ed757700 1 -- --> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] -- mon_getmap magic: 0 v1 -- 0x7f72e811ead0 con 0x7f72e81311c0 2023-09-30T01:33:08.309 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.293+0000 7f72ed757700 10 monclient(hunting): _renew_subs 2023-09-30T01:33:08.310 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.293+0000 7f72ed757700 10 monclient(hunting): authenticate will time out at 860.333221s 2023-09-30T01:33:08.310 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.293+0000 7f72e7fff700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f72e8097a40 0x7f72e811cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:33:08.310 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.293+0000 7f72e7fff700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f72e8097a40 0x7f72e811cd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:57936/0 (socket says 172.21.15.88:57936) 2023-09-30T01:33:08.310 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.293+0000 7f72dffff700 1 --2- >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f72e81311c0 0x7f72e811d290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:33:08.310 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.293+0000 7f72e7fff700 1 -- 172.21.15.88:0/3458514802 learned_addr learned my addr 172.21.15.88:0/3458514802 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-09-30T01:33:08.310 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.293+0000 7f72e7fff700 10 monclient(hunting): get_auth_request con 0x7f72e8097a40 auth_method 0 2023-09-30T01:33:08.310 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.293+0000 7f72e7fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:33:08.310 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.293+0000 7f72e7fff700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:33:08.310 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.293+0000 7f72e7fff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:33:08.310 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.293+0000 7f72dffff700 10 monclient(hunting): get_auth_request con 0x7f72e81311c0 auth_method 0 2023-09-30T01:33:08.310 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.293+0000 7f72dffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:33:08.311 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.293+0000 7f72dffff700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:33:08.311 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.293+0000 7f72dffff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:33:08.311 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.293+0000 7f72e7fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:33:08.311 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.293+0000 7f72e7fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:33:08.311 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.293+0000 7f72e7fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:33:08.311 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.293+0000 7f72dffff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:33:08.311 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.293+0000 7f72dffff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:33:08.311 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.293+0000 7f72ecf56700 1 --2- 172.21.15.88:0/3458514802 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f72e81332e0 0x7f72e811d7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:33:08.311 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.293+0000 7f72dffff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:33:08.311 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.293+0000 7f72ecf56700 10 monclient(hunting): get_auth_request con 0x7f72e81332e0 auth_method 0 2023-09-30T01:33:08.311 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.293+0000 7f72ecf56700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:33:08.311 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.293+0000 7f72ecf56700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:33:08.312 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.293+0000 7f72ecf56700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:33:08.312 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.293+0000 7f72e7fff700 10 monclient(hunting): handle_auth_done global_id 4446 payload 824 2023-09-30T01:33:08.312 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.293+0000 7f72e7fff700 1 -- 172.21.15.88:0/3458514802 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f72e81311c0 msgr2=0x7f72e811d290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:33:08.312 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.293+0000 7f72e7fff700 1 --2- 172.21.15.88:0/3458514802 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f72e81311c0 0x7f72e811d290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:08.312 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.293+0000 7f72e7fff700 1 -- 172.21.15.88:0/3458514802 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f72e81332e0 msgr2=0x7f72e811d7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:33:08.312 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.293+0000 7f72e7fff700 1 --2- 172.21.15.88:0/3458514802 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f72e81332e0 0x7f72e811d7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:08.312 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.293+0000 7f72e7fff700 10 monclient: _finish_hunting 0 2023-09-30T01:33:08.312 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.293+0000 7f72e7fff700 1 monclient: found mon.noname-a 2023-09-30T01:33:08.312 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.293+0000 7f72e7fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:33:08.312 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.293+0000 7f72e7fff700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.88:3300/0 2023-09-30T01:33:08.312 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.293+0000 7f72e7fff700 1 -- 172.21.15.88:0/3458514802 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72e0009ae0 con 0x7f72e8097a40 2023-09-30T01:33:08.312 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.293+0000 7f72ecf56700 1 --2- 172.21.15.88:0/3458514802 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f72e81332e0 0x7f72e811d7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-30T01:33:08.313 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.293+0000 7f72e7fff700 1 --2- 172.21.15.88:0/3458514802 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f72e8097a40 0x7f72e811cd50 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f72d800e8d0 tx=0x7f72d8008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:33:08.313 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.293+0000 7f72e4ff9700 1 -- 172.21.15.88:0/3458514802 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f72d8013890 con 0x7f72e8097a40 2023-09-30T01:33:08.313 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.293+0000 7f72e4ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:33:08.313 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.293+0000 7f72e4ff9700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T01:33:08.313 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.293+0000 7f72e4ff9700 10 monclient: dump: 2023-09-30T01:33:08.313 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:33:08.313 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:33:08.313 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:08.313 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:08.313 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:33:08.313 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:33:08.313 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:33:08.314 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:33:08.314 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:33:08.314 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:33:08.314 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:33:08.314 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:33:08.314 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:33:08.314 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:33:08.314 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:33:08.314 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:33:08.314 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.293+0000 7f72e4ff9700 10 monclient: _finish_auth 0 2023-09-30T01:33:08.314 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.293+0000 7f72e4ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:33:08.314 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.293+0000 7f72e4ff9700 1 -- 172.21.15.88:0/3458514802 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f72d8013a30 con 0x7f72e8097a40 2023-09-30T01:33:08.315 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.293+0000 7f72ed757700 5 monclient: authenticate success, global_id 4446 2023-09-30T01:33:08.315 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.293+0000 7f72e4ff9700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:33:08.315 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.293+0000 7f72ed757700 10 monclient: _renew_subs 2023-09-30T01:33:08.315 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.293+0000 7f72ed757700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.88:3300/0 2023-09-30T01:33:08.315 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.293+0000 7f72e4ff9700 1 -- 172.21.15.88:0/3458514802 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f72d8013ba0 con 0x7f72e8097a40 2023-09-30T01:33:08.315 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.293+0000 7f72ed757700 1 -- 172.21.15.88:0/3458514802 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f72e8130e50 con 0x7f72e8097a40 2023-09-30T01:33:08.315 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.293+0000 7f72e4ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:33:08.315 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.293+0000 7f72e4ff9700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:33:08.315 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.293+0000 7f72e6ffd700 4 set_mon_vals no callback set 2023-09-30T01:33:08.315 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.293+0000 7f72e4ff9700 10 monclient: dump: 2023-09-30T01:33:08.315 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:33:08.315 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:33:08.315 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:08.316 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:08.316 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:33:08.316 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:33:08.316 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:33:08.316 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:33:08.316 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:33:08.316 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:33:08.316 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:33:08.316 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:33:08.316 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:33:08.316 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:33:08.316 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:33:08.316 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:33:08.316 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.293+0000 7f72ed757700 10 monclient: _renew_subs 2023-09-30T01:33:08.317 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.293+0000 7f72ed757700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.88:3300/0 2023-09-30T01:33:08.317 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.293+0000 7f72ed757700 1 -- 172.21.15.88:0/3458514802 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f72e811e280 con 0x7f72e8097a40 2023-09-30T01:33:08.317 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.293+0000 7f72ed757700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:33:08.317 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.297+0000 7f72ed757700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:33:08.317 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.297+0000 7f72ed757700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.88:3300/0 2023-09-30T01:33:08.317 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.297+0000 7f72ed757700 1 -- 172.21.15.88:0/3458514802 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f72e81202d0 con 0x7f72e8097a40 2023-09-30T01:33:08.317 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.297+0000 7f72e4ff9700 1 -- 172.21.15.88:0/3458514802 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 40144+0+0 (secure 0 0 0) 0x7f72d8013d20 con 0x7f72e8097a40 2023-09-30T01:33:08.317 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.297+0000 7f72e4ff9700 1 --2- 172.21.15.88:0/3458514802 >> v2:172.21.15.107:6812/29926 conn(0x7f72c8033360 0x7f72c8035810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:33:08.317 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.297+0000 7f72e4ff9700 1 -- 172.21.15.88:0/3458514802 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 3502+0+0 (secure 0 0 0) 0x7f72d8043220 con 0x7f72e8097a40 2023-09-30T01:33:08.317 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.297+0000 7f72dffff700 1 --2- 172.21.15.88:0/3458514802 >> v2:172.21.15.107:6812/29926 conn(0x7f72c8033360 0x7f72c8035810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:33:08.317 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.297+0000 7f72dffff700 10 monclient: get_auth_request con 0x7f72c8033360 auth_method 0 2023-09-30T01:33:08.317 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.297+0000 7f72dffff700 1 --2- 172.21.15.88:0/3458514802 >> v2:172.21.15.107:6812/29926 conn(0x7f72c8033360 0x7f72c8035810 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f72d0006fd0 tx=0x7f72d0008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:33:08.318 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.305+0000 7f72e4ff9700 1 -- 172.21.15.88:0/3458514802 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f72d80224e0 con 0x7f72e8097a40 2023-09-30T01:33:08.318 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.305+0000 7f72e4ff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:33:08.318 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.305+0000 7f72e4ff9700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:33:08.508 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.505+0000 7f72ed757700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T01:33:08.508 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.505+0000 7f72ed757700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:33:08.509 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.505+0000 7f72ed757700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.88:3300/0 2023-09-30T01:33:08.509 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.505+0000 7f72ed757700 1 -- 172.21.15.88:0/3458514802 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f72e81200f0 con 0x7f72e8097a40 2023-09-30T01:33:08.509 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.505+0000 7f72e4ff9700 1 -- 172.21.15.88:0/3458514802 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f72d8022300 con 0x7f72e8097a40 2023-09-30T01:33:08.509 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.505+0000 7f72e4ff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:33:08.509 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.505+0000 7f72e4ff9700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:33:08.510 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-09-30T01:33:08.510 INFO:teuthology.orchestra.run.smithi088.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:33:08.512 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.509+0000 7f72ed757700 1 -- 172.21.15.88:0/3458514802 >> v2:172.21.15.107:6812/29926 conn(0x7f72c8033360 msgr2=0x7f72c8035810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:33:08.512 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.509+0000 7f72ed757700 1 --2- 172.21.15.88:0/3458514802 >> v2:172.21.15.107:6812/29926 conn(0x7f72c8033360 0x7f72c8035810 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f72d0006fd0 tx=0x7f72d0008040 comp rx=0 tx=0).stop 2023-09-30T01:33:08.512 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.509+0000 7f72ed757700 10 monclient: shutdown 2023-09-30T01:33:08.512 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.509+0000 7f72ed757700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:33:08.512 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.509+0000 7f72ed757700 1 -- 172.21.15.88:0/3458514802 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f72e8097a40 msgr2=0x7f72e811cd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:33:08.512 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.509+0000 7f72ed757700 1 --2- 172.21.15.88:0/3458514802 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f72e8097a40 0x7f72e811cd50 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f72d800e8d0 tx=0x7f72d8008040 comp rx=0 tx=0).stop 2023-09-30T01:33:08.512 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.509+0000 7f72ed757700 1 -- 172.21.15.88:0/3458514802 shutdown_connections 2023-09-30T01:33:08.512 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.509+0000 7f72ed757700 1 --2- 172.21.15.88:0/3458514802 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f72e81332e0 0x7f72e811d7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:08.512 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.509+0000 7f72ed757700 1 --2- 172.21.15.88:0/3458514802 >> v2:172.21.15.107:6812/29926 conn(0x7f72c8033360 0x7f72c8035810 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:08.513 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.509+0000 7f72ed757700 1 --2- 172.21.15.88:0/3458514802 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f72e81311c0 0x7f72e811d290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:08.513 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.509+0000 7f72ed757700 1 --2- 172.21.15.88:0/3458514802 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f72e8097a40 0x7f72e811cd50 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:08.513 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.509+0000 7f72ed757700 1 -- 172.21.15.88:0/3458514802 >> 172.21.15.88:0/3458514802 conn(0x7f72e806e5e0 msgr2=0x7f72e80a4860 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:33:08.513 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.509+0000 7f72ed757700 1 -- 172.21.15.88:0/3458514802 shutdown_connections 2023-09-30T01:33:08.513 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:08.509+0000 7f72ed757700 1 -- 172.21.15.88:0/3458514802 wait complete. 2023-09-30T01:33:11.524 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:33:11.635 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:11.629+0000 7fc68eea6700 10 monclient: get_monmap_and_config 2023-09-30T01:33:11.635 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:11.629+0000 7fc68eea6700 10 monclient: build_initial_monmap 2023-09-30T01:33:11.635 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:11.629+0000 7fc68eea6700 10 monclient: monmap: 2023-09-30T01:33:11.635 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2023-09-30T01:33:11.635 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:33:11.635 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:33:11.635217+0000 2023-09-30T01:33:11.635 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:33:11.635217+0000 2023-09-30T01:33:11.635 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2023-09-30T01:33:11.635 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:33:11.636 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a-legacy 2023-09-30T01:33:11.636 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.107:6789/0 mon.noname-f-legacy 2023-09-30T01:33:11.636 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.noname-b 2023-09-30T01:33:11.636 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.noname-g 2023-09-30T01:33:11.636 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.noname-c 2023-09-30T01:33:11.636 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.noname-h 2023-09-30T01:33:11.636 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.noname-d 2023-09-30T01:33:11.636 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.noname-i 2023-09-30T01:33:11.636 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.noname-e 2023-09-30T01:33:11.636 INFO:teuthology.orchestra.run.smithi088.stderr:9: v2:172.21.15.88:3300/0 mon.noname-a 2023-09-30T01:33:11.636 INFO:teuthology.orchestra.run.smithi088.stderr:10: v2:172.21.15.107:3300/0 mon.noname-f 2023-09-30T01:33:11.636 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:33:11.637 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:11.633+0000 7fc68eea6700 1 Processor -- start 2023-09-30T01:33:11.637 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:11.633+0000 7fc68eea6700 1 -- start start 2023-09-30T01:33:11.637 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:11.633+0000 7fc68eea6700 10 monclient: init 2023-09-30T01:33:11.644 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:11.633+0000 7fc68eea6700 10 monclient: _reopen_session rank -1 2023-09-30T01:33:11.644 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:11.633+0000 7fc68eea6700 10 monclient: _add_conns ranks=[6,4,5,10,1,7,3,9,8,0,2] 2023-09-30T01:33:11.644 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:11.633+0000 7fc68eea6700 1 --2- >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7fc688097a40 0x7fc688132700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:33:11.644 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:11.633+0000 7fc68eea6700 10 monclient(hunting): picked mon.noname-d con 0x7fc688097a40 addr [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] 2023-09-30T01:33:11.644 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:11.633+0000 7fc68eea6700 1 --2- >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7fc6881311c0 0x7fc688132c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:33:11.644 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:11.633+0000 7fc68eea6700 10 monclient(hunting): picked mon.noname-c con 0x7fc6881311c0 addr [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] 2023-09-30T01:33:11.645 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:11.633+0000 7fc68eea6700 1 --2- >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7fc6881332e0 0x7fc688133710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:33:11.645 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:11.633+0000 7fc68eea6700 10 monclient(hunting): picked mon.noname-h con 0x7fc6881332e0 addr [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] 2023-09-30T01:33:11.645 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:11.633+0000 7fc68eea6700 10 monclient(hunting): start opening mon connection 2023-09-30T01:33:11.645 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:11.633+0000 7fc68eea6700 1 -- --> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] -- mon_getmap magic: 0 v1 -- 0x7fc688131590 con 0x7fc6881311c0 2023-09-30T01:33:11.645 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:11.633+0000 7fc68eea6700 10 monclient(hunting): start opening mon connection 2023-09-30T01:33:11.645 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:11.633+0000 7fc68eea6700 1 -- --> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] -- mon_getmap magic: 0 v1 -- 0x7fc68809a5d0 con 0x7fc6881332e0 2023-09-30T01:33:11.645 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:11.633+0000 7fc68eea6700 10 monclient(hunting): start opening mon connection 2023-09-30T01:33:11.645 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:11.633+0000 7fc68eea6700 1 -- --> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] -- mon_getmap magic: 0 v1 -- 0x7fc688097e10 con 0x7fc688097a40 2023-09-30T01:33:11.645 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:11.633+0000 7fc68eea6700 10 monclient(hunting): _renew_subs 2023-09-30T01:33:11.645 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:11.633+0000 7fc68eea6700 10 monclient(hunting): authenticate will time out at 863.672692s 2023-09-30T01:33:11.645 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:11.633+0000 7fc68d6a3700 1 --2- >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7fc6881311c0 0x7fc688132c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:33:11.646 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:11.633+0000 7fc68d6a3700 1 --2- >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7fc6881311c0 0x7fc688132c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3302/0 says I am v2:172.21.15.88:34820/0 (socket says 172.21.15.88:34820) 2023-09-30T01:33:11.646 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:11.633+0000 7fc68d6a3700 1 -- 172.21.15.88:0/4209425033 learned_addr learned my addr 172.21.15.88:0/4209425033 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-09-30T01:33:11.646 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:11.633+0000 7fc68d6a3700 10 monclient(hunting): get_auth_request con 0x7fc6881311c0 auth_method 0 2023-09-30T01:33:11.646 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:11.633+0000 7fc68d6a3700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:33:11.646 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:11.633+0000 7fc68d6a3700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:33:11.646 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:11.633+0000 7fc68d6a3700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:33:11.646 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:11.633+0000 7fc68dea4700 1 --2- 172.21.15.88:0/4209425033 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7fc688097a40 0x7fc688132700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:33:11.646 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:11.633+0000 7fc68d6a3700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:33:11.646 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:11.633+0000 7fc68d6a3700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:33:11.646 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:11.633+0000 7fc68d6a3700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:33:11.646 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:11.633+0000 7fc68dea4700 10 monclient(hunting): get_auth_request con 0x7fc688097a40 auth_method 0 2023-09-30T01:33:11.647 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:11.633+0000 7fc68dea4700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:33:11.647 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:11.633+0000 7fc68dea4700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:33:11.647 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:11.633+0000 7fc68dea4700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:33:11.647 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:11.633+0000 7fc68e6a5700 1 --2- 172.21.15.88:0/4209425033 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7fc6881332e0 0x7fc688133710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:33:11.647 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:11.633+0000 7fc68e6a5700 10 monclient(hunting): get_auth_request con 0x7fc6881332e0 auth_method 0 2023-09-30T01:33:11.647 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:11.633+0000 7fc68e6a5700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:33:11.647 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:11.633+0000 7fc68e6a5700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:33:11.647 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:11.633+0000 7fc68e6a5700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:33:11.647 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:11.633+0000 7fc68d6a3700 10 monclient(hunting): handle_auth_done global_id 4369 payload 274 2023-09-30T01:33:11.647 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:11.633+0000 7fc68d6a3700 1 -- 172.21.15.88:0/4209425033 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7fc688097a40 msgr2=0x7fc688132700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:33:11.647 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:11.633+0000 7fc68d6a3700 1 --2- 172.21.15.88:0/4209425033 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7fc688097a40 0x7fc688132700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:11.647 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:11.633+0000 7fc68d6a3700 1 -- 172.21.15.88:0/4209425033 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7fc6881332e0 msgr2=0x7fc688133710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:33:11.648 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:11.633+0000 7fc68d6a3700 1 --2- 172.21.15.88:0/4209425033 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7fc6881332e0 0x7fc688133710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:11.648 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:11.633+0000 7fc68d6a3700 10 monclient: _finish_hunting 0 2023-09-30T01:33:11.648 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:11.633+0000 7fc68d6a3700 1 monclient: found mon.noname-c 2023-09-30T01:33:11.648 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:11.633+0000 7fc68d6a3700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:33:11.648 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:11.633+0000 7fc68d6a3700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.88:3302/0 2023-09-30T01:33:11.648 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:11.633+0000 7fc68d6a3700 1 -- 172.21.15.88:0/4209425033 --> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc68804bb20 con 0x7fc6881311c0 2023-09-30T01:33:11.648 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:11.633+0000 7fc68d6a3700 1 --2- 172.21.15.88:0/4209425033 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7fc6881311c0 0x7fc688132c40 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fc6780099e0 tx=0x7fc678009250 comp rx=0 tx=0).ready entity=mon.4 client_cookie=895e50e1f8a95fc7 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:33:11.648 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:11.633+0000 7fc68cea2700 1 -- 172.21.15.88:0/4209425033 <== mon.4 v2:172.21.15.88:3302/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7fc67800bce0 con 0x7fc6881311c0 2023-09-30T01:33:11.648 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:11.633+0000 7fc68cea2700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:33:11.648 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:11.633+0000 7fc68cea2700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T01:33:11.648 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:11.633+0000 7fc68cea2700 10 monclient: dump: 2023-09-30T01:33:11.648 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:33:11.649 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:33:11.649 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:11.649 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:11.649 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:33:11.649 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:33:11.649 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:33:11.649 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:33:11.649 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:33:11.649 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:33:11.649 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:33:11.649 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:33:11.649 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:33:11.650 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:33:11.650 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:33:11.650 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:33:11.650 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:11.633+0000 7fc68cea2700 10 monclient: _finish_auth 0 2023-09-30T01:33:11.650 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:11.633+0000 7fc68cea2700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:33:11.650 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:11.633+0000 7fc68eea6700 5 monclient: authenticate success, global_id 4369 2023-09-30T01:33:11.650 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:11.633+0000 7fc68cea2700 1 -- 172.21.15.88:0/4209425033 <== mon.4 v2:172.21.15.88:3302/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc67800be60 con 0x7fc6881311c0 2023-09-30T01:33:11.650 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:11.633+0000 7fc68eea6700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:33:11.650 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:11.633+0000 7fc68cea2700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:33:11.650 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:11.633+0000 7fc68eea6700 10 monclient: get_monmap_and_config success 2023-09-30T01:33:11.650 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:11.633+0000 7fc68eea6700 4 set_mon_vals no callback set 2023-09-30T01:33:11.650 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:11.633+0000 7fc68cea2700 1 -- 172.21.15.88:0/4209425033 <== mon.4 v2:172.21.15.88:3302/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7fc6780118a0 con 0x7fc6881311c0 2023-09-30T01:33:11.651 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:11.633+0000 7fc68cea2700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:33:11.651 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:11.633+0000 7fc68cea2700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T01:33:11.651 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:11.633+0000 7fc68cea2700 10 monclient: dump: 2023-09-30T01:33:11.651 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:33:11.651 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:33:11.651 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:11.651 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:11.651 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:33:11.651 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:33:11.651 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:33:11.651 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:33:11.651 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:33:11.652 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:33:11.652 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:33:11.652 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:33:11.652 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:33:11.652 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:33:11.652 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:33:11.652 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:33:11.652 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:11.633+0000 7fc68eea6700 10 monclient: shutdown 2023-09-30T01:33:11.652 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:11.633+0000 7fc68eea6700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:33:11.652 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:11.633+0000 7fc68eea6700 1 -- 172.21.15.88:0/4209425033 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7fc6881311c0 msgr2=0x7fc688132c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:33:11.652 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:11.633+0000 7fc68eea6700 1 --2- 172.21.15.88:0/4209425033 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7fc6881311c0 0x7fc688132c40 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fc6780099e0 tx=0x7fc678009250 comp rx=0 tx=0).stop 2023-09-30T01:33:11.652 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:11.633+0000 7fc68eea6700 1 -- 172.21.15.88:0/4209425033 shutdown_connections 2023-09-30T01:33:11.653 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:11.633+0000 7fc68eea6700 1 --2- 172.21.15.88:0/4209425033 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7fc6881311c0 0x7fc688132c40 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:11.653 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:11.633+0000 7fc68eea6700 1 --2- 172.21.15.88:0/4209425033 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7fc6881332e0 0x7fc688133710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:11.653 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:11.633+0000 7fc68eea6700 1 --2- 172.21.15.88:0/4209425033 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7fc688097a40 0x7fc688132700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:11.653 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:11.633+0000 7fc68eea6700 1 -- 172.21.15.88:0/4209425033 >> 172.21.15.88:0/4209425033 conn(0x7fc68806e5e0 msgr2=0x7fc68806e9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:33:11.653 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:11.633+0000 7fc68eea6700 1 -- 172.21.15.88:0/4209425033 shutdown_connections 2023-09-30T01:33:11.653 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:11.633+0000 7fc68eea6700 1 -- 172.21.15.88:0/4209425033 wait complete. 2023-09-30T01:33:11.653 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:11.633+0000 7fc68eea6700 10 monclient: build_initial_monmap 2023-09-30T01:33:11.653 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:11.633+0000 7fc68eea6700 10 monclient: monmap: 2023-09-30T01:33:11.653 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2023-09-30T01:33:11.653 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:33:11.653 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 0.000000 2023-09-30T01:33:11.653 INFO:teuthology.orchestra.run.smithi088.stderr:created 0.000000 2023-09-30T01:33:11.653 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2023-09-30T01:33:11.654 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 1 2023-09-30T01:33:11.654 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.noname-a 2023-09-30T01:33:11.654 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.noname-f 2023-09-30T01:33:11.654 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.noname-b 2023-09-30T01:33:11.654 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.noname-g 2023-09-30T01:33:11.654 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.noname-c 2023-09-30T01:33:11.654 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.noname-h 2023-09-30T01:33:11.654 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.noname-d 2023-09-30T01:33:11.654 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.noname-i 2023-09-30T01:33:11.654 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.noname-e 2023-09-30T01:33:11.654 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:33:11.654 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:11.633+0000 7fc68eea6700 1 Processor -- start 2023-09-30T01:33:11.655 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:11.633+0000 7fc68eea6700 1 -- start start 2023-09-30T01:33:11.655 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:11.633+0000 7fc68eea6700 10 monclient: init 2023-09-30T01:33:11.655 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:11.637+0000 7fc68eea6700 10 monclient: _reopen_session rank -1 2023-09-30T01:33:11.655 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:11.637+0000 7fc68eea6700 10 monclient: _add_conns ranks=[8,1,7,6,0,5,2,3,4] 2023-09-30T01:33:11.655 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:11.637+0000 7fc68eea6700 1 --2- >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7fc688097a40 0x7fc6880a1b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:33:11.655 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:11.637+0000 7fc68eea6700 10 monclient(hunting): picked mon.noname-e con 0x7fc688097a40 addr [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] 2023-09-30T01:33:11.655 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:11.637+0000 7fc68eea6700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc6881311c0 0x7fc6880a2050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:33:11.655 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:11.637+0000 7fc68eea6700 10 monclient(hunting): picked mon.noname-f con 0x7fc6881311c0 addr [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] 2023-09-30T01:33:11.655 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:11.637+0000 7fc68eea6700 1 --2- >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7fc6881332e0 0x7fc6880a2590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:33:11.655 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:11.637+0000 7fc68eea6700 10 monclient(hunting): picked mon.noname-i con 0x7fc6881332e0 addr [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] 2023-09-30T01:33:11.655 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:11.637+0000 7fc68eea6700 10 monclient(hunting): start opening mon connection 2023-09-30T01:33:11.655 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:11.637+0000 7fc68eea6700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc688099c30 con 0x7fc6881311c0 2023-09-30T01:33:11.655 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:11.637+0000 7fc68eea6700 10 monclient(hunting): start opening mon connection 2023-09-30T01:33:11.656 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:11.637+0000 7fc68eea6700 1 -- --> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] -- mon_getmap magic: 0 v1 -- 0x7fc6880996b0 con 0x7fc6881332e0 2023-09-30T01:33:11.656 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:11.637+0000 7fc68eea6700 10 monclient(hunting): start opening mon connection 2023-09-30T01:33:11.656 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:11.637+0000 7fc68eea6700 1 -- --> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] -- mon_getmap magic: 0 v1 -- 0x7fc68811ecd0 con 0x7fc688097a40 2023-09-30T01:33:11.656 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:11.637+0000 7fc68eea6700 10 monclient(hunting): _renew_subs 2023-09-30T01:33:11.656 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:11.637+0000 7fc68eea6700 10 monclient(hunting): authenticate will time out at 863.676166s 2023-09-30T01:33:11.656 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:11.637+0000 7fc68dea4700 1 --2- >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7fc688097a40 0x7fc6880a1b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:33:11.656 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:11.637+0000 7fc68dea4700 1 --2- >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7fc688097a40 0x7fc6880a1b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3304/0 says I am v2:172.21.15.88:50292/0 (socket says 172.21.15.88:50292) 2023-09-30T01:33:11.656 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:11.637+0000 7fc68dea4700 1 -- 172.21.15.88:0/105199668 learned_addr learned my addr 172.21.15.88:0/105199668 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-09-30T01:33:11.656 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:11.637+0000 7fc68dea4700 10 monclient(hunting): get_auth_request con 0x7fc688097a40 auth_method 0 2023-09-30T01:33:11.656 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:11.637+0000 7fc68dea4700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:33:11.656 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:11.637+0000 7fc68dea4700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:33:11.656 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:11.637+0000 7fc68dea4700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:33:11.657 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:11.637+0000 7fc68dea4700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:33:11.657 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:11.637+0000 7fc68dea4700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:33:11.657 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:11.637+0000 7fc68d6a3700 1 --2- 172.21.15.88:0/105199668 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc6881311c0 0x7fc6880a2050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:33:11.657 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:11.637+0000 7fc68dea4700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:33:11.657 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:11.637+0000 7fc68e6a5700 1 --2- 172.21.15.88:0/105199668 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7fc6881332e0 0x7fc6880a2590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:33:11.657 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:11.637+0000 7fc68d6a3700 10 monclient(hunting): get_auth_request con 0x7fc6881311c0 auth_method 0 2023-09-30T01:33:11.657 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:11.637+0000 7fc68d6a3700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:33:11.657 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:11.637+0000 7fc68d6a3700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:33:11.657 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:11.637+0000 7fc68d6a3700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:33:11.657 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:11.637+0000 7fc68e6a5700 10 monclient(hunting): get_auth_request con 0x7fc6881332e0 auth_method 0 2023-09-30T01:33:11.657 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:11.637+0000 7fc68e6a5700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:33:11.657 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:11.637+0000 7fc68e6a5700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:33:11.658 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:11.637+0000 7fc68e6a5700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:33:11.658 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:11.637+0000 7fc68d6a3700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:33:11.658 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:11.637+0000 7fc68d6a3700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:33:11.658 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:11.637+0000 7fc68d6a3700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:33:11.658 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:11.637+0000 7fc68dea4700 10 monclient(hunting): handle_auth_done global_id 4409 payload 824 2023-09-30T01:33:11.658 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:11.637+0000 7fc68dea4700 1 -- 172.21.15.88:0/105199668 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7fc6881332e0 msgr2=0x7fc6880a2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:33:11.658 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:11.637+0000 7fc68dea4700 1 --2- 172.21.15.88:0/105199668 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7fc6881332e0 0x7fc6880a2590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:11.658 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:11.637+0000 7fc68dea4700 1 -- 172.21.15.88:0/105199668 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc6881311c0 msgr2=0x7fc6880a2050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:33:11.658 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:11.637+0000 7fc68dea4700 1 --2- 172.21.15.88:0/105199668 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc6881311c0 0x7fc6880a2050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:11.658 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:11.637+0000 7fc68dea4700 10 monclient: _finish_hunting 0 2023-09-30T01:33:11.658 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:11.637+0000 7fc68dea4700 1 monclient: found mon.noname-e 2023-09-30T01:33:11.658 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:11.637+0000 7fc68dea4700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:33:11.659 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:11.637+0000 7fc68dea4700 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.88:3304/0 2023-09-30T01:33:11.659 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:11.637+0000 7fc68dea4700 1 -- 172.21.15.88:0/105199668 --> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6780094f0 con 0x7fc688097a40 2023-09-30T01:33:11.659 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:11.637+0000 7fc68dea4700 1 --2- 172.21.15.88:0/105199668 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7fc688097a40 0x7fc6880a1b10 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fc68400bfc0 tx=0x7fc684008040 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:33:11.659 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:11.637+0000 7fc67dffb700 1 -- 172.21.15.88:0/105199668 <== mon.8 v2:172.21.15.88:3304/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7fc684016d20 con 0x7fc688097a40 2023-09-30T01:33:11.659 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:11.637+0000 7fc67dffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:33:11.659 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:11.637+0000 7fc67dffb700 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2023-09-30T01:33:11.659 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:11.637+0000 7fc67dffb700 10 monclient: dump: 2023-09-30T01:33:11.659 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:33:11.659 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:33:11.659 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:11.659 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:11.659 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:33:11.660 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:33:11.660 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:33:11.660 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:33:11.661 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:33:11.661 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:33:11.661 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:33:11.661 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:33:11.661 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:33:11.661 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:33:11.661 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:33:11.661 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:33:11.662 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:11.637+0000 7fc67dffb700 10 monclient: _finish_auth 0 2023-09-30T01:33:11.662 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:11.637+0000 7fc67dffb700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:33:11.662 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:11.637+0000 7fc67dffb700 1 -- 172.21.15.88:0/105199668 <== mon.8 v2:172.21.15.88:3304/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc68401a070 con 0x7fc688097a40 2023-09-30T01:33:11.662 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:11.637+0000 7fc68eea6700 5 monclient: authenticate success, global_id 4409 2023-09-30T01:33:11.662 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:11.637+0000 7fc68eea6700 10 monclient: _renew_subs 2023-09-30T01:33:11.662 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:11.637+0000 7fc68eea6700 10 monclient: _send_mon_message to mon.e at v2:172.21.15.88:3304/0 2023-09-30T01:33:11.662 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:11.637+0000 7fc68eea6700 1 -- 172.21.15.88:0/105199668 --> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc68811de10 con 0x7fc688097a40 2023-09-30T01:33:11.662 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:11.637+0000 7fc67dffb700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:33:11.662 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:11.637+0000 7fc67dffb700 1 -- 172.21.15.88:0/105199668 <== mon.8 v2:172.21.15.88:3304/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7fc684021d80 con 0x7fc688097a40 2023-09-30T01:33:11.662 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:11.637+0000 7fc67dffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:33:11.662 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:11.637+0000 7fc67dffb700 10 monclient: got monmap 1 from mon.e (according to old e1) 2023-09-30T01:33:11.662 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:11.637+0000 7fc67dffb700 10 monclient: dump: 2023-09-30T01:33:11.663 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:33:11.663 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:33:11.663 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:11.663 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:11.663 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:33:11.663 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:33:11.663 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:33:11.663 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:33:11.663 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:33:11.663 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:33:11.663 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:33:11.663 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:33:11.663 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:33:11.664 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:33:11.664 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:33:11.664 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:33:11.664 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:11.637+0000 7fc68eea6700 10 monclient: _renew_subs 2023-09-30T01:33:11.664 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:11.637+0000 7fc67f7fe700 4 set_mon_vals no callback set 2023-09-30T01:33:11.664 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:11.637+0000 7fc68eea6700 10 monclient: _send_mon_message to mon.e at v2:172.21.15.88:3304/0 2023-09-30T01:33:11.664 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:11.637+0000 7fc68eea6700 1 -- 172.21.15.88:0/105199668 --> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc68811d1c0 con 0x7fc688097a40 2023-09-30T01:33:11.664 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:11.637+0000 7fc68eea6700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:33:11.664 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:11.637+0000 7fc68eea6700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:33:11.664 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:11.637+0000 7fc68eea6700 10 monclient: _send_mon_message to mon.e at v2:172.21.15.88:3304/0 2023-09-30T01:33:11.664 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:11.637+0000 7fc68eea6700 1 -- 172.21.15.88:0/105199668 --> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc68811f130 con 0x7fc688097a40 2023-09-30T01:33:11.664 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:11.637+0000 7fc67dffb700 1 -- 172.21.15.88:0/105199668 <== mon.8 v2:172.21.15.88:3304/0 4 ==== mgrmap(e 5) v1 ==== 40144+0+0 (secure 0 0 0) 0x7fc684021480 con 0x7fc688097a40 2023-09-30T01:33:11.665 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:11.641+0000 7fc67dffb700 1 --2- 172.21.15.88:0/105199668 >> v2:172.21.15.107:6812/29926 conn(0x7fc66c033360 0x7fc66c035810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:33:11.665 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:11.641+0000 7fc67dffb700 1 -- 172.21.15.88:0/105199668 <== mon.8 v2:172.21.15.88:3304/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 3502+0+0 (secure 0 0 0) 0x7fc68401c070 con 0x7fc688097a40 2023-09-30T01:33:11.665 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:11.641+0000 7fc68d6a3700 1 --2- 172.21.15.88:0/105199668 >> v2:172.21.15.107:6812/29926 conn(0x7fc66c033360 0x7fc66c035810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:33:11.665 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:11.641+0000 7fc68d6a3700 10 monclient: get_auth_request con 0x7fc66c033360 auth_method 0 2023-09-30T01:33:11.665 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:11.641+0000 7fc68d6a3700 1 --2- 172.21.15.88:0/105199668 >> v2:172.21.15.107:6812/29926 conn(0x7fc66c033360 0x7fc66c035810 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fc67800bfd0 tx=0x7fc678017000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:33:11.665 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:11.645+0000 7fc67dffb700 1 -- 172.21.15.88:0/105199668 <== mon.8 v2:172.21.15.88:3304/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fc6840288f0 con 0x7fc688097a40 2023-09-30T01:33:11.666 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:11.645+0000 7fc67dffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:33:11.666 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:11.645+0000 7fc67dffb700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:33:11.844 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:11.841+0000 7fc68eea6700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T01:33:11.845 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:11.841+0000 7fc68eea6700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:33:11.845 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:11.841+0000 7fc68eea6700 10 monclient: _send_mon_message to mon.e at v2:172.21.15.88:3304/0 2023-09-30T01:33:11.845 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:11.841+0000 7fc68eea6700 1 -- 172.21.15.88:0/105199668 --> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fc68811ee40 con 0x7fc688097a40 2023-09-30T01:33:11.845 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:11.841+0000 7fc67dffb700 1 -- 172.21.15.88:0/105199668 <== mon.8 v2:172.21.15.88:3304/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7fc684026070 con 0x7fc688097a40 2023-09-30T01:33:11.845 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:11.841+0000 7fc67dffb700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:33:11.845 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:11.841+0000 7fc67dffb700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:33:11.846 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-09-30T01:33:11.846 INFO:teuthology.orchestra.run.smithi088.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:33:11.847 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:11.845+0000 7fc68eea6700 1 -- 172.21.15.88:0/105199668 >> v2:172.21.15.107:6812/29926 conn(0x7fc66c033360 msgr2=0x7fc66c035810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:33:11.848 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:11.845+0000 7fc68eea6700 1 --2- 172.21.15.88:0/105199668 >> v2:172.21.15.107:6812/29926 conn(0x7fc66c033360 0x7fc66c035810 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fc67800bfd0 tx=0x7fc678017000 comp rx=0 tx=0).stop 2023-09-30T01:33:11.848 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:11.845+0000 7fc68eea6700 10 monclient: shutdown 2023-09-30T01:33:11.848 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:11.845+0000 7fc68eea6700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:33:11.848 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:11.845+0000 7fc68eea6700 1 -- 172.21.15.88:0/105199668 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7fc688097a40 msgr2=0x7fc6880a1b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:33:11.848 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:11.845+0000 7fc68eea6700 1 --2- 172.21.15.88:0/105199668 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7fc688097a40 0x7fc6880a1b10 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fc68400bfc0 tx=0x7fc684008040 comp rx=0 tx=0).stop 2023-09-30T01:33:11.848 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:11.845+0000 7fc68eea6700 1 -- 172.21.15.88:0/105199668 shutdown_connections 2023-09-30T01:33:11.848 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:11.845+0000 7fc68eea6700 1 --2- 172.21.15.88:0/105199668 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7fc6881332e0 0x7fc6880a2590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:11.848 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:11.845+0000 7fc68eea6700 1 --2- 172.21.15.88:0/105199668 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc6881311c0 0x7fc6880a2050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:11.848 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:11.845+0000 7fc68eea6700 1 --2- 172.21.15.88:0/105199668 >> v2:172.21.15.107:6812/29926 conn(0x7fc66c033360 0x7fc66c035810 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:11.848 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:11.845+0000 7fc68eea6700 1 --2- 172.21.15.88:0/105199668 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7fc688097a40 0x7fc6880a1b10 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:11.848 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:11.845+0000 7fc68eea6700 1 -- 172.21.15.88:0/105199668 >> 172.21.15.88:0/105199668 conn(0x7fc68806e5e0 msgr2=0x7fc688131930 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:33:11.849 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:11.845+0000 7fc68eea6700 1 -- 172.21.15.88:0/105199668 shutdown_connections 2023-09-30T01:33:11.849 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:11.845+0000 7fc68eea6700 1 -- 172.21.15.88:0/105199668 wait complete. 2023-09-30T01:33:14.860 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:33:14.971 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:14.969+0000 7fa484a1c700 10 monclient: get_monmap_and_config 2023-09-30T01:33:14.972 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:14.969+0000 7fa484a1c700 10 monclient: build_initial_monmap 2023-09-30T01:33:14.972 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:14.969+0000 7fa484a1c700 10 monclient: monmap: 2023-09-30T01:33:14.972 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2023-09-30T01:33:14.972 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:33:14.972 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:33:14.972037+0000 2023-09-30T01:33:14.972 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:33:14.972037+0000 2023-09-30T01:33:14.972 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2023-09-30T01:33:14.972 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:33:14.972 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a-legacy 2023-09-30T01:33:14.972 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.107:6789/0 mon.noname-f-legacy 2023-09-30T01:33:14.973 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.noname-b 2023-09-30T01:33:14.973 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.noname-g 2023-09-30T01:33:14.973 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.noname-c 2023-09-30T01:33:14.973 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.noname-h 2023-09-30T01:33:14.973 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.noname-d 2023-09-30T01:33:14.973 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.noname-i 2023-09-30T01:33:14.973 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.noname-e 2023-09-30T01:33:14.973 INFO:teuthology.orchestra.run.smithi088.stderr:9: v2:172.21.15.88:3300/0 mon.noname-a 2023-09-30T01:33:14.973 INFO:teuthology.orchestra.run.smithi088.stderr:10: v2:172.21.15.107:3300/0 mon.noname-f 2023-09-30T01:33:14.973 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:33:14.973 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:14.969+0000 7fa484a1c700 1 Processor -- start 2023-09-30T01:33:14.973 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:14.969+0000 7fa484a1c700 1 -- start start 2023-09-30T01:33:14.973 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:14.969+0000 7fa484a1c700 10 monclient: init 2023-09-30T01:33:14.979 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:14.969+0000 7fa484a1c700 10 monclient: _reopen_session rank -1 2023-09-30T01:33:14.979 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:14.969+0000 7fa484a1c700 10 monclient: _add_conns ranks=[4,0,8,7,5,10,3,2,1,6,9] 2023-09-30T01:33:14.979 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:14.969+0000 7fa484a1c700 1 --2- >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7fa480097a40 0x7fa480132700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:33:14.979 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:14.969+0000 7fa484a1c700 10 monclient(hunting): picked mon.noname-c con 0x7fa480097a40 addr [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] 2023-09-30T01:33:14.979 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:14.969+0000 7fa484a1c700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fa4801311c0 addr v1:172.21.15.88:6789/0 2023-09-30T01:33:14.979 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:14.969+0000 7fa484a1c700 1 --2- >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7fa480132c40 0x7fa4801339f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:33:14.980 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:14.969+0000 7fa484a1c700 10 monclient(hunting): picked mon.noname-e con 0x7fa480132c40 addr [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] 2023-09-30T01:33:14.980 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:14.969+0000 7fa484a1c700 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa48004bb20 con 0x7fa4801311c0 2023-09-30T01:33:14.980 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:14.969+0000 7fa484a1c700 10 monclient(hunting): start opening mon connection 2023-09-30T01:33:14.980 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:14.969+0000 7fa484a1c700 1 -- --> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] -- mon_getmap magic: 0 v1 -- 0x7fa480131590 con 0x7fa480097a40 2023-09-30T01:33:14.980 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:14.969+0000 7fa484a1c700 10 monclient(hunting): start opening mon connection 2023-09-30T01:33:14.980 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:14.969+0000 7fa484a1c700 1 -- --> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] -- mon_getmap magic: 0 v1 -- 0x7fa48009a5d0 con 0x7fa480132c40 2023-09-30T01:33:14.980 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:14.969+0000 7fa484a1c700 10 monclient(hunting): _renew_subs 2023-09-30T01:33:14.980 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:14.969+0000 7fa484a1c700 10 monclient(hunting): authenticate will time out at 867.009507s 2023-09-30T01:33:14.980 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:14.969+0000 7fa47ffff700 1 --2- >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7fa480132c40 0x7fa4801339f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:33:14.980 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:14.969+0000 7fa476ffd700 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7fa4801311c0 0x7fa4801332e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:43806/0 (socket says 172.21.15.88:43806) 2023-09-30T01:33:14.980 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:14.969+0000 7fa476ffd700 1 -- 172.21.15.88:0/3840896350 learned_addr learned my addr 172.21.15.88:0/3840896350 (peer_addr_for_me v1:172.21.15.88:0/0) 2023-09-30T01:33:14.980 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:14.969+0000 7fa47ffff700 10 monclient(hunting): get_auth_request con 0x7fa480132c40 auth_method 0 2023-09-30T01:33:14.981 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:14.969+0000 7fa47ffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:33:14.981 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:14.969+0000 7fa47ffff700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:33:14.981 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:14.969+0000 7fa47ffff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:33:14.981 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:14.969+0000 7fa47f7fe700 1 --2- 172.21.15.88:0/3840896350 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7fa480097a40 0x7fa480132700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:33:14.981 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:14.969+0000 7fa47ffff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:33:14.981 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:14.969+0000 7fa47ffff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:33:14.981 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:14.969+0000 7fa47ffff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:33:14.981 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:14.969+0000 7fa47f7fe700 10 monclient(hunting): get_auth_request con 0x7fa480097a40 auth_method 0 2023-09-30T01:33:14.981 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:14.969+0000 7fa47f7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:33:14.981 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:14.969+0000 7fa47f7fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:33:14.981 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:14.969+0000 7fa47f7fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:33:14.981 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:14.969+0000 7fa47f7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:33:14.982 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:14.969+0000 7fa47f7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:33:14.982 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:14.969+0000 7fa47f7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:33:14.982 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:14.969+0000 7fa47effd700 1 -- 172.21.15.88:0/3840896350 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3497093599 0 0) 0x7fa48004bb20 con 0x7fa4801311c0 2023-09-30T01:33:14.982 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:14.969+0000 7fa47effd700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:33:14.982 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:14.969+0000 7fa47effd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:33:14.982 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:14.969+0000 7fa47effd700 10 monclient(hunting): my global_id is 4455 2023-09-30T01:33:14.982 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:14.969+0000 7fa47effd700 1 -- 172.21.15.88:0/3840896350 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa46c003570 con 0x7fa4801311c0 2023-09-30T01:33:14.982 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:14.969+0000 7fa47ffff700 10 monclient(hunting): handle_auth_done global_id 4418 payload 274 2023-09-30T01:33:14.982 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:14.969+0000 7fa47ffff700 1 -- 172.21.15.88:0/3840896350 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7fa480097a40 msgr2=0x7fa480132700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:33:14.982 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:14.969+0000 7fa47ffff700 1 --2- 172.21.15.88:0/3840896350 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7fa480097a40 0x7fa480132700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:14.982 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:14.969+0000 7fa47ffff700 1 -- 172.21.15.88:0/3840896350 >> v1:172.21.15.88:6789/0 conn(0x7fa4801311c0 legacy=0x7fa4801332e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:33:14.982 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:14.969+0000 7fa47ffff700 10 monclient: _finish_hunting 0 2023-09-30T01:33:14.983 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:14.969+0000 7fa47ffff700 1 monclient: found mon.noname-e 2023-09-30T01:33:14.983 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:14.969+0000 7fa47ffff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:33:14.983 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:14.969+0000 7fa47ffff700 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.88:3304/0 2023-09-30T01:33:14.983 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:14.969+0000 7fa47ffff700 1 -- 172.21.15.88:0/3840896350 --> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4801330a0 con 0x7fa480132c40 2023-09-30T01:33:14.983 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:14.969+0000 7fa47ffff700 1 --2- 172.21.15.88:0/3840896350 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7fa480132c40 0x7fa4801339f0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fa478009ec0 tx=0x7fa478009780 comp rx=0 tx=0).ready entity=mon.8 client_cookie=dcc111f49c0302c0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:33:14.983 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:14.969+0000 7fa47effd700 1 -- 172.21.15.88:0/3840896350 <== mon.8 v2:172.21.15.88:3304/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7fa47800e070 con 0x7fa480132c40 2023-09-30T01:33:14.983 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:14.969+0000 7fa47effd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:33:14.983 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:14.969+0000 7fa47effd700 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2023-09-30T01:33:14.983 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:14.969+0000 7fa47effd700 10 monclient: dump: 2023-09-30T01:33:14.983 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:33:14.983 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:33:14.983 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:14.984 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:14.984 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:33:14.984 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:33:14.984 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:33:14.984 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:33:14.984 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:33:14.984 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:33:14.984 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:33:14.984 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:33:14.984 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:33:14.984 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:33:14.984 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:33:14.984 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:33:14.985 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:14.969+0000 7fa47effd700 10 monclient: _finish_auth 0 2023-09-30T01:33:14.985 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:14.969+0000 7fa47effd700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:33:14.985 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:14.969+0000 7fa484a1c700 5 monclient: authenticate success, global_id 4418 2023-09-30T01:33:14.985 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:14.969+0000 7fa47effd700 1 -- 172.21.15.88:0/3840896350 <== mon.8 v2:172.21.15.88:3304/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa478012430 con 0x7fa480132c40 2023-09-30T01:33:14.985 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:14.969+0000 7fa484a1c700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:33:14.985 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:14.969+0000 7fa47effd700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:33:14.985 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:14.969+0000 7fa47effd700 1 -- 172.21.15.88:0/3840896350 <== mon.8 v2:172.21.15.88:3304/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7fa478012a20 con 0x7fa480132c40 2023-09-30T01:33:14.985 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:14.969+0000 7fa484a1c700 10 monclient: get_monmap_and_config success 2023-09-30T01:33:14.985 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:14.969+0000 7fa484a1c700 4 set_mon_vals no callback set 2023-09-30T01:33:14.985 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:14.969+0000 7fa47effd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:33:14.985 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:14.969+0000 7fa47effd700 10 monclient: got monmap 1 from mon.e (according to old e1) 2023-09-30T01:33:14.985 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:14.969+0000 7fa47effd700 10 monclient: dump: 2023-09-30T01:33:14.986 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:33:14.986 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:33:14.986 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:14.986 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:14.986 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:33:14.986 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:33:14.986 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:33:14.986 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:33:14.986 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:33:14.986 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:33:14.986 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:33:14.986 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:33:14.986 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:33:14.987 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:33:14.987 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:33:14.987 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:33:14.987 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:14.973+0000 7fa484a1c700 10 monclient: shutdown 2023-09-30T01:33:14.987 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:14.973+0000 7fa484a1c700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:33:14.987 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:14.973+0000 7fa484a1c700 1 -- 172.21.15.88:0/3840896350 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7fa480132c40 msgr2=0x7fa4801339f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:33:14.987 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:14.973+0000 7fa484a1c700 1 --2- 172.21.15.88:0/3840896350 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7fa480132c40 0x7fa4801339f0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fa478009ec0 tx=0x7fa478009780 comp rx=0 tx=0).stop 2023-09-30T01:33:14.987 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:14.973+0000 7fa484a1c700 1 -- 172.21.15.88:0/3840896350 shutdown_connections 2023-09-30T01:33:14.987 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:14.973+0000 7fa484a1c700 1 --2- 172.21.15.88:0/3840896350 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7fa480132c40 0x7fa4801339f0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:14.987 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:14.973+0000 7fa484a1c700 1 --2- 172.21.15.88:0/3840896350 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7fa480097a40 0x7fa480132700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:14.987 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:14.973+0000 7fa484a1c700 1 -- 172.21.15.88:0/3840896350 >> 172.21.15.88:0/3840896350 conn(0x7fa48006e5e0 msgr2=0x7fa48006e9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:33:14.987 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:14.973+0000 7fa484a1c700 1 -- 172.21.15.88:0/3840896350 shutdown_connections 2023-09-30T01:33:14.988 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:14.973+0000 7fa484a1c700 1 -- 172.21.15.88:0/3840896350 wait complete. 2023-09-30T01:33:14.988 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:14.973+0000 7fa484a1c700 10 monclient: build_initial_monmap 2023-09-30T01:33:14.988 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:14.973+0000 7fa484a1c700 10 monclient: monmap: 2023-09-30T01:33:14.988 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2023-09-30T01:33:14.988 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:33:14.988 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 0.000000 2023-09-30T01:33:14.988 INFO:teuthology.orchestra.run.smithi088.stderr:created 0.000000 2023-09-30T01:33:14.988 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2023-09-30T01:33:14.988 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 1 2023-09-30T01:33:14.988 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.noname-a 2023-09-30T01:33:14.988 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.noname-f 2023-09-30T01:33:14.988 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.noname-b 2023-09-30T01:33:14.988 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.noname-g 2023-09-30T01:33:14.989 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.noname-c 2023-09-30T01:33:14.989 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.noname-h 2023-09-30T01:33:14.989 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.noname-d 2023-09-30T01:33:14.989 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.noname-i 2023-09-30T01:33:14.989 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.noname-e 2023-09-30T01:33:14.989 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:33:14.989 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:14.973+0000 7fa484a1c700 1 Processor -- start 2023-09-30T01:33:14.989 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:14.973+0000 7fa484a1c700 1 -- start start 2023-09-30T01:33:14.989 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:14.973+0000 7fa484a1c700 10 monclient: init 2023-09-30T01:33:14.989 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:14.973+0000 7fa484a1c700 10 monclient: _reopen_session rank -1 2023-09-30T01:33:14.989 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:14.973+0000 7fa484a1c700 10 monclient: _add_conns ranks=[5,2,8,3,0,4,1,7,6] 2023-09-30T01:33:14.989 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:14.973+0000 7fa484a1c700 1 --2- >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7fa480097a40 0x7fa4800a1a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:33:14.990 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:14.973+0000 7fa484a1c700 10 monclient(hunting): picked mon.noname-h con 0x7fa480097a40 addr [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] 2023-09-30T01:33:14.990 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:14.973+0000 7fa484a1c700 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa4801311c0 0x7fa4800a1fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:33:14.990 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:14.973+0000 7fa484a1c700 10 monclient(hunting): picked mon.noname-b con 0x7fa4801311c0 addr [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] 2023-09-30T01:33:14.990 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:14.973+0000 7fa484a1c700 1 --2- >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7fa480132c40 0x7fa4800a2510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:33:14.990 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:14.973+0000 7fa484a1c700 10 monclient(hunting): picked mon.noname-e con 0x7fa480132c40 addr [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] 2023-09-30T01:33:14.990 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:14.973+0000 7fa484a1c700 10 monclient(hunting): start opening mon connection 2023-09-30T01:33:14.990 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:14.973+0000 7fa484a1c700 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa480099c30 con 0x7fa4801311c0 2023-09-30T01:33:14.990 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:14.973+0000 7fa484a1c700 10 monclient(hunting): start opening mon connection 2023-09-30T01:33:14.990 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:14.973+0000 7fa484a1c700 1 -- --> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] -- mon_getmap magic: 0 v1 -- 0x7fa4800996b0 con 0x7fa480097a40 2023-09-30T01:33:14.990 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:14.973+0000 7fa484a1c700 10 monclient(hunting): start opening mon connection 2023-09-30T01:33:14.990 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:14.973+0000 7fa484a1c700 1 -- --> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] -- mon_getmap magic: 0 v1 -- 0x7fa48011ed70 con 0x7fa480132c40 2023-09-30T01:33:14.990 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:14.973+0000 7fa484a1c700 10 monclient(hunting): _renew_subs 2023-09-30T01:33:14.991 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:14.973+0000 7fa484a1c700 10 monclient(hunting): authenticate will time out at 867.014172s 2023-09-30T01:33:14.991 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:14.973+0000 7fa476ffd700 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa4801311c0 0x7fa4800a1fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:33:14.991 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:14.973+0000 7fa47ffff700 1 --2- >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7fa480132c40 0x7fa4800a2510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:33:14.991 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:14.973+0000 7fa476ffd700 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa4801311c0 0x7fa4800a1fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3301/0 says I am v2:172.21.15.88:41980/0 (socket says 172.21.15.88:41980) 2023-09-30T01:33:14.991 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:14.973+0000 7fa476ffd700 1 -- 172.21.15.88:0/2539995042 learned_addr learned my addr 172.21.15.88:0/2539995042 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-09-30T01:33:14.991 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:14.973+0000 7fa476ffd700 10 monclient(hunting): get_auth_request con 0x7fa4801311c0 auth_method 0 2023-09-30T01:33:14.991 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:14.973+0000 7fa476ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:33:14.991 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:14.973+0000 7fa476ffd700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:33:14.991 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:14.973+0000 7fa476ffd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:33:14.991 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:14.973+0000 7fa47ffff700 10 monclient(hunting): get_auth_request con 0x7fa480132c40 auth_method 0 2023-09-30T01:33:14.991 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:14.973+0000 7fa47ffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:33:14.991 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:14.973+0000 7fa47ffff700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:33:14.992 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:14.973+0000 7fa47ffff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:33:14.992 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:14.973+0000 7fa47f7fe700 1 --2- 172.21.15.88:0/2539995042 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7fa480097a40 0x7fa4800a1a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:33:14.992 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:14.973+0000 7fa476ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:33:14.992 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:14.973+0000 7fa476ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:33:14.992 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:14.973+0000 7fa476ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:33:14.992 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:14.973+0000 7fa47f7fe700 10 monclient(hunting): get_auth_request con 0x7fa480097a40 auth_method 0 2023-09-30T01:33:14.992 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:14.973+0000 7fa47f7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:33:14.992 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:14.973+0000 7fa47f7fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:33:14.992 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:14.973+0000 7fa47f7fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:33:14.992 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:14.973+0000 7fa47f7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:33:14.992 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:14.973+0000 7fa47f7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:33:14.992 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:14.973+0000 7fa47f7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:33:14.993 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:14.973+0000 7fa476ffd700 10 monclient(hunting): handle_auth_done global_id 4349 payload 824 2023-09-30T01:33:14.993 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:14.973+0000 7fa476ffd700 1 -- 172.21.15.88:0/2539995042 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7fa480132c40 msgr2=0x7fa4800a2510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:33:14.993 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:14.973+0000 7fa476ffd700 1 --2- 172.21.15.88:0/2539995042 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7fa480132c40 0x7fa4800a2510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:14.993 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:14.973+0000 7fa476ffd700 1 -- 172.21.15.88:0/2539995042 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7fa480097a40 msgr2=0x7fa4800a1a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:33:14.993 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:14.973+0000 7fa476ffd700 1 --2- 172.21.15.88:0/2539995042 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7fa480097a40 0x7fa4800a1a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:14.993 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:14.973+0000 7fa476ffd700 10 monclient: _finish_hunting 0 2023-09-30T01:33:14.993 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:14.973+0000 7fa476ffd700 1 monclient: found mon.noname-b 2023-09-30T01:33:14.993 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:14.973+0000 7fa476ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:33:14.993 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:14.973+0000 7fa476ffd700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.88:3301/0 2023-09-30T01:33:14.993 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:14.973+0000 7fa476ffd700 1 -- 172.21.15.88:0/2539995042 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa478009a20 con 0x7fa4801311c0 2023-09-30T01:33:14.993 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:14.973+0000 7fa47ffff700 1 --2- 172.21.15.88:0/2539995042 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7fa480132c40 0x7fa4800a2510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-30T01:33:14.993 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:14.973+0000 7fa476ffd700 1 --2- 172.21.15.88:0/2539995042 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa4801311c0 0x7fa4800a1fd0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fa4700011e0 tx=0x7fa470007df0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:33:14.993 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:14.973+0000 7fa477fff700 1 -- 172.21.15.88:0/2539995042 <== mon.2 v2:172.21.15.88:3301/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7fa4700070d0 con 0x7fa4801311c0 2023-09-30T01:33:14.994 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:14.973+0000 7fa477fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:33:14.994 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:14.973+0000 7fa477fff700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T01:33:14.994 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:14.973+0000 7fa477fff700 10 monclient: dump: 2023-09-30T01:33:14.994 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:33:14.994 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:33:14.994 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:14.994 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:14.994 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:33:14.994 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:33:14.994 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:33:14.994 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:33:14.994 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:33:14.995 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:33:14.995 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:33:14.995 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:33:14.995 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:33:14.995 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:33:14.995 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:33:14.995 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:33:14.995 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:14.973+0000 7fa477fff700 10 monclient: _finish_auth 0 2023-09-30T01:33:14.995 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:14.973+0000 7fa477fff700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:33:14.995 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:14.973+0000 7fa484a1c700 5 monclient: authenticate success, global_id 4349 2023-09-30T01:33:14.995 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:14.973+0000 7fa477fff700 1 -- 172.21.15.88:0/2539995042 <== mon.2 v2:172.21.15.88:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa470007290 con 0x7fa4801311c0 2023-09-30T01:33:14.995 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:14.973+0000 7fa484a1c700 10 monclient: _renew_subs 2023-09-30T01:33:14.996 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:14.973+0000 7fa484a1c700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.88:3301/0 2023-09-30T01:33:14.996 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:14.973+0000 7fa484a1c700 1 -- 172.21.15.88:0/2539995042 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa480130db0 con 0x7fa4801311c0 2023-09-30T01:33:14.996 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:14.973+0000 7fa477fff700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:33:14.996 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:14.973+0000 7fa477fff700 1 -- 172.21.15.88:0/2539995042 <== mon.2 v2:172.21.15.88:3301/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7fa470011360 con 0x7fa4801311c0 2023-09-30T01:33:14.996 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:14.973+0000 7fa477fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:33:14.996 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:14.973+0000 7fa477fff700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T01:33:14.996 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:14.973+0000 7fa477fff700 10 monclient: dump: 2023-09-30T01:33:14.996 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:33:14.996 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:33:14.996 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:14.996 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:14.996 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:33:14.996 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:33:14.997 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:33:14.997 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:33:14.997 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:33:14.997 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:33:14.997 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:33:14.997 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:33:14.997 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:33:14.997 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:33:14.997 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:33:14.997 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:33:14.997 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:14.973+0000 7fa47dffb700 4 set_mon_vals no callback set 2023-09-30T01:33:14.997 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:14.973+0000 7fa484a1c700 10 monclient: _renew_subs 2023-09-30T01:33:14.997 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:14.973+0000 7fa484a1c700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.88:3301/0 2023-09-30T01:33:14.998 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:14.973+0000 7fa484a1c700 1 -- 172.21.15.88:0/2539995042 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa48011de10 con 0x7fa4801311c0 2023-09-30T01:33:14.999 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:14.977+0000 7fa477fff700 1 -- 172.21.15.88:0/2539995042 <== mon.2 v2:172.21.15.88:3301/0 4 ==== mgrmap(e 5) v1 ==== 40144+0+0 (secure 0 0 0) 0x7fa470005b30 con 0x7fa4801311c0 2023-09-30T01:33:14.999 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:14.977+0000 7fa477fff700 1 --2- 172.21.15.88:0/2539995042 >> v2:172.21.15.107:6812/29926 conn(0x7fa460033360 0x7fa460035810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:33:14.999 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:14.977+0000 7fa484a1c700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:33:14.999 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:14.977+0000 7fa484a1c700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:33:14.999 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:14.977+0000 7fa484a1c700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.88:3301/0 2023-09-30T01:33:14.999 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:14.977+0000 7fa477fff700 1 -- 172.21.15.88:0/2539995042 <== mon.2 v2:172.21.15.88:3301/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 3502+0+0 (secure 0 0 0) 0x7fa470042500 con 0x7fa4801311c0 2023-09-30T01:33:14.999 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:14.977+0000 7fa484a1c700 1 -- 172.21.15.88:0/2539995042 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa48011f1f0 con 0x7fa4801311c0 2023-09-30T01:33:15.000 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:14.977+0000 7fa47f7fe700 1 --2- 172.21.15.88:0/2539995042 >> v2:172.21.15.107:6812/29926 conn(0x7fa460033360 0x7fa460035810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:33:15.000 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:14.977+0000 7fa47f7fe700 10 monclient: get_auth_request con 0x7fa460033360 auth_method 0 2023-09-30T01:33:15.000 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:14.977+0000 7fa47f7fe700 1 --2- 172.21.15.88:0/2539995042 >> v2:172.21.15.107:6812/29926 conn(0x7fa460033360 0x7fa460035810 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fa468009d10 tx=0x7fa4680091a0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:33:15.000 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:14.985+0000 7fa477fff700 1 -- 172.21.15.88:0/2539995042 <== mon.2 v2:172.21.15.88:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fa470017070 con 0x7fa4801311c0 2023-09-30T01:33:15.000 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:14.985+0000 7fa477fff700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:33:15.000 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:14.985+0000 7fa477fff700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:33:15.182 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:15.177+0000 7fa484a1c700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T01:33:15.183 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:15.177+0000 7fa484a1c700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:33:15.183 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:15.177+0000 7fa484a1c700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.88:3301/0 2023-09-30T01:33:15.183 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:15.177+0000 7fa484a1c700 1 -- 172.21.15.88:0/2539995042 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fa48009e7b0 con 0x7fa4801311c0 2023-09-30T01:33:15.183 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:15.181+0000 7fa477fff700 1 -- 172.21.15.88:0/2539995042 <== mon.2 v2:172.21.15.88:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7fa47000e470 con 0x7fa4801311c0 2023-09-30T01:33:15.183 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:15.181+0000 7fa477fff700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:33:15.184 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:15.181+0000 7fa477fff700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:33:15.184 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-09-30T01:33:15.184 INFO:teuthology.orchestra.run.smithi088.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:33:15.185 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:15.181+0000 7fa484a1c700 1 -- 172.21.15.88:0/2539995042 >> v2:172.21.15.107:6812/29926 conn(0x7fa460033360 msgr2=0x7fa460035810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:33:15.186 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:15.181+0000 7fa484a1c700 1 --2- 172.21.15.88:0/2539995042 >> v2:172.21.15.107:6812/29926 conn(0x7fa460033360 0x7fa460035810 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fa468009d10 tx=0x7fa4680091a0 comp rx=0 tx=0).stop 2023-09-30T01:33:15.186 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:15.181+0000 7fa484a1c700 10 monclient: shutdown 2023-09-30T01:33:15.186 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:15.181+0000 7fa484a1c700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:33:15.186 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:15.181+0000 7fa484a1c700 1 -- 172.21.15.88:0/2539995042 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa4801311c0 msgr2=0x7fa4800a1fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:33:15.186 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:15.181+0000 7fa484a1c700 1 --2- 172.21.15.88:0/2539995042 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa4801311c0 0x7fa4800a1fd0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fa4700011e0 tx=0x7fa470007df0 comp rx=0 tx=0).stop 2023-09-30T01:33:15.186 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:15.181+0000 7fa484a1c700 1 -- 172.21.15.88:0/2539995042 shutdown_connections 2023-09-30T01:33:15.186 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:15.181+0000 7fa484a1c700 1 --2- 172.21.15.88:0/2539995042 >> v2:172.21.15.107:6812/29926 conn(0x7fa460033360 0x7fa460035810 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:15.186 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:15.181+0000 7fa484a1c700 1 --2- 172.21.15.88:0/2539995042 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7fa480132c40 0x7fa4800a2510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:15.186 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:15.181+0000 7fa484a1c700 1 --2- 172.21.15.88:0/2539995042 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa4801311c0 0x7fa4800a1fd0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:15.186 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:15.181+0000 7fa484a1c700 1 --2- 172.21.15.88:0/2539995042 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7fa480097a40 0x7fa4800a1a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:15.186 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:15.181+0000 7fa484a1c700 1 -- 172.21.15.88:0/2539995042 >> 172.21.15.88:0/2539995042 conn(0x7fa48006e5e0 msgr2=0x7fa4800a4bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:33:15.187 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:15.181+0000 7fa484a1c700 1 -- 172.21.15.88:0/2539995042 shutdown_connections 2023-09-30T01:33:15.187 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:15.181+0000 7fa484a1c700 1 -- 172.21.15.88:0/2539995042 wait complete. 2023-09-30T01:33:18.202 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:33:18.313 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:18.309+0000 7f7344257700 10 monclient: get_monmap_and_config 2023-09-30T01:33:18.313 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:18.309+0000 7f7344257700 10 monclient: build_initial_monmap 2023-09-30T01:33:18.313 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:18.309+0000 7f7344257700 10 monclient: monmap: 2023-09-30T01:33:18.314 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2023-09-30T01:33:18.314 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:33:18.314 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:33:18.314071+0000 2023-09-30T01:33:18.314 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:33:18.314071+0000 2023-09-30T01:33:18.314 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2023-09-30T01:33:18.314 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:33:18.314 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a-legacy 2023-09-30T01:33:18.314 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.107:6789/0 mon.noname-f-legacy 2023-09-30T01:33:18.314 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.noname-b 2023-09-30T01:33:18.314 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.noname-g 2023-09-30T01:33:18.314 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.noname-c 2023-09-30T01:33:18.315 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.noname-h 2023-09-30T01:33:18.315 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.noname-d 2023-09-30T01:33:18.315 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.noname-i 2023-09-30T01:33:18.315 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.noname-e 2023-09-30T01:33:18.315 INFO:teuthology.orchestra.run.smithi088.stderr:9: v2:172.21.15.88:3300/0 mon.noname-a 2023-09-30T01:33:18.315 INFO:teuthology.orchestra.run.smithi088.stderr:10: v2:172.21.15.107:3300/0 mon.noname-f 2023-09-30T01:33:18.315 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:33:18.317 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:18.309+0000 7f7344257700 1 Processor -- start 2023-09-30T01:33:18.318 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:18.309+0000 7f7344257700 1 -- start start 2023-09-30T01:33:18.318 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:18.309+0000 7f7344257700 10 monclient: init 2023-09-30T01:33:18.318 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:18.313+0000 7f7344257700 10 monclient: _reopen_session rank -1 2023-09-30T01:33:18.318 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:18.313+0000 7f7344257700 10 monclient: _add_conns ranks=[4,3,1,7,5,10,8,9,6,2,0] 2023-09-30T01:33:18.318 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:18.313+0000 7f7344257700 1 --2- >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f733c062c70 0x7f733c096680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:33:18.318 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:18.313+0000 7f7344257700 10 monclient(hunting): picked mon.noname-c con 0x7f733c062c70 addr [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] 2023-09-30T01:33:18.318 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:18.313+0000 7f7344257700 1 --2- >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7f733c098f50 0x7f733c09ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:33:18.318 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:18.313+0000 7f7344257700 10 monclient(hunting): picked mon.noname-g con 0x7f733c098f50 addr [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] 2023-09-30T01:33:18.318 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:18.313+0000 7f7344257700 10 monclient(hunting): picked mon.noname-f-legacy con 0x7f733c099320 addr v1:172.21.15.107:6789/0 2023-09-30T01:33:18.318 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:18.313+0000 7f7344257700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f733c05ecb0 con 0x7f733c099320 2023-09-30T01:33:18.318 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:18.313+0000 7f7344257700 10 monclient(hunting): start opening mon connection 2023-09-30T01:33:18.319 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:18.313+0000 7f7344257700 1 -- --> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f733c096bc0 con 0x7f733c098f50 2023-09-30T01:33:18.319 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:18.313+0000 7f7344257700 10 monclient(hunting): start opening mon connection 2023-09-30T01:33:18.319 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:18.313+0000 7f7344257700 1 -- --> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] -- mon_getmap magic: 0 v1 -- 0x7f733c097460 con 0x7f733c062c70 2023-09-30T01:33:18.319 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:18.313+0000 7f7344257700 10 monclient(hunting): _renew_subs 2023-09-30T01:33:18.319 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:18.313+0000 7f7344257700 10 monclient(hunting): authenticate will time out at 870.352683s 2023-09-30T01:33:18.319 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:18.313+0000 7f7341ff3700 1 --2- >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f733c062c70 0x7f733c096680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:33:18.319 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:18.313+0000 7f7341ff3700 1 --2- >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f733c062c70 0x7f733c096680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3302/0 says I am v2:172.21.15.88:34832/0 (socket says 172.21.15.88:34832) 2023-09-30T01:33:18.319 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:18.313+0000 7f7341ff3700 1 -- 172.21.15.88:0/2514836571 learned_addr learned my addr 172.21.15.88:0/2514836571 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-09-30T01:33:18.319 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:18.313+0000 7f7341ff3700 10 monclient(hunting): get_auth_request con 0x7f733c062c70 auth_method 0 2023-09-30T01:33:18.319 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:18.313+0000 7f7341ff3700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:33:18.319 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:18.313+0000 7f7341ff3700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:33:18.319 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:18.313+0000 7f7341ff3700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:33:18.320 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:18.313+0000 7f73417f2700 1 --2- 172.21.15.88:0/2514836571 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7f733c098f50 0x7f733c09ddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:33:18.320 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:18.313+0000 7f73417f2700 10 monclient(hunting): get_auth_request con 0x7f733c098f50 auth_method 0 2023-09-30T01:33:18.320 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:18.313+0000 7f73417f2700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:33:18.320 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:18.313+0000 7f73417f2700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:33:18.320 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:18.313+0000 7f73417f2700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:33:18.320 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:18.313+0000 7f7341ff3700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:33:18.320 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:18.313+0000 7f7341ff3700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:33:18.320 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:18.313+0000 7f7341ff3700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:33:18.320 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:18.313+0000 7f73417f2700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:33:18.320 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:18.313+0000 7f73417f2700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:33:18.320 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:18.313+0000 7f73417f2700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:33:18.321 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:18.313+0000 7f7341ff3700 10 monclient(hunting): handle_auth_done global_id 4387 payload 274 2023-09-30T01:33:18.321 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:18.313+0000 7f7341ff3700 1 -- 172.21.15.88:0/2514836571 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7f733c098f50 msgr2=0x7f733c09ddb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:33:18.321 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:18.313+0000 7f7341ff3700 1 --2- 172.21.15.88:0/2514836571 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7f733c098f50 0x7f733c09ddb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:18.321 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:18.313+0000 7f7341ff3700 1 -- 172.21.15.88:0/2514836571 >> v1:172.21.15.107:6789/0 conn(0x7f733c099320 legacy=0x7f733c09e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:33:18.321 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:18.313+0000 7f7341ff3700 10 monclient: _finish_hunting 0 2023-09-30T01:33:18.321 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:18.313+0000 7f7341ff3700 1 monclient: found mon.noname-c 2023-09-30T01:33:18.321 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:18.313+0000 7f7341ff3700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:33:18.321 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:18.313+0000 7f7341ff3700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.88:3302/0 2023-09-30T01:33:18.322 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:18.313+0000 7f7341ff3700 1 -- 172.21.15.88:0/2514836571 --> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f733c063040 con 0x7f733c062c70 2023-09-30T01:33:18.322 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:18.313+0000 7f7341ff3700 1 --2- 172.21.15.88:0/2514836571 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f733c062c70 0x7f733c096680 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f73380099e0 tx=0x7f7338009250 comp rx=0 tx=0).ready entity=mon.4 client_cookie=dbb72be03e6a0242 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:33:18.322 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:18.313+0000 7f7340ff1700 1 -- 172.21.15.88:0/2514836571 <== mon.4 v2:172.21.15.88:3302/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f733800bce0 con 0x7f733c062c70 2023-09-30T01:33:18.322 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:18.313+0000 7f7340ff1700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:33:18.322 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:18.313+0000 7f7340ff1700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T01:33:18.322 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:18.313+0000 7f7340ff1700 10 monclient: dump: 2023-09-30T01:33:18.323 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:33:18.323 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:33:18.323 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:18.323 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:18.323 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:33:18.323 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:33:18.323 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:33:18.323 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:33:18.323 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:33:18.323 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:33:18.323 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:33:18.324 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:33:18.324 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:33:18.324 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:33:18.324 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:33:18.324 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:33:18.324 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:18.313+0000 7f7340ff1700 10 monclient: _finish_auth 0 2023-09-30T01:33:18.324 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:18.313+0000 7f7340ff1700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:33:18.324 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:18.313+0000 7f7340ff1700 1 -- 172.21.15.88:0/2514836571 <== mon.4 v2:172.21.15.88:3302/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f733800be40 con 0x7f733c062c70 2023-09-30T01:33:18.324 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:18.313+0000 7f7344257700 5 monclient: authenticate success, global_id 4387 2023-09-30T01:33:18.324 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:18.313+0000 7f7344257700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:33:18.324 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:18.313+0000 7f7340ff1700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:33:18.324 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:18.313+0000 7f7340ff1700 1 -- 172.21.15.88:0/2514836571 <== mon.4 v2:172.21.15.88:3302/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f73380118a0 con 0x7f733c062c70 2023-09-30T01:33:18.324 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:18.313+0000 7f7344257700 10 monclient: get_monmap_and_config success 2023-09-30T01:33:18.325 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:18.313+0000 7f7344257700 4 set_mon_vals no callback set 2023-09-30T01:33:18.325 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:18.313+0000 7f7340ff1700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:33:18.325 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:18.313+0000 7f7340ff1700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T01:33:18.325 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:18.313+0000 7f7340ff1700 10 monclient: dump: 2023-09-30T01:33:18.325 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:33:18.325 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:33:18.325 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:18.325 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:18.325 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:33:18.325 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:33:18.325 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:33:18.326 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:33:18.326 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:33:18.326 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:33:18.326 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:33:18.326 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:33:18.326 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:33:18.326 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:33:18.326 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:33:18.326 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:33:18.331 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:18.313+0000 7f7344257700 10 monclient: shutdown 2023-09-30T01:33:18.331 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:18.313+0000 7f7344257700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:33:18.331 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:18.313+0000 7f7344257700 1 -- 172.21.15.88:0/2514836571 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f733c062c70 msgr2=0x7f733c096680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:33:18.331 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:18.313+0000 7f7344257700 1 --2- 172.21.15.88:0/2514836571 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f733c062c70 0x7f733c096680 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f73380099e0 tx=0x7f7338009250 comp rx=0 tx=0).stop 2023-09-30T01:33:18.332 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:18.313+0000 7f7344257700 1 -- 172.21.15.88:0/2514836571 shutdown_connections 2023-09-30T01:33:18.332 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:18.313+0000 7f7344257700 1 --2- 172.21.15.88:0/2514836571 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7f733c098f50 0x7f733c09ddb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:18.332 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:18.313+0000 7f7344257700 1 --2- 172.21.15.88:0/2514836571 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f733c062c70 0x7f733c096680 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:18.332 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:18.313+0000 7f7344257700 1 -- 172.21.15.88:0/2514836571 >> 172.21.15.88:0/2514836571 conn(0x7f733c0a3440 msgr2=0x7f733c0a3840 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:33:18.332 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:18.313+0000 7f7344257700 1 -- 172.21.15.88:0/2514836571 shutdown_connections 2023-09-30T01:33:18.332 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:18.313+0000 7f7344257700 1 -- 172.21.15.88:0/2514836571 wait complete. 2023-09-30T01:33:18.332 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:18.317+0000 7f7344257700 10 monclient: build_initial_monmap 2023-09-30T01:33:18.332 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:18.317+0000 7f7344257700 10 monclient: monmap: 2023-09-30T01:33:18.332 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2023-09-30T01:33:18.332 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:33:18.332 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 0.000000 2023-09-30T01:33:18.332 INFO:teuthology.orchestra.run.smithi088.stderr:created 0.000000 2023-09-30T01:33:18.333 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2023-09-30T01:33:18.333 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 1 2023-09-30T01:33:18.333 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.noname-a 2023-09-30T01:33:18.333 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.noname-f 2023-09-30T01:33:18.333 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.noname-b 2023-09-30T01:33:18.333 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.noname-g 2023-09-30T01:33:18.333 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.noname-c 2023-09-30T01:33:18.333 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.noname-h 2023-09-30T01:33:18.333 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.noname-d 2023-09-30T01:33:18.333 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.noname-i 2023-09-30T01:33:18.333 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.noname-e 2023-09-30T01:33:18.333 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:33:18.334 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:18.317+0000 7f7344257700 1 Processor -- start 2023-09-30T01:33:18.334 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:18.317+0000 7f7344257700 1 -- start start 2023-09-30T01:33:18.334 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:18.317+0000 7f7344257700 10 monclient: init 2023-09-30T01:33:18.334 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:18.317+0000 7f7344257700 10 monclient: _reopen_session rank -1 2023-09-30T01:33:18.334 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:18.317+0000 7f7344257700 10 monclient: _add_conns ranks=[0,6,3,5,2,7,1,8,4] 2023-09-30T01:33:18.334 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:18.317+0000 7f7344257700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f733c062c70 0x7f733c136820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:33:18.334 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:18.317+0000 7f7344257700 10 monclient(hunting): picked mon.noname-a con 0x7f733c062c70 addr [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] 2023-09-30T01:33:18.334 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:18.317+0000 7f7344257700 1 --2- >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7f733c098f50 0x7f733c136d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:33:18.334 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:18.317+0000 7f7344257700 10 monclient(hunting): picked mon.noname-d con 0x7f733c098f50 addr [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] 2023-09-30T01:33:18.334 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:18.317+0000 7f7344257700 1 --2- >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7f733c099320 0x7f733c08e3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:33:18.334 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:18.317+0000 7f7344257700 10 monclient(hunting): picked mon.noname-g con 0x7f733c099320 addr [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] 2023-09-30T01:33:18.334 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:18.317+0000 7f7344257700 10 monclient(hunting): start opening mon connection 2023-09-30T01:33:18.335 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:18.317+0000 7f7344257700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f733c0916e0 con 0x7f733c062c70 2023-09-30T01:33:18.335 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:18.317+0000 7f7344257700 10 monclient(hunting): start opening mon connection 2023-09-30T01:33:18.335 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:18.317+0000 7f7344257700 1 -- --> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f733c091c10 con 0x7f733c099320 2023-09-30T01:33:18.335 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:18.317+0000 7f7344257700 10 monclient(hunting): start opening mon connection 2023-09-30T01:33:18.335 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:18.317+0000 7f7344257700 1 -- --> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] -- mon_getmap magic: 0 v1 -- 0x7f733c09f780 con 0x7f733c098f50 2023-09-30T01:33:18.335 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:18.317+0000 7f7344257700 10 monclient(hunting): _renew_subs 2023-09-30T01:33:18.335 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:18.317+0000 7f7344257700 10 monclient(hunting): authenticate will time out at 870.358535s 2023-09-30T01:33:18.335 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:18.317+0000 7f7341ff3700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f733c062c70 0x7f733c136820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:33:18.335 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:18.317+0000 7f73417f2700 1 --2- >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7f733c098f50 0x7f733c136d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:33:18.335 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:18.317+0000 7f7341ff3700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f733c062c70 0x7f733c136820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:32952/0 (socket says 172.21.15.88:32952) 2023-09-30T01:33:18.335 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:18.317+0000 7f7341ff3700 1 -- 172.21.15.88:0/2119774486 learned_addr learned my addr 172.21.15.88:0/2119774486 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-09-30T01:33:18.335 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:18.317+0000 7f7341ff3700 10 monclient(hunting): get_auth_request con 0x7f733c062c70 auth_method 0 2023-09-30T01:33:18.336 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:18.317+0000 7f7341ff3700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:33:18.336 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:18.317+0000 7f7341ff3700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:33:18.336 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:18.317+0000 7f7341ff3700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:33:18.336 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:18.317+0000 7f73417f2700 10 monclient(hunting): get_auth_request con 0x7f733c098f50 auth_method 0 2023-09-30T01:33:18.336 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:18.317+0000 7f73417f2700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:33:18.336 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:18.317+0000 7f73417f2700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:33:18.336 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:18.317+0000 7f73417f2700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:33:18.336 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:18.317+0000 7f73427f4700 1 --2- 172.21.15.88:0/2119774486 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7f733c099320 0x7f733c08e3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:33:18.336 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:18.317+0000 7f73427f4700 10 monclient(hunting): get_auth_request con 0x7f733c099320 auth_method 0 2023-09-30T01:33:18.336 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:18.317+0000 7f73427f4700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:33:18.336 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:18.317+0000 7f73427f4700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:33:18.336 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:18.317+0000 7f73427f4700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:33:18.337 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:18.317+0000 7f73417f2700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:33:18.337 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:18.317+0000 7f73417f2700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:33:18.337 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:18.317+0000 7f73417f2700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:33:18.337 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:18.317+0000 7f7341ff3700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:33:18.337 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:18.317+0000 7f7341ff3700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:33:18.337 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:18.317+0000 7f7341ff3700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:33:18.337 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:18.317+0000 7f73427f4700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:33:18.337 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:18.317+0000 7f73427f4700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:33:18.337 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:18.317+0000 7f73427f4700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:33:18.337 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:18.317+0000 7f7341ff3700 10 monclient(hunting): handle_auth_done global_id 4464 payload 824 2023-09-30T01:33:18.337 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:18.317+0000 7f7341ff3700 1 -- 172.21.15.88:0/2119774486 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7f733c098f50 msgr2=0x7f733c136d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:33:18.337 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:18.317+0000 7f7341ff3700 1 --2- 172.21.15.88:0/2119774486 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7f733c098f50 0x7f733c136d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:18.338 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:18.317+0000 7f7341ff3700 1 -- 172.21.15.88:0/2119774486 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7f733c099320 msgr2=0x7f733c08e3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:33:18.338 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:18.317+0000 7f7341ff3700 1 --2- 172.21.15.88:0/2119774486 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7f733c099320 0x7f733c08e3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:18.338 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:18.317+0000 7f7341ff3700 10 monclient: _finish_hunting 0 2023-09-30T01:33:18.338 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:18.317+0000 7f7341ff3700 1 monclient: found mon.noname-a 2023-09-30T01:33:18.338 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:18.317+0000 7f7341ff3700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:33:18.338 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:18.317+0000 7f7341ff3700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.88:3300/0 2023-09-30T01:33:18.338 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:18.317+0000 7f7341ff3700 1 -- 172.21.15.88:0/2119774486 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73380094f0 con 0x7f733c062c70 2023-09-30T01:33:18.338 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:18.317+0000 7f73427f4700 1 --2- 172.21.15.88:0/2119774486 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7f733c099320 0x7f733c08e3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:33:18.338 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:18.317+0000 7f73417f2700 1 --2- 172.21.15.88:0/2119774486 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7f733c098f50 0x7f733c136d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:33:18.338 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:18.317+0000 7f7341ff3700 1 --2- 172.21.15.88:0/2119774486 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f733c062c70 0x7f733c136820 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f7338006fd0 tx=0x7f7338017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:33:18.338 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:18.321+0000 7f7331ffb700 1 -- 172.21.15.88:0/2119774486 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f7338011ab0 con 0x7f733c062c70 2023-09-30T01:33:18.339 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:18.321+0000 7f7331ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:33:18.339 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:18.321+0000 7f7331ffb700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T01:33:18.339 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:18.321+0000 7f7331ffb700 10 monclient: dump: 2023-09-30T01:33:18.339 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:33:18.339 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:33:18.339 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:18.339 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:18.339 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:33:18.339 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:33:18.339 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:33:18.339 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:33:18.339 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:33:18.339 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:33:18.340 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:33:18.340 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:33:18.340 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:33:18.340 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:33:18.340 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:33:18.340 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:33:18.340 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:18.321+0000 7f7331ffb700 10 monclient: _finish_auth 0 2023-09-30T01:33:18.340 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:18.321+0000 7f7331ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:33:18.340 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:18.321+0000 7f7331ffb700 1 -- 172.21.15.88:0/2119774486 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7338011c70 con 0x7f733c062c70 2023-09-30T01:33:18.340 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:18.321+0000 7f7344257700 5 monclient: authenticate success, global_id 4464 2023-09-30T01:33:18.340 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:18.321+0000 7f7344257700 10 monclient: _renew_subs 2023-09-30T01:33:18.340 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:18.321+0000 7f7344257700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.88:3300/0 2023-09-30T01:33:18.341 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:18.321+0000 7f7344257700 1 -- 172.21.15.88:0/2119774486 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f733c090680 con 0x7f733c062c70 2023-09-30T01:33:18.341 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:18.321+0000 7f7331ffb700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:33:18.341 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:18.321+0000 7f7331ffb700 1 -- 172.21.15.88:0/2119774486 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f7338011de0 con 0x7f733c062c70 2023-09-30T01:33:18.341 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:18.321+0000 7f7331ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:33:18.341 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:18.321+0000 7f7331ffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:33:18.341 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:18.321+0000 7f7333fff700 4 set_mon_vals no callback set 2023-09-30T01:33:18.341 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:18.321+0000 7f7331ffb700 10 monclient: dump: 2023-09-30T01:33:18.341 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:33:18.341 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:33:18.341 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:18.341 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:18.341 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:33:18.342 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:33:18.342 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:33:18.342 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:33:18.342 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:33:18.342 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:33:18.342 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:33:18.342 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:33:18.342 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:33:18.342 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:33:18.342 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:33:18.342 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:33:18.343 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:18.321+0000 7f7344257700 10 monclient: _renew_subs 2023-09-30T01:33:18.343 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:18.321+0000 7f7344257700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.88:3300/0 2023-09-30T01:33:18.343 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:18.321+0000 7f7344257700 1 -- 172.21.15.88:0/2119774486 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f733c090200 con 0x7f733c062c70 2023-09-30T01:33:18.343 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:18.321+0000 7f7331ffb700 1 -- 172.21.15.88:0/2119774486 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 40144+0+0 (secure 0 0 0) 0x7f7338011630 con 0x7f733c062c70 2023-09-30T01:33:18.343 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:18.321+0000 7f7344257700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:33:18.343 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:18.321+0000 7f7344257700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:33:18.343 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:18.321+0000 7f7344257700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.88:3300/0 2023-09-30T01:33:18.343 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:18.321+0000 7f7331ffb700 1 --2- 172.21.15.88:0/2119774486 >> v2:172.21.15.107:6812/29926 conn(0x7f7320033360 0x7f7320035810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:33:18.343 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:18.321+0000 7f7344257700 1 -- 172.21.15.88:0/2119774486 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f733c071960 con 0x7f733c062c70 2023-09-30T01:33:18.343 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:18.321+0000 7f7331ffb700 1 -- 172.21.15.88:0/2119774486 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 3502+0+0 (secure 0 0 0) 0x7f7338044ff0 con 0x7f733c062c70 2023-09-30T01:33:18.343 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:18.321+0000 7f73417f2700 1 --2- 172.21.15.88:0/2119774486 >> v2:172.21.15.107:6812/29926 conn(0x7f7320033360 0x7f7320035810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:33:18.343 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:18.321+0000 7f73417f2700 10 monclient: get_auth_request con 0x7f7320033360 auth_method 0 2023-09-30T01:33:18.344 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:18.321+0000 7f73417f2700 1 --2- 172.21.15.88:0/2119774486 >> v2:172.21.15.107:6812/29926 conn(0x7f7320033360 0x7f7320035810 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f732c011480 tx=0x7f732c0091a0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:33:18.344 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:18.329+0000 7f7331ffb700 1 -- 172.21.15.88:0/2119774486 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f733802d440 con 0x7f733c062c70 2023-09-30T01:33:18.344 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:18.329+0000 7f7331ffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:33:18.344 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:18.329+0000 7f7331ffb700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:33:18.529 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:18.525+0000 7f7344257700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T01:33:18.530 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:18.525+0000 7f7344257700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:33:18.530 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:18.525+0000 7f7344257700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.88:3300/0 2023-09-30T01:33:18.530 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:18.525+0000 7f7344257700 1 -- 172.21.15.88:0/2119774486 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f733c09f560 con 0x7f733c062c70 2023-09-30T01:33:18.530 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:18.525+0000 7f7331ffb700 1 -- 172.21.15.88:0/2119774486 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f7338015070 con 0x7f733c062c70 2023-09-30T01:33:18.531 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:18.525+0000 7f7331ffb700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:33:18.531 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:18.525+0000 7f7331ffb700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:33:18.531 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-09-30T01:33:18.531 INFO:teuthology.orchestra.run.smithi088.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:33:18.533 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:18.529+0000 7f7344257700 1 -- 172.21.15.88:0/2119774486 >> v2:172.21.15.107:6812/29926 conn(0x7f7320033360 msgr2=0x7f7320035810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:33:18.533 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:18.529+0000 7f7344257700 1 --2- 172.21.15.88:0/2119774486 >> v2:172.21.15.107:6812/29926 conn(0x7f7320033360 0x7f7320035810 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f732c011480 tx=0x7f732c0091a0 comp rx=0 tx=0).stop 2023-09-30T01:33:18.533 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:18.529+0000 7f7344257700 10 monclient: shutdown 2023-09-30T01:33:18.533 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:18.529+0000 7f7344257700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:33:18.533 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:18.529+0000 7f7344257700 1 -- 172.21.15.88:0/2119774486 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f733c062c70 msgr2=0x7f733c136820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:33:18.533 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:18.529+0000 7f7344257700 1 --2- 172.21.15.88:0/2119774486 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f733c062c70 0x7f733c136820 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f7338006fd0 tx=0x7f7338017000 comp rx=0 tx=0).stop 2023-09-30T01:33:18.533 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:18.529+0000 7f7344257700 1 -- 172.21.15.88:0/2119774486 shutdown_connections 2023-09-30T01:33:18.533 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:18.529+0000 7f7344257700 1 --2- 172.21.15.88:0/2119774486 >> v2:172.21.15.107:6812/29926 conn(0x7f7320033360 0x7f7320035810 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:18.533 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:18.529+0000 7f7344257700 1 --2- 172.21.15.88:0/2119774486 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7f733c099320 0x7f733c08e3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:18.534 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:18.529+0000 7f7344257700 1 --2- 172.21.15.88:0/2119774486 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7f733c098f50 0x7f733c136d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:18.534 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:18.529+0000 7f7344257700 1 --2- 172.21.15.88:0/2119774486 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f733c062c70 0x7f733c136820 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:18.534 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:18.529+0000 7f7344257700 1 -- 172.21.15.88:0/2119774486 >> 172.21.15.88:0/2119774486 conn(0x7f733c0a3440 msgr2=0x7f733c09dcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:33:18.534 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:18.529+0000 7f7344257700 1 -- 172.21.15.88:0/2119774486 shutdown_connections 2023-09-30T01:33:18.534 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:18.529+0000 7f7344257700 1 -- 172.21.15.88:0/2119774486 wait complete. 2023-09-30T01:33:21.549 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:33:21.660 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:21.657+0000 7f4c680f2700 10 monclient: get_monmap_and_config 2023-09-30T01:33:21.661 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:21.657+0000 7f4c680f2700 10 monclient: build_initial_monmap 2023-09-30T01:33:21.661 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:21.657+0000 7f4c680f2700 10 monclient: monmap: 2023-09-30T01:33:21.661 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2023-09-30T01:33:21.661 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:33:21.661 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:33:21.660767+0000 2023-09-30T01:33:21.661 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:33:21.660767+0000 2023-09-30T01:33:21.661 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2023-09-30T01:33:21.661 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:33:21.662 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a-legacy 2023-09-30T01:33:21.662 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.107:6789/0 mon.noname-f-legacy 2023-09-30T01:33:21.662 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.noname-b 2023-09-30T01:33:21.662 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.noname-g 2023-09-30T01:33:21.662 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.noname-c 2023-09-30T01:33:21.662 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.noname-h 2023-09-30T01:33:21.662 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.noname-d 2023-09-30T01:33:21.662 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.noname-i 2023-09-30T01:33:21.662 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.noname-e 2023-09-30T01:33:21.662 INFO:teuthology.orchestra.run.smithi088.stderr:9: v2:172.21.15.88:3300/0 mon.noname-a 2023-09-30T01:33:21.663 INFO:teuthology.orchestra.run.smithi088.stderr:10: v2:172.21.15.107:3300/0 mon.noname-f 2023-09-30T01:33:21.663 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:33:21.663 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:21.657+0000 7f4c680f2700 1 Processor -- start 2023-09-30T01:33:21.663 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:21.657+0000 7f4c680f2700 1 -- start start 2023-09-30T01:33:21.663 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:21.657+0000 7f4c680f2700 10 monclient: init 2023-09-30T01:33:21.671 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:21.657+0000 7f4c680f2700 10 monclient: _reopen_session rank -1 2023-09-30T01:33:21.671 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:21.657+0000 7f4c680f2700 10 monclient: _add_conns ranks=[5,4,10,1,6,0,3,2,9,7,8] 2023-09-30T01:33:21.671 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:21.657+0000 7f4c680f2700 1 --2- >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7f4c600a3270 0x7f4c600a0160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:33:21.671 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:21.657+0000 7f4c680f2700 10 monclient(hunting): picked mon.noname-h con 0x7f4c600a3270 addr [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] 2023-09-30T01:33:21.671 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:21.657+0000 7f4c680f2700 1 --2- >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f4c600a4b00 0x7f4c600a06a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:33:21.672 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:21.657+0000 7f4c680f2700 10 monclient(hunting): picked mon.noname-c con 0x7f4c600a4b00 addr [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] 2023-09-30T01:33:21.672 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:21.657+0000 7f4c680f2700 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f4c601333c0 0x7f4c601337b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:33:21.672 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:21.657+0000 7f4c680f2700 10 monclient(hunting): picked mon.noname-f con 0x7f4c601333c0 addr v2:172.21.15.107:3300/0 2023-09-30T01:33:21.672 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:21.657+0000 7f4c680f2700 10 monclient(hunting): start opening mon connection 2023-09-30T01:33:21.672 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:21.657+0000 7f4c680f2700 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4c600a4ed0 con 0x7f4c601333c0 2023-09-30T01:33:21.672 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:21.657+0000 7f4c680f2700 10 monclient(hunting): start opening mon connection 2023-09-30T01:33:21.672 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:21.657+0000 7f4c680f2700 1 -- --> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] -- mon_getmap magic: 0 v1 -- 0x7f4c600a6330 con 0x7f4c600a4b00 2023-09-30T01:33:21.672 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:21.657+0000 7f4c680f2700 10 monclient(hunting): start opening mon connection 2023-09-30T01:33:21.672 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:21.657+0000 7f4c680f2700 1 -- --> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] -- mon_getmap magic: 0 v1 -- 0x7f4c60098610 con 0x7f4c600a3270 2023-09-30T01:33:21.672 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:21.657+0000 7f4c680f2700 10 monclient(hunting): _renew_subs 2023-09-30T01:33:21.672 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:21.657+0000 7f4c680f2700 10 monclient(hunting): authenticate will time out at 873.698364s 2023-09-30T01:33:21.673 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:21.657+0000 7f4c6568d700 1 --2- >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f4c600a4b00 0x7f4c600a06a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:33:21.673 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:21.657+0000 7f4c6568d700 1 --2- >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f4c600a4b00 0x7f4c600a06a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3302/0 says I am v2:172.21.15.88:39562/0 (socket says 172.21.15.88:39562) 2023-09-30T01:33:21.673 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:21.657+0000 7f4c6568d700 1 -- 172.21.15.88:0/1101136046 learned_addr learned my addr 172.21.15.88:0/1101136046 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-09-30T01:33:21.673 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:21.657+0000 7f4c6568d700 10 monclient(hunting): get_auth_request con 0x7f4c600a4b00 auth_method 0 2023-09-30T01:33:21.673 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:21.657+0000 7f4c6568d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:33:21.673 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:21.657+0000 7f4c6568d700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:33:21.673 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:21.657+0000 7f4c6568d700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:33:21.673 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:21.657+0000 7f4c65e8e700 1 --2- 172.21.15.88:0/1101136046 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7f4c600a3270 0x7f4c600a0160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:33:21.673 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:21.657+0000 7f4c65e8e700 10 monclient(hunting): get_auth_request con 0x7f4c600a3270 auth_method 0 2023-09-30T01:33:21.673 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:21.657+0000 7f4c65e8e700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:33:21.673 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:21.657+0000 7f4c65e8e700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:33:21.674 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:21.657+0000 7f4c65e8e700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:33:21.674 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:21.657+0000 7f4c6668f700 1 --2- 172.21.15.88:0/1101136046 >> v2:172.21.15.107:3300/0 conn(0x7f4c601333c0 0x7f4c601337b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:33:21.674 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:21.657+0000 7f4c6668f700 10 monclient(hunting): get_auth_request con 0x7f4c601333c0 auth_method 0 2023-09-30T01:33:21.674 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:21.657+0000 7f4c6668f700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:33:21.674 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:21.657+0000 7f4c6668f700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:33:21.674 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:21.657+0000 7f4c6668f700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:33:21.674 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:21.657+0000 7f4c6568d700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:33:21.674 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:21.657+0000 7f4c6568d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:33:21.674 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:21.657+0000 7f4c6568d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:33:21.674 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:21.657+0000 7f4c65e8e700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:33:21.674 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:21.657+0000 7f4c65e8e700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:33:21.674 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:21.657+0000 7f4c65e8e700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:33:21.675 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:21.657+0000 7f4c6668f700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:33:21.675 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:21.657+0000 7f4c6668f700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:33:21.675 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:21.657+0000 7f4c6668f700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:33:21.675 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:21.657+0000 7f4c6568d700 10 monclient(hunting): handle_auth_done global_id 4396 payload 274 2023-09-30T01:33:21.675 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:21.657+0000 7f4c6568d700 1 -- 172.21.15.88:0/1101136046 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7f4c600a3270 msgr2=0x7f4c600a0160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:33:21.675 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:21.657+0000 7f4c6568d700 1 --2- 172.21.15.88:0/1101136046 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7f4c600a3270 0x7f4c600a0160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:21.675 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:21.657+0000 7f4c6568d700 1 -- 172.21.15.88:0/1101136046 >> v2:172.21.15.107:3300/0 conn(0x7f4c601333c0 msgr2=0x7f4c601337b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:33:21.675 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:21.657+0000 7f4c6568d700 1 --2- 172.21.15.88:0/1101136046 >> v2:172.21.15.107:3300/0 conn(0x7f4c601333c0 0x7f4c601337b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:21.675 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:21.657+0000 7f4c6568d700 10 monclient: _finish_hunting 0 2023-09-30T01:33:21.675 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:21.657+0000 7f4c6568d700 1 monclient: found mon.noname-c 2023-09-30T01:33:21.675 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:21.657+0000 7f4c6568d700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:33:21.675 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:21.657+0000 7f4c6568d700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.88:3302/0 2023-09-30T01:33:21.676 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:21.657+0000 7f4c6568d700 1 -- 172.21.15.88:0/1101136046 --> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c6005ecb0 con 0x7f4c600a4b00 2023-09-30T01:33:21.676 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:21.657+0000 7f4c65e8e700 1 --2- 172.21.15.88:0/1101136046 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7f4c600a3270 0x7f4c600a0160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:33:21.676 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:21.657+0000 7f4c6568d700 1 --2- 172.21.15.88:0/1101136046 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f4c600a4b00 0x7f4c600a06a0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f4c5c009f20 tx=0x7f4c5c009840 comp rx=0 tx=0).ready entity=mon.4 client_cookie=8dfd96034f1384b4 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:33:21.676 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:21.661+0000 7f4c64e8c700 1 -- 172.21.15.88:0/1101136046 <== mon.4 v2:172.21.15.88:3302/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f4c5c00e070 con 0x7f4c600a4b00 2023-09-30T01:33:21.676 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:21.661+0000 7f4c64e8c700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:33:21.676 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:21.661+0000 7f4c64e8c700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T01:33:21.676 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:21.661+0000 7f4c64e8c700 10 monclient: dump: 2023-09-30T01:33:21.676 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:33:21.676 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:33:21.676 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:21.677 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:21.677 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:33:21.677 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:33:21.677 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:33:21.677 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:33:21.677 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:33:21.677 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:33:21.677 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:33:21.677 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:33:21.677 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:33:21.677 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:33:21.677 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:33:21.678 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:33:21.678 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:21.661+0000 7f4c64e8c700 10 monclient: _finish_auth 0 2023-09-30T01:33:21.678 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:21.661+0000 7f4c64e8c700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:33:21.678 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:21.661+0000 7f4c680f2700 5 monclient: authenticate success, global_id 4396 2023-09-30T01:33:21.678 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:21.661+0000 7f4c64e8c700 1 -- 172.21.15.88:0/1101136046 <== mon.4 v2:172.21.15.88:3302/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c5c00bdf0 con 0x7f4c600a4b00 2023-09-30T01:33:21.678 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:21.661+0000 7f4c680f2700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:33:21.678 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:21.661+0000 7f4c64e8c700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:33:21.678 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:21.661+0000 7f4c680f2700 10 monclient: get_monmap_and_config success 2023-09-30T01:33:21.678 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:21.661+0000 7f4c64e8c700 1 -- 172.21.15.88:0/1101136046 <== mon.4 v2:172.21.15.88:3302/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f4c5c0092e0 con 0x7f4c600a4b00 2023-09-30T01:33:21.678 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:21.661+0000 7f4c64e8c700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:33:21.679 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:21.661+0000 7f4c680f2700 4 set_mon_vals no callback set 2023-09-30T01:33:21.679 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:21.661+0000 7f4c64e8c700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T01:33:21.679 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:21.661+0000 7f4c64e8c700 10 monclient: dump: 2023-09-30T01:33:21.679 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:33:21.679 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:33:21.679 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:21.679 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:21.679 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:33:21.679 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:33:21.679 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:33:21.679 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:33:21.679 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:33:21.680 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:33:21.680 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:33:21.680 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:33:21.680 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:33:21.680 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:33:21.680 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:33:21.680 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:33:21.680 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:21.661+0000 7f4c680f2700 10 monclient: shutdown 2023-09-30T01:33:21.680 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:21.661+0000 7f4c680f2700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:33:21.680 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:21.661+0000 7f4c680f2700 1 -- 172.21.15.88:0/1101136046 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f4c600a4b00 msgr2=0x7f4c600a06a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:33:21.681 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:21.661+0000 7f4c680f2700 1 --2- 172.21.15.88:0/1101136046 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f4c600a4b00 0x7f4c600a06a0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f4c5c009f20 tx=0x7f4c5c009840 comp rx=0 tx=0).stop 2023-09-30T01:33:21.681 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:21.661+0000 7f4c680f2700 1 -- 172.21.15.88:0/1101136046 shutdown_connections 2023-09-30T01:33:21.681 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:21.661+0000 7f4c680f2700 1 --2- 172.21.15.88:0/1101136046 >> v2:172.21.15.107:3300/0 conn(0x7f4c601333c0 0x7f4c601337b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:21.681 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:21.661+0000 7f4c680f2700 1 --2- 172.21.15.88:0/1101136046 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f4c600a4b00 0x7f4c600a06a0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:21.681 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:21.661+0000 7f4c680f2700 1 --2- 172.21.15.88:0/1101136046 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7f4c600a3270 0x7f4c600a0160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:21.681 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:21.661+0000 7f4c680f2700 1 -- 172.21.15.88:0/1101136046 >> 172.21.15.88:0/1101136046 conn(0x7f4c600a67f0 msgr2=0x7f4c600a6bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:33:21.681 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:21.661+0000 7f4c680f2700 1 -- 172.21.15.88:0/1101136046 shutdown_connections 2023-09-30T01:33:21.681 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:21.661+0000 7f4c680f2700 1 -- 172.21.15.88:0/1101136046 wait complete. 2023-09-30T01:33:21.681 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:21.661+0000 7f4c680f2700 10 monclient: build_initial_monmap 2023-09-30T01:33:21.681 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:21.661+0000 7f4c680f2700 10 monclient: monmap: 2023-09-30T01:33:21.682 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2023-09-30T01:33:21.682 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:33:21.682 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 0.000000 2023-09-30T01:33:21.682 INFO:teuthology.orchestra.run.smithi088.stderr:created 0.000000 2023-09-30T01:33:21.682 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2023-09-30T01:33:21.682 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 1 2023-09-30T01:33:21.682 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.noname-a 2023-09-30T01:33:21.682 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.noname-f 2023-09-30T01:33:21.682 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.noname-b 2023-09-30T01:33:21.682 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.noname-g 2023-09-30T01:33:21.683 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.noname-c 2023-09-30T01:33:21.683 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.noname-h 2023-09-30T01:33:21.683 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.noname-d 2023-09-30T01:33:21.683 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.noname-i 2023-09-30T01:33:21.683 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.noname-e 2023-09-30T01:33:21.683 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:33:21.683 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:21.665+0000 7f4c680f2700 1 Processor -- start 2023-09-30T01:33:21.683 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:21.665+0000 7f4c680f2700 1 -- start start 2023-09-30T01:33:21.683 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:21.665+0000 7f4c680f2700 10 monclient: init 2023-09-30T01:33:21.683 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:21.665+0000 7f4c680f2700 10 monclient: _reopen_session rank -1 2023-09-30T01:33:21.684 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:21.665+0000 7f4c680f2700 10 monclient: _add_conns ranks=[8,6,7,1,5,3,4,0,2] 2023-09-30T01:33:21.684 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:21.665+0000 7f4c680f2700 1 --2- >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f4c600a3270 0x7f4c6011da00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:33:21.684 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:21.665+0000 7f4c680f2700 10 monclient(hunting): picked mon.noname-e con 0x7f4c600a3270 addr [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] 2023-09-30T01:33:21.684 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:21.665+0000 7f4c680f2700 1 --2- >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7f4c600a4b00 0x7f4c6011df40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:33:21.684 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:21.665+0000 7f4c680f2700 10 monclient(hunting): picked mon.noname-d con 0x7f4c600a4b00 addr [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] 2023-09-30T01:33:21.684 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:21.665+0000 7f4c680f2700 1 --2- >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7f4c601333c0 0x7f4c6011e480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:33:21.684 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:21.665+0000 7f4c680f2700 10 monclient(hunting): picked mon.noname-i con 0x7f4c601333c0 addr [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] 2023-09-30T01:33:21.684 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:21.665+0000 7f4c680f2700 10 monclient(hunting): start opening mon connection 2023-09-30T01:33:21.684 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:21.665+0000 7f4c680f2700 1 -- --> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] -- mon_getmap magic: 0 v1 -- 0x7f4c60097c70 con 0x7f4c600a4b00 2023-09-30T01:33:21.684 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:21.665+0000 7f4c680f2700 10 monclient(hunting): start opening mon connection 2023-09-30T01:33:21.685 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:21.665+0000 7f4c680f2700 1 -- --> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] -- mon_getmap magic: 0 v1 -- 0x7f4c600a0e80 con 0x7f4c601333c0 2023-09-30T01:33:21.685 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:21.665+0000 7f4c680f2700 10 monclient(hunting): start opening mon connection 2023-09-30T01:33:21.685 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:21.665+0000 7f4c680f2700 1 -- --> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] -- mon_getmap magic: 0 v1 -- 0x7f4c60131050 con 0x7f4c600a3270 2023-09-30T01:33:21.685 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:21.665+0000 7f4c680f2700 10 monclient(hunting): _renew_subs 2023-09-30T01:33:21.685 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:21.665+0000 7f4c680f2700 10 monclient(hunting): authenticate will time out at 873.705529s 2023-09-30T01:33:21.685 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:21.665+0000 7f4c65e8e700 1 --2- >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f4c600a3270 0x7f4c6011da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:33:21.685 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:21.665+0000 7f4c65e8e700 1 --2- >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f4c600a3270 0x7f4c6011da00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3304/0 says I am v2:172.21.15.88:55862/0 (socket says 172.21.15.88:55862) 2023-09-30T01:33:21.685 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:21.665+0000 7f4c65e8e700 1 -- 172.21.15.88:0/134839655 learned_addr learned my addr 172.21.15.88:0/134839655 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-09-30T01:33:21.685 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:21.665+0000 7f4c65e8e700 10 monclient(hunting): get_auth_request con 0x7f4c600a3270 auth_method 0 2023-09-30T01:33:21.685 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:21.665+0000 7f4c65e8e700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:33:21.686 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:21.665+0000 7f4c65e8e700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:33:21.686 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:21.665+0000 7f4c65e8e700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:33:21.686 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:21.665+0000 7f4c6568d700 1 --2- 172.21.15.88:0/134839655 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7f4c600a4b00 0x7f4c6011df40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:33:21.686 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:21.665+0000 7f4c65e8e700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:33:21.686 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:21.665+0000 7f4c65e8e700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:33:21.686 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:21.665+0000 7f4c65e8e700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:33:21.686 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:21.665+0000 7f4c6568d700 10 monclient(hunting): get_auth_request con 0x7f4c600a4b00 auth_method 0 2023-09-30T01:33:21.686 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:21.665+0000 7f4c6568d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:33:21.686 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:21.665+0000 7f4c6568d700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:33:21.686 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:21.665+0000 7f4c6568d700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:33:21.687 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:21.665+0000 7f4c6668f700 1 --2- 172.21.15.88:0/134839655 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7f4c601333c0 0x7f4c6011e480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:33:21.687 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:21.665+0000 7f4c6668f700 10 monclient(hunting): get_auth_request con 0x7f4c601333c0 auth_method 0 2023-09-30T01:33:21.687 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:21.665+0000 7f4c6668f700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:33:21.687 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:21.665+0000 7f4c6668f700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:33:21.687 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:21.665+0000 7f4c6668f700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:33:21.687 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:21.665+0000 7f4c6568d700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:33:21.687 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:21.665+0000 7f4c6568d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:33:21.687 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:21.665+0000 7f4c6568d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:33:21.687 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:21.665+0000 7f4c65e8e700 10 monclient(hunting): handle_auth_done global_id 4436 payload 824 2023-09-30T01:33:21.687 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:21.665+0000 7f4c65e8e700 1 -- 172.21.15.88:0/134839655 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7f4c601333c0 msgr2=0x7f4c6011e480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:33:21.688 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:21.665+0000 7f4c65e8e700 1 --2- 172.21.15.88:0/134839655 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7f4c601333c0 0x7f4c6011e480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:21.688 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:21.665+0000 7f4c65e8e700 1 -- 172.21.15.88:0/134839655 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7f4c600a4b00 msgr2=0x7f4c6011df40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:33:21.688 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:21.665+0000 7f4c65e8e700 1 --2- 172.21.15.88:0/134839655 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7f4c600a4b00 0x7f4c6011df40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:21.688 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:21.665+0000 7f4c65e8e700 10 monclient: _finish_hunting 0 2023-09-30T01:33:21.688 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:21.665+0000 7f4c65e8e700 1 monclient: found mon.noname-e 2023-09-30T01:33:21.688 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:21.665+0000 7f4c65e8e700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:33:21.688 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:21.665+0000 7f4c65e8e700 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.88:3304/0 2023-09-30T01:33:21.688 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:21.665+0000 7f4c65e8e700 1 -- 172.21.15.88:0/134839655 --> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c50009420 con 0x7f4c600a3270 2023-09-30T01:33:21.688 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:21.665+0000 7f4c65e8e700 1 --2- 172.21.15.88:0/134839655 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f4c600a3270 0x7f4c6011da00 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f4c5000ec80 tx=0x7f4c500091a0 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:33:21.689 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:21.665+0000 7f4c55ffb700 1 -- 172.21.15.88:0/134839655 <== mon.8 v2:172.21.15.88:3304/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f4c5000cd70 con 0x7f4c600a3270 2023-09-30T01:33:21.689 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:21.665+0000 7f4c55ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:33:21.689 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:21.665+0000 7f4c55ffb700 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2023-09-30T01:33:21.689 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:21.665+0000 7f4c55ffb700 10 monclient: dump: 2023-09-30T01:33:21.689 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:33:21.689 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:33:21.689 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:21.689 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:21.689 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:33:21.689 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:33:21.690 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:33:21.690 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:33:21.690 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:33:21.690 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:33:21.690 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:33:21.690 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:33:21.690 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:33:21.690 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:33:21.690 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:33:21.690 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:33:21.691 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:21.665+0000 7f4c55ffb700 10 monclient: _finish_auth 0 2023-09-30T01:33:21.691 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:21.665+0000 7f4c55ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:33:21.691 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:21.665+0000 7f4c680f2700 5 monclient: authenticate success, global_id 4436 2023-09-30T01:33:21.691 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:21.665+0000 7f4c55ffb700 1 -- 172.21.15.88:0/134839655 <== mon.8 v2:172.21.15.88:3304/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c5000ced0 con 0x7f4c600a3270 2023-09-30T01:33:21.691 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:21.665+0000 7f4c55ffb700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:33:21.691 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:21.665+0000 7f4c680f2700 10 monclient: _renew_subs 2023-09-30T01:33:21.691 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:21.665+0000 7f4c55ffb700 1 -- 172.21.15.88:0/134839655 <== mon.8 v2:172.21.15.88:3304/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f4c500138f0 con 0x7f4c600a3270 2023-09-30T01:33:21.691 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:21.665+0000 7f4c680f2700 10 monclient: _send_mon_message to mon.e at v2:172.21.15.88:3304/0 2023-09-30T01:33:21.691 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:21.665+0000 7f4c57fff700 4 set_mon_vals no callback set 2023-09-30T01:33:21.691 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:21.665+0000 7f4c680f2700 1 -- 172.21.15.88:0/134839655 --> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c5c009ae0 con 0x7f4c600a3270 2023-09-30T01:33:21.692 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:21.665+0000 7f4c55ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:33:21.692 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:21.665+0000 7f4c55ffb700 10 monclient: got monmap 1 from mon.e (according to old e1) 2023-09-30T01:33:21.692 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:21.665+0000 7f4c55ffb700 10 monclient: dump: 2023-09-30T01:33:21.692 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:33:21.692 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:33:21.692 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:21.692 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:21.692 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:33:21.692 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:33:21.692 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:33:21.693 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:33:21.693 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:33:21.693 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:33:21.693 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:33:21.693 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:33:21.693 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:33:21.693 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:33:21.693 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:33:21.693 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:33:21.693 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:21.665+0000 7f4c680f2700 10 monclient: _renew_subs 2023-09-30T01:33:21.694 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:21.665+0000 7f4c680f2700 10 monclient: _send_mon_message to mon.e at v2:172.21.15.88:3304/0 2023-09-30T01:33:21.694 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:21.665+0000 7f4c680f2700 1 -- 172.21.15.88:0/134839655 --> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c60064b80 con 0x7f4c600a3270 2023-09-30T01:33:21.695 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:21.669+0000 7f4c680f2700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:33:21.695 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:21.669+0000 7f4c680f2700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:33:21.695 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:21.669+0000 7f4c680f2700 10 monclient: _send_mon_message to mon.e at v2:172.21.15.88:3304/0 2023-09-30T01:33:21.695 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:21.669+0000 7f4c680f2700 1 -- 172.21.15.88:0/134839655 --> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4c60071960 con 0x7f4c600a3270 2023-09-30T01:33:21.695 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:21.669+0000 7f4c55ffb700 1 -- 172.21.15.88:0/134839655 <== mon.8 v2:172.21.15.88:3304/0 4 ==== mgrmap(e 5) v1 ==== 40144+0+0 (secure 0 0 0) 0x7f4c50013470 con 0x7f4c600a3270 2023-09-30T01:33:21.696 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:21.669+0000 7f4c55ffb700 1 --2- 172.21.15.88:0/134839655 >> v2:172.21.15.107:6812/29926 conn(0x7f4c44033390 0x7f4c44035840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:33:21.696 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:21.669+0000 7f4c55ffb700 1 -- 172.21.15.88:0/134839655 <== mon.8 v2:172.21.15.88:3304/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 3502+0+0 (secure 0 0 0) 0x7f4c50043b00 con 0x7f4c600a3270 2023-09-30T01:33:21.696 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:21.669+0000 7f4c6568d700 1 --2- 172.21.15.88:0/134839655 >> v2:172.21.15.107:6812/29926 conn(0x7f4c44033390 0x7f4c44035840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:33:21.696 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:21.669+0000 7f4c6568d700 10 monclient: get_auth_request con 0x7f4c44033390 auth_method 0 2023-09-30T01:33:21.696 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:21.669+0000 7f4c6568d700 1 --2- 172.21.15.88:0/134839655 >> v2:172.21.15.107:6812/29926 conn(0x7f4c44033390 0x7f4c44035840 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f4c5c009f20 tx=0x7f4c5c018000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:33:21.696 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:21.673+0000 7f4c55ffb700 1 -- 172.21.15.88:0/134839655 <== mon.8 v2:172.21.15.88:3304/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f4c5001b3b0 con 0x7f4c600a3270 2023-09-30T01:33:21.696 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:21.673+0000 7f4c55ffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:33:21.696 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:21.673+0000 7f4c55ffb700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:33:21.878 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:21.873+0000 7f4c680f2700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T01:33:21.878 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:21.873+0000 7f4c680f2700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:33:21.878 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:21.873+0000 7f4c680f2700 10 monclient: _send_mon_message to mon.e at v2:172.21.15.88:3304/0 2023-09-30T01:33:21.878 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:21.873+0000 7f4c680f2700 1 -- 172.21.15.88:0/134839655 --> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f4c60062c70 con 0x7f4c600a3270 2023-09-30T01:33:21.879 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:21.873+0000 7f4c55ffb700 1 -- 172.21.15.88:0/134839655 <== mon.8 v2:172.21.15.88:3304/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f4c5001ce40 con 0x7f4c600a3270 2023-09-30T01:33:21.879 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:21.873+0000 7f4c55ffb700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:33:21.879 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:21.873+0000 7f4c55ffb700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:33:21.879 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-09-30T01:33:21.880 INFO:teuthology.orchestra.run.smithi088.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:33:21.882 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:21.877+0000 7f4c680f2700 1 -- 172.21.15.88:0/134839655 >> v2:172.21.15.107:6812/29926 conn(0x7f4c44033390 msgr2=0x7f4c44035840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:33:21.882 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:21.877+0000 7f4c680f2700 1 --2- 172.21.15.88:0/134839655 >> v2:172.21.15.107:6812/29926 conn(0x7f4c44033390 0x7f4c44035840 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f4c5c009f20 tx=0x7f4c5c018000 comp rx=0 tx=0).stop 2023-09-30T01:33:21.882 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:21.877+0000 7f4c680f2700 10 monclient: shutdown 2023-09-30T01:33:21.882 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:21.877+0000 7f4c680f2700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:33:21.882 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:21.877+0000 7f4c680f2700 1 -- 172.21.15.88:0/134839655 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f4c600a3270 msgr2=0x7f4c6011da00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:33:21.882 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:21.877+0000 7f4c680f2700 1 --2- 172.21.15.88:0/134839655 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f4c600a3270 0x7f4c6011da00 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f4c5000ec80 tx=0x7f4c500091a0 comp rx=0 tx=0).stop 2023-09-30T01:33:21.882 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:21.877+0000 7f4c680f2700 1 -- 172.21.15.88:0/134839655 shutdown_connections 2023-09-30T01:33:21.882 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:21.877+0000 7f4c680f2700 1 --2- 172.21.15.88:0/134839655 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7f4c601333c0 0x7f4c6011e480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:21.882 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:21.877+0000 7f4c680f2700 1 --2- 172.21.15.88:0/134839655 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7f4c600a4b00 0x7f4c6011df40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:21.882 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:21.877+0000 7f4c680f2700 1 --2- 172.21.15.88:0/134839655 >> v2:172.21.15.107:6812/29926 conn(0x7f4c44033390 0x7f4c44035840 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:21.883 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:21.877+0000 7f4c680f2700 1 --2- 172.21.15.88:0/134839655 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f4c600a3270 0x7f4c6011da00 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:21.883 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:21.877+0000 7f4c680f2700 1 -- 172.21.15.88:0/134839655 >> 172.21.15.88:0/134839655 conn(0x7f4c600a67f0 msgr2=0x7f4c600a20e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:33:21.883 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:21.877+0000 7f4c680f2700 1 -- 172.21.15.88:0/134839655 shutdown_connections 2023-09-30T01:33:21.883 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:21.877+0000 7f4c680f2700 1 -- 172.21.15.88:0/134839655 wait complete. 2023-09-30T01:33:24.894 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:33:25.006 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:25.001+0000 7f8515cee700 10 monclient: get_monmap_and_config 2023-09-30T01:33:25.006 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:25.001+0000 7f8515cee700 10 monclient: build_initial_monmap 2023-09-30T01:33:25.006 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:25.001+0000 7f8515cee700 10 monclient: monmap: 2023-09-30T01:33:25.006 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2023-09-30T01:33:25.007 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:33:25.007 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:33:25.006912+0000 2023-09-30T01:33:25.007 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:33:25.006912+0000 2023-09-30T01:33:25.007 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2023-09-30T01:33:25.007 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:33:25.007 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a-legacy 2023-09-30T01:33:25.007 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.107:6789/0 mon.noname-f-legacy 2023-09-30T01:33:25.007 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.noname-b 2023-09-30T01:33:25.007 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.noname-g 2023-09-30T01:33:25.007 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.noname-c 2023-09-30T01:33:25.007 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.noname-h 2023-09-30T01:33:25.007 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.noname-d 2023-09-30T01:33:25.008 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.noname-i 2023-09-30T01:33:25.008 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.noname-e 2023-09-30T01:33:25.008 INFO:teuthology.orchestra.run.smithi088.stderr:9: v2:172.21.15.88:3300/0 mon.noname-a 2023-09-30T01:33:25.008 INFO:teuthology.orchestra.run.smithi088.stderr:10: v2:172.21.15.107:3300/0 mon.noname-f 2023-09-30T01:33:25.008 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:33:25.013 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:25.001+0000 7f8515cee700 1 Processor -- start 2023-09-30T01:33:25.013 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:25.001+0000 7f8515cee700 1 -- start start 2023-09-30T01:33:25.013 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:25.001+0000 7f8515cee700 10 monclient: init 2023-09-30T01:33:25.014 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:25.005+0000 7f8515cee700 10 monclient: _reopen_session rank -1 2023-09-30T01:33:25.014 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:25.005+0000 7f8515cee700 10 monclient: _add_conns ranks=[4,1,9,10,5,2,6,3,7,8,0] 2023-09-30T01:33:25.014 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:25.005+0000 7f8515cee700 1 --2- >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f85100a3380 0x7f85100a48b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:33:25.014 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:25.005+0000 7f8515cee700 10 monclient(hunting): picked mon.noname-c con 0x7f85100a3380 addr [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] 2023-09-30T01:33:25.014 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:25.005+0000 7f8515cee700 10 monclient(hunting): picked mon.noname-f-legacy con 0x7f851009ffd0 addr v1:172.21.15.107:6789/0 2023-09-30T01:33:25.014 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:25.005+0000 7f8515cee700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f85100a03a0 0x7f8510133ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:33:25.014 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:25.005+0000 7f8515cee700 10 monclient(hunting): picked mon.noname-a con 0x7f85100a03a0 addr v2:172.21.15.88:3300/0 2023-09-30T01:33:25.014 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:25.005+0000 7f8515cee700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f851004bb20 con 0x7f851009ffd0 2023-09-30T01:33:25.014 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:25.005+0000 7f8515cee700 10 monclient(hunting): start opening mon connection 2023-09-30T01:33:25.014 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:25.005+0000 7f8515cee700 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8510097ab0 con 0x7f85100a03a0 2023-09-30T01:33:25.014 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:25.005+0000 7f8515cee700 10 monclient(hunting): start opening mon connection 2023-09-30T01:33:25.015 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:25.005+0000 7f8515cee700 1 -- --> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] -- mon_getmap magic: 0 v1 -- 0x7f8510098480 con 0x7f85100a3380 2023-09-30T01:33:25.015 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:25.005+0000 7f8515cee700 10 monclient(hunting): _renew_subs 2023-09-30T01:33:25.015 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:25.005+0000 7f8515cee700 10 monclient(hunting): authenticate will time out at 877.044348s 2023-09-30T01:33:25.015 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:25.005+0000 7f85154ed700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f85100a03a0 0x7f8510133ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:33:25.015 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:25.005+0000 7f8514cec700 1 --2- >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f85100a3380 0x7f85100a48b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:33:25.015 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:25.005+0000 7f8514cec700 1 --2- >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f85100a3380 0x7f85100a48b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3302/0 says I am v2:172.21.15.88:39564/0 (socket says 172.21.15.88:39564) 2023-09-30T01:33:25.015 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:25.005+0000 7f85154ed700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f85100a03a0 0x7f8510133ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:36472/0 (socket says 172.21.15.88:36472) 2023-09-30T01:33:25.015 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:25.005+0000 7f8514cec700 1 -- 172.21.15.88:0/3400478814 learned_addr learned my addr 172.21.15.88:0/3400478814 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-09-30T01:33:25.015 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:25.005+0000 7f8514cec700 10 monclient(hunting): get_auth_request con 0x7f85100a3380 auth_method 0 2023-09-30T01:33:25.015 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:25.005+0000 7f8514cec700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:33:25.015 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:25.005+0000 7f8514cec700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:33:25.015 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:25.005+0000 7f8514cec700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:33:25.016 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:25.005+0000 7f85154ed700 10 monclient(hunting): get_auth_request con 0x7f85100a03a0 auth_method 0 2023-09-30T01:33:25.016 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:25.005+0000 7f85154ed700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:33:25.016 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:25.005+0000 7f85154ed700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:33:25.016 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:25.005+0000 7f85154ed700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:33:25.016 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:25.005+0000 7f85154ed700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:33:25.016 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:25.005+0000 7f85154ed700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:33:25.016 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:25.005+0000 7f85154ed700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:33:25.016 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:25.005+0000 7f8514cec700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:33:25.016 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:25.005+0000 7f8514cec700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:33:25.016 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:25.005+0000 7f8514cec700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:33:25.016 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:25.005+0000 7f85154ed700 10 monclient(hunting): handle_auth_done global_id 4473 payload 274 2023-09-30T01:33:25.016 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:25.005+0000 7f85154ed700 1 -- 172.21.15.88:0/3400478814 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f85100a3380 msgr2=0x7f85100a48b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:33:25.017 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:25.005+0000 7f85154ed700 1 --2- 172.21.15.88:0/3400478814 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f85100a3380 0x7f85100a48b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:25.017 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:25.005+0000 7f85154ed700 1 -- 172.21.15.88:0/3400478814 >> v1:172.21.15.107:6789/0 conn(0x7f851009ffd0 legacy=0x7f85101333b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:33:25.017 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:25.005+0000 7f85154ed700 10 monclient: _finish_hunting 0 2023-09-30T01:33:25.017 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:25.005+0000 7f85154ed700 1 monclient: found mon.noname-a 2023-09-30T01:33:25.017 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:25.005+0000 7f85154ed700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:33:25.017 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:25.005+0000 7f85154ed700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.88:3300/0 2023-09-30T01:33:25.017 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:25.005+0000 7f85154ed700 1 -- 172.21.15.88:0/3400478814 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85100a3750 con 0x7f85100a03a0 2023-09-30T01:33:25.017 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:25.005+0000 7f85154ed700 1 --2- 172.21.15.88:0/3400478814 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f85100a03a0 0x7f8510133ac0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f8508009e60 tx=0x7f8508009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3f5d5c8d1759edb3 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:33:25.017 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:25.005+0000 7f85077fe700 1 -- 172.21.15.88:0/3400478814 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f850800e070 con 0x7f85100a03a0 2023-09-30T01:33:25.017 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:25.005+0000 7f85077fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:33:25.017 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:25.005+0000 7f85077fe700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T01:33:25.018 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:25.005+0000 7f85077fe700 10 monclient: dump: 2023-09-30T01:33:25.018 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:33:25.018 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:33:25.018 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:25.018 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:25.018 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:33:25.018 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:33:25.018 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:33:25.018 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:33:25.018 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:33:25.018 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:33:25.018 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:33:25.019 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:33:25.019 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:33:25.019 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:33:25.019 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:33:25.019 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:33:25.019 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:25.005+0000 7f85077fe700 10 monclient: _finish_auth 0 2023-09-30T01:33:25.019 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:25.005+0000 7f85077fe700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:33:25.019 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:25.005+0000 7f8515cee700 5 monclient: authenticate success, global_id 4473 2023-09-30T01:33:25.019 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:25.005+0000 7f8515cee700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:33:25.019 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:25.005+0000 7f85077fe700 1 -- 172.21.15.88:0/3400478814 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8508012430 con 0x7f85100a03a0 2023-09-30T01:33:25.019 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:25.005+0000 7f85077fe700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:33:25.019 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:25.005+0000 7f8515cee700 10 monclient: get_monmap_and_config success 2023-09-30T01:33:25.020 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:25.005+0000 7f8515cee700 4 set_mon_vals no callback set 2023-09-30T01:33:25.020 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:25.005+0000 7f85077fe700 1 -- 172.21.15.88:0/3400478814 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f8508012a20 con 0x7f85100a03a0 2023-09-30T01:33:25.020 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:25.005+0000 7f85077fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:33:25.020 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:25.005+0000 7f85077fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:33:25.020 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:25.005+0000 7f85077fe700 10 monclient: dump: 2023-09-30T01:33:25.020 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:33:25.020 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:33:25.020 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:25.020 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:25.020 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:33:25.020 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:33:25.020 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:33:25.021 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:33:25.021 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:33:25.021 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:33:25.021 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:33:25.021 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:33:25.021 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:33:25.021 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:33:25.021 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:33:25.021 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:33:25.021 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:25.005+0000 7f8515cee700 10 monclient: shutdown 2023-09-30T01:33:25.021 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:25.005+0000 7f8515cee700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:33:25.021 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:25.005+0000 7f8515cee700 1 -- 172.21.15.88:0/3400478814 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f85100a03a0 msgr2=0x7f8510133ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:33:25.022 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:25.005+0000 7f8515cee700 1 --2- 172.21.15.88:0/3400478814 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f85100a03a0 0x7f8510133ac0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f8508009e60 tx=0x7f8508009780 comp rx=0 tx=0).stop 2023-09-30T01:33:25.022 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:25.005+0000 7f8515cee700 1 -- 172.21.15.88:0/3400478814 shutdown_connections 2023-09-30T01:33:25.022 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:25.005+0000 7f8515cee700 1 --2- 172.21.15.88:0/3400478814 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f85100a03a0 0x7f8510133ac0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:25.022 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:25.005+0000 7f8515cee700 1 --2- 172.21.15.88:0/3400478814 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f85100a3380 0x7f85100a48b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:25.022 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:25.005+0000 7f8515cee700 1 -- 172.21.15.88:0/3400478814 >> 172.21.15.88:0/3400478814 conn(0x7f851006e5e0 msgr2=0x7f851006e9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:33:25.022 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:25.005+0000 7f8515cee700 1 -- 172.21.15.88:0/3400478814 shutdown_connections 2023-09-30T01:33:25.022 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:25.005+0000 7f8515cee700 1 -- 172.21.15.88:0/3400478814 wait complete. 2023-09-30T01:33:25.022 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:25.005+0000 7f8515cee700 10 monclient: build_initial_monmap 2023-09-30T01:33:25.022 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:25.005+0000 7f8515cee700 10 monclient: monmap: 2023-09-30T01:33:25.022 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2023-09-30T01:33:25.022 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:33:25.022 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 0.000000 2023-09-30T01:33:25.023 INFO:teuthology.orchestra.run.smithi088.stderr:created 0.000000 2023-09-30T01:33:25.023 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2023-09-30T01:33:25.023 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 1 2023-09-30T01:33:25.023 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.noname-a 2023-09-30T01:33:25.023 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.noname-f 2023-09-30T01:33:25.023 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.noname-b 2023-09-30T01:33:25.023 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.noname-g 2023-09-30T01:33:25.023 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.noname-c 2023-09-30T01:33:25.023 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.noname-h 2023-09-30T01:33:25.023 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.noname-d 2023-09-30T01:33:25.023 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.noname-i 2023-09-30T01:33:25.023 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.noname-e 2023-09-30T01:33:25.024 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:33:25.024 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:25.005+0000 7f8515cee700 1 Processor -- start 2023-09-30T01:33:25.024 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:25.005+0000 7f8515cee700 1 -- start start 2023-09-30T01:33:25.024 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:25.005+0000 7f8515cee700 10 monclient: init 2023-09-30T01:33:25.024 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:25.009+0000 7f8515cee700 10 monclient: _reopen_session rank -1 2023-09-30T01:33:25.024 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:25.009+0000 7f8515cee700 10 monclient: _add_conns ranks=[2,3,0,7,5,4,8,6,1] 2023-09-30T01:33:25.024 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:25.009+0000 7f8515cee700 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f851009ffd0 0x7f851009de60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:33:25.024 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:25.009+0000 7f8515cee700 10 monclient(hunting): picked mon.noname-b con 0x7f851009ffd0 addr [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] 2023-09-30T01:33:25.024 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:25.009+0000 7f8515cee700 1 --2- >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7f85100a03a0 0x7f851009e3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:33:25.024 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:25.009+0000 7f8515cee700 10 monclient(hunting): picked mon.noname-g con 0x7f85100a03a0 addr [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] 2023-09-30T01:33:25.024 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:25.009+0000 7f8515cee700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f85100a3380 0x7f851009e8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:33:25.024 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:25.009+0000 7f8515cee700 10 monclient(hunting): picked mon.noname-a con 0x7f85100a3380 addr [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] 2023-09-30T01:33:25.025 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:25.009+0000 7f8515cee700 10 monclient(hunting): start opening mon connection 2023-09-30T01:33:25.025 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:25.009+0000 7f8515cee700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8510131070 con 0x7f85100a3380 2023-09-30T01:33:25.025 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:25.009+0000 7f8515cee700 10 monclient(hunting): start opening mon connection 2023-09-30T01:33:25.025 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:25.009+0000 7f8515cee700 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f851011f030 con 0x7f851009ffd0 2023-09-30T01:33:25.025 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:25.009+0000 7f8515cee700 10 monclient(hunting): start opening mon connection 2023-09-30T01:33:25.025 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:25.009+0000 7f8515cee700 1 -- --> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f851011f310 con 0x7f85100a03a0 2023-09-30T01:33:25.025 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:25.009+0000 7f8515cee700 10 monclient(hunting): _renew_subs 2023-09-30T01:33:25.025 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:25.009+0000 7f8515cee700 10 monclient(hunting): authenticate will time out at 877.048939s 2023-09-30T01:33:25.025 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:25.009+0000 7f8514cec700 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f851009ffd0 0x7f851009de60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:33:25.025 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:25.009+0000 7f8514cec700 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f851009ffd0 0x7f851009de60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3301/0 says I am v2:172.21.15.88:42480/0 (socket says 172.21.15.88:42480) 2023-09-30T01:33:25.025 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:25.009+0000 7f8514cec700 1 -- 172.21.15.88:0/1295015319 learned_addr learned my addr 172.21.15.88:0/1295015319 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-09-30T01:33:25.025 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:25.009+0000 7f8514cec700 10 monclient(hunting): get_auth_request con 0x7f851009ffd0 auth_method 0 2023-09-30T01:33:25.026 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:25.009+0000 7f8514cec700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:33:25.026 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:25.009+0000 7f8514cec700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:33:25.026 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:25.009+0000 7f8514cec700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:33:25.026 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:25.009+0000 7f85154ed700 1 --2- 172.21.15.88:0/1295015319 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f85100a3380 0x7f851009e8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:33:25.026 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:25.009+0000 7f85154ed700 10 monclient(hunting): get_auth_request con 0x7f85100a3380 auth_method 0 2023-09-30T01:33:25.026 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:25.009+0000 7f85154ed700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:33:25.026 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:25.009+0000 7f85154ed700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:33:25.026 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:25.009+0000 7f85154ed700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:33:25.026 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:25.009+0000 7f8514cec700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:33:25.026 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:25.009+0000 7f8514cec700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:33:25.026 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:25.009+0000 7f8514cec700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:33:25.027 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:25.009+0000 7f8507fff700 1 --2- 172.21.15.88:0/1295015319 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7f85100a03a0 0x7f851009e3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:33:25.027 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:25.009+0000 7f85154ed700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:33:25.027 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:25.009+0000 7f85154ed700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:33:25.027 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:25.009+0000 7f85154ed700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:33:25.027 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:25.009+0000 7f8507fff700 10 monclient(hunting): get_auth_request con 0x7f85100a03a0 auth_method 0 2023-09-30T01:33:25.027 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:25.009+0000 7f8507fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:33:25.027 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:25.009+0000 7f8507fff700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:33:25.027 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:25.009+0000 7f8507fff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:33:25.027 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:25.009+0000 7f8514cec700 10 monclient(hunting): handle_auth_done global_id 4358 payload 824 2023-09-30T01:33:25.027 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:25.009+0000 7f8514cec700 1 -- 172.21.15.88:0/1295015319 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7f85100a03a0 msgr2=0x7f851009e3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:33:25.027 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:25.009+0000 7f8514cec700 1 --2- 172.21.15.88:0/1295015319 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7f85100a03a0 0x7f851009e3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:25.027 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:25.009+0000 7f8514cec700 1 -- 172.21.15.88:0/1295015319 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f85100a3380 msgr2=0x7f851009e8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:33:25.028 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:25.009+0000 7f8514cec700 1 --2- 172.21.15.88:0/1295015319 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f85100a3380 0x7f851009e8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:25.028 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:25.009+0000 7f8514cec700 10 monclient: _finish_hunting 0 2023-09-30T01:33:25.028 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:25.009+0000 7f8514cec700 1 monclient: found mon.noname-b 2023-09-30T01:33:25.028 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:25.009+0000 7f8514cec700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:33:25.028 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:25.009+0000 7f8514cec700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.88:3301/0 2023-09-30T01:33:25.028 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:25.009+0000 7f8514cec700 1 -- 172.21.15.88:0/1295015319 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8508009a20 con 0x7f851009ffd0 2023-09-30T01:33:25.028 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:25.009+0000 7f8514cec700 1 --2- 172.21.15.88:0/1295015319 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f851009ffd0 0x7f851009de60 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f850c00e980 tx=0x7f850c009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:33:25.028 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:25.009+0000 7f8504ff9700 1 -- 172.21.15.88:0/1295015319 <== mon.2 v2:172.21.15.88:3301/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f850c009520 con 0x7f851009ffd0 2023-09-30T01:33:25.028 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:25.009+0000 7f8504ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:33:25.028 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:25.009+0000 7f8504ff9700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T01:33:25.028 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:25.009+0000 7f8504ff9700 10 monclient: dump: 2023-09-30T01:33:25.028 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:33:25.029 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:33:25.029 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:25.029 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:25.029 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:33:25.029 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:33:25.029 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:33:25.029 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:33:25.029 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:33:25.029 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:33:25.029 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:33:25.029 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:33:25.029 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:33:25.030 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:33:25.030 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:33:25.030 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:33:25.030 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:25.009+0000 7f8504ff9700 10 monclient: _finish_auth 0 2023-09-30T01:33:25.030 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:25.009+0000 7f8504ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:33:25.030 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:25.009+0000 7f8515cee700 5 monclient: authenticate success, global_id 4358 2023-09-30T01:33:25.030 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:25.009+0000 7f8504ff9700 1 -- 172.21.15.88:0/1295015319 <== mon.2 v2:172.21.15.88:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f850c00ccd0 con 0x7f851009ffd0 2023-09-30T01:33:25.030 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:25.009+0000 7f8504ff9700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:33:25.030 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:25.009+0000 7f8504ff9700 1 -- 172.21.15.88:0/1295015319 <== mon.2 v2:172.21.15.88:3301/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f850c00ce40 con 0x7f851009ffd0 2023-09-30T01:33:25.030 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:25.009+0000 7f8515cee700 10 monclient: _renew_subs 2023-09-30T01:33:25.030 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:25.009+0000 7f8506ffd700 4 set_mon_vals no callback set 2023-09-30T01:33:25.030 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:25.009+0000 7f8515cee700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.88:3301/0 2023-09-30T01:33:25.031 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:25.009+0000 7f8515cee700 1 -- 172.21.15.88:0/1295015319 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f851011f5f0 con 0x7f851009ffd0 2023-09-30T01:33:25.031 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:25.009+0000 7f8504ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:33:25.031 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:25.009+0000 7f8504ff9700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T01:33:25.031 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:25.009+0000 7f8504ff9700 10 monclient: dump: 2023-09-30T01:33:25.031 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:33:25.031 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:33:25.031 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:25.031 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:25.031 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:33:25.031 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:33:25.031 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:33:25.031 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:33:25.032 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:33:25.032 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:33:25.032 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:33:25.032 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:33:25.032 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:33:25.032 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:33:25.032 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:33:25.032 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:33:25.032 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:25.009+0000 7f8515cee700 10 monclient: _renew_subs 2023-09-30T01:33:25.032 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:25.009+0000 7f8515cee700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.88:3301/0 2023-09-30T01:33:25.032 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:25.009+0000 7f8515cee700 1 -- 172.21.15.88:0/1295015319 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f851011e140 con 0x7f851009ffd0 2023-09-30T01:33:25.033 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:25.009+0000 7f8504ff9700 1 -- 172.21.15.88:0/1295015319 <== mon.2 v2:172.21.15.88:3301/0 4 ==== mgrmap(e 5) v1 ==== 40144+0+0 (secure 0 0 0) 0x7f850c00c850 con 0x7f851009ffd0 2023-09-30T01:33:25.034 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:25.013+0000 7f8504ff9700 1 --2- 172.21.15.88:0/1295015319 >> v2:172.21.15.107:6812/29926 conn(0x7f84fc033290 0x7f84fc035740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:33:25.034 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:25.013+0000 7f8504ff9700 1 -- 172.21.15.88:0/1295015319 <== mon.2 v2:172.21.15.88:3301/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 3502+0+0 (secure 0 0 0) 0x7f850c043690 con 0x7f851009ffd0 2023-09-30T01:33:25.034 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:25.013+0000 7f8507fff700 1 --2- 172.21.15.88:0/1295015319 >> v2:172.21.15.107:6812/29926 conn(0x7f84fc033290 0x7f84fc035740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:33:25.034 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:25.013+0000 7f8515cee700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:33:25.034 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:25.013+0000 7f8515cee700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:33:25.034 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:25.013+0000 7f8515cee700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.88:3301/0 2023-09-30T01:33:25.034 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:25.013+0000 7f8515cee700 1 -- 172.21.15.88:0/1295015319 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f851011f780 con 0x7f851009ffd0 2023-09-30T01:33:25.034 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:25.013+0000 7f8507fff700 10 monclient: get_auth_request con 0x7f84fc033290 auth_method 0 2023-09-30T01:33:25.034 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:25.013+0000 7f8507fff700 1 --2- 172.21.15.88:0/1295015319 >> v2:172.21.15.107:6812/29926 conn(0x7f84fc033290 0x7f84fc035740 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f8500003880 tx=0x7f8500009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:33:25.034 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:25.021+0000 7f8504ff9700 1 -- 172.21.15.88:0/1295015319 <== mon.2 v2:172.21.15.88:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f850c00cb00 con 0x7f851009ffd0 2023-09-30T01:33:25.035 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:25.021+0000 7f8504ff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:33:25.035 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:25.021+0000 7f8504ff9700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:33:25.215 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:25.209+0000 7f8515cee700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T01:33:25.215 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:25.209+0000 7f8515cee700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:33:25.215 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:25.209+0000 7f8515cee700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.88:3301/0 2023-09-30T01:33:25.215 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:25.209+0000 7f8515cee700 1 -- 172.21.15.88:0/1295015319 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f85100710d0 con 0x7f851009ffd0 2023-09-30T01:33:25.215 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:25.213+0000 7f8504ff9700 1 -- 172.21.15.88:0/1295015319 <== mon.2 v2:172.21.15.88:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f850c01bd00 con 0x7f851009ffd0 2023-09-30T01:33:25.216 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:25.213+0000 7f8504ff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:33:25.216 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:25.213+0000 7f8504ff9700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:33:25.216 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-09-30T01:33:25.216 INFO:teuthology.orchestra.run.smithi088.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:33:25.220 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:25.217+0000 7f8515cee700 1 -- 172.21.15.88:0/1295015319 >> v2:172.21.15.107:6812/29926 conn(0x7f84fc033290 msgr2=0x7f84fc035740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:33:25.220 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:25.217+0000 7f8515cee700 1 --2- 172.21.15.88:0/1295015319 >> v2:172.21.15.107:6812/29926 conn(0x7f84fc033290 0x7f84fc035740 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f8500003880 tx=0x7f8500009040 comp rx=0 tx=0).stop 2023-09-30T01:33:25.220 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:25.217+0000 7f8515cee700 10 monclient: shutdown 2023-09-30T01:33:25.220 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:25.217+0000 7f8515cee700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:33:25.220 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:25.217+0000 7f8515cee700 1 -- 172.21.15.88:0/1295015319 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f851009ffd0 msgr2=0x7f851009de60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:33:25.220 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:25.217+0000 7f8515cee700 1 --2- 172.21.15.88:0/1295015319 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f851009ffd0 0x7f851009de60 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f850c00e980 tx=0x7f850c009250 comp rx=0 tx=0).stop 2023-09-30T01:33:25.221 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:25.217+0000 7f8515cee700 1 -- 172.21.15.88:0/1295015319 shutdown_connections 2023-09-30T01:33:25.221 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:25.217+0000 7f8515cee700 1 --2- 172.21.15.88:0/1295015319 >> v2:172.21.15.107:6812/29926 conn(0x7f84fc033290 0x7f84fc035740 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:25.221 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:25.217+0000 7f8515cee700 1 --2- 172.21.15.88:0/1295015319 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f85100a3380 0x7f851009e8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:25.221 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:25.217+0000 7f8515cee700 1 --2- 172.21.15.88:0/1295015319 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7f85100a03a0 0x7f851009e3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:25.221 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:25.217+0000 7f8515cee700 1 --2- 172.21.15.88:0/1295015319 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f851009ffd0 0x7f851009de60 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:25.221 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:25.217+0000 7f8515cee700 1 -- 172.21.15.88:0/1295015319 >> 172.21.15.88:0/1295015319 conn(0x7f851006e5e0 msgr2=0x7f8510134580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:33:25.221 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:25.217+0000 7f8515cee700 1 -- 172.21.15.88:0/1295015319 shutdown_connections 2023-09-30T01:33:25.221 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:25.217+0000 7f8515cee700 1 -- 172.21.15.88:0/1295015319 wait complete. 2023-09-30T01:33:28.236 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:33:28.345 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:28.341+0000 7fb36ce3c700 10 monclient: get_monmap_and_config 2023-09-30T01:33:28.346 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:28.341+0000 7fb36ce3c700 10 monclient: build_initial_monmap 2023-09-30T01:33:28.346 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:28.341+0000 7fb36ce3c700 10 monclient: monmap: 2023-09-30T01:33:28.346 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2023-09-30T01:33:28.346 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:33:28.346 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:33:28.346375+0000 2023-09-30T01:33:28.346 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:33:28.346375+0000 2023-09-30T01:33:28.346 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2023-09-30T01:33:28.346 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:33:28.346 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a-legacy 2023-09-30T01:33:28.346 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.107:6789/0 mon.noname-f-legacy 2023-09-30T01:33:28.347 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.noname-b 2023-09-30T01:33:28.347 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.noname-g 2023-09-30T01:33:28.347 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.noname-c 2023-09-30T01:33:28.347 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.noname-h 2023-09-30T01:33:28.347 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.noname-d 2023-09-30T01:33:28.347 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.noname-i 2023-09-30T01:33:28.347 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.noname-e 2023-09-30T01:33:28.347 INFO:teuthology.orchestra.run.smithi088.stderr:9: v2:172.21.15.88:3300/0 mon.noname-a 2023-09-30T01:33:28.347 INFO:teuthology.orchestra.run.smithi088.stderr:10: v2:172.21.15.107:3300/0 mon.noname-f 2023-09-30T01:33:28.347 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:33:28.354 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:28.341+0000 7fb36ce3c700 1 Processor -- start 2023-09-30T01:33:28.354 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:28.341+0000 7fb36ce3c700 1 -- start start 2023-09-30T01:33:28.354 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:28.341+0000 7fb36ce3c700 10 monclient: init 2023-09-30T01:33:28.354 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:28.341+0000 7fb36ce3c700 10 monclient: _reopen_session rank -1 2023-09-30T01:33:28.354 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:28.341+0000 7fb36ce3c700 10 monclient: _add_conns ranks=[10,3,7,9,4,0,2,6,8,5,1] 2023-09-30T01:33:28.354 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:28.341+0000 7fb36ce3c700 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fb368097a40 0x7fb368132700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:33:28.354 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:28.341+0000 7fb36ce3c700 10 monclient(hunting): picked mon.noname-f con 0x7fb368097a40 addr v2:172.21.15.107:3300/0 2023-09-30T01:33:28.354 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:28.341+0000 7fb36ce3c700 1 --2- >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7fb3681311c0 0x7fb368132c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:33:28.354 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:28.341+0000 7fb36ce3c700 10 monclient(hunting): picked mon.noname-g con 0x7fb3681311c0 addr [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] 2023-09-30T01:33:28.355 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:28.341+0000 7fb36ce3c700 1 --2- >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7fb3681332e0 0x7fb368133740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:33:28.355 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:28.341+0000 7fb36ce3c700 10 monclient(hunting): picked mon.noname-i con 0x7fb3681332e0 addr [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] 2023-09-30T01:33:28.355 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:28.341+0000 7fb36ce3c700 10 monclient(hunting): start opening mon connection 2023-09-30T01:33:28.355 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:28.341+0000 7fb36ce3c700 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb368131590 con 0x7fb368097a40 2023-09-30T01:33:28.355 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:28.341+0000 7fb36ce3c700 10 monclient(hunting): start opening mon connection 2023-09-30T01:33:28.355 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:28.341+0000 7fb36ce3c700 1 -- --> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb36809a5d0 con 0x7fb3681311c0 2023-09-30T01:33:28.355 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:28.341+0000 7fb36ce3c700 10 monclient(hunting): start opening mon connection 2023-09-30T01:33:28.355 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:28.341+0000 7fb36ce3c700 1 -- --> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] -- mon_getmap magic: 0 v1 -- 0x7fb368097e10 con 0x7fb3681332e0 2023-09-30T01:33:28.355 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:28.341+0000 7fb36ce3c700 10 monclient(hunting): _renew_subs 2023-09-30T01:33:28.355 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:28.341+0000 7fb36ce3c700 10 monclient(hunting): authenticate will time out at 880.383877s 2023-09-30T01:33:28.355 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:28.345+0000 7fb366ffd700 1 --2- >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7fb3681311c0 0x7fb368132c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:33:28.355 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:28.345+0000 7fb366ffd700 1 --2- >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7fb3681311c0 0x7fb368132c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3301/0 says I am v2:172.21.15.88:60834/0 (socket says 172.21.15.88:60834) 2023-09-30T01:33:28.356 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:28.345+0000 7fb3677fe700 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fb368097a40 0x7fb368132700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:33:28.356 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:28.345+0000 7fb366ffd700 1 -- 172.21.15.88:0/1271020827 learned_addr learned my addr 172.21.15.88:0/1271020827 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-09-30T01:33:28.356 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:28.345+0000 7fb366ffd700 10 monclient(hunting): get_auth_request con 0x7fb3681311c0 auth_method 0 2023-09-30T01:33:28.356 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:28.345+0000 7fb366ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:33:28.356 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:28.345+0000 7fb366ffd700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:33:28.356 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:28.345+0000 7fb366ffd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:33:28.356 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:28.345+0000 7fb3677fe700 10 monclient(hunting): get_auth_request con 0x7fb368097a40 auth_method 0 2023-09-30T01:33:28.356 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:28.345+0000 7fb3677fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:33:28.356 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:28.345+0000 7fb3677fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:33:28.356 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:28.345+0000 7fb3677fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:33:28.356 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:28.345+0000 7fb367fff700 1 --2- 172.21.15.88:0/1271020827 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7fb3681332e0 0x7fb368133740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:33:28.356 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:28.345+0000 7fb366ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:33:28.356 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:28.345+0000 7fb366ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:33:28.357 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:28.345+0000 7fb366ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:33:28.357 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:28.345+0000 7fb367fff700 10 monclient(hunting): get_auth_request con 0x7fb3681332e0 auth_method 0 2023-09-30T01:33:28.357 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:28.345+0000 7fb367fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:33:28.357 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:28.345+0000 7fb367fff700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:33:28.357 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:28.345+0000 7fb367fff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:33:28.357 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:28.345+0000 7fb3677fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:33:28.357 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:28.345+0000 7fb3677fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:33:28.357 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:28.345+0000 7fb3677fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:33:28.357 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:28.345+0000 7fb367fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:33:28.357 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:28.345+0000 7fb367fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:33:28.357 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:28.345+0000 7fb367fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:33:28.357 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:28.345+0000 7fb366ffd700 10 monclient(hunting): handle_auth_done global_id 4359 payload 274 2023-09-30T01:33:28.357 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:28.345+0000 7fb366ffd700 1 -- 172.21.15.88:0/1271020827 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7fb3681332e0 msgr2=0x7fb368133740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:33:28.358 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:28.345+0000 7fb366ffd700 1 --2- 172.21.15.88:0/1271020827 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7fb3681332e0 0x7fb368133740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:28.358 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:28.345+0000 7fb366ffd700 1 -- 172.21.15.88:0/1271020827 >> v2:172.21.15.107:3300/0 conn(0x7fb368097a40 msgr2=0x7fb368132700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:33:28.358 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:28.345+0000 7fb366ffd700 1 --2- 172.21.15.88:0/1271020827 >> v2:172.21.15.107:3300/0 conn(0x7fb368097a40 0x7fb368132700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:28.358 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:28.345+0000 7fb366ffd700 10 monclient: _finish_hunting 0 2023-09-30T01:33:28.358 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:28.345+0000 7fb366ffd700 1 monclient: found mon.noname-g 2023-09-30T01:33:28.358 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:28.345+0000 7fb366ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:33:28.358 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:28.345+0000 7fb366ffd700 10 monclient: _send_mon_message to mon.noname-g at v2:172.21.15.107:3301/0 2023-09-30T01:33:28.358 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:28.345+0000 7fb366ffd700 1 -- 172.21.15.88:0/1271020827 --> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb36804bb20 con 0x7fb3681311c0 2023-09-30T01:33:28.358 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:28.345+0000 7fb366ffd700 1 --2- 172.21.15.88:0/1271020827 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7fb3681311c0 0x7fb368132c40 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fb358009f20 tx=0x7fb358009840 comp rx=0 tx=0).ready entity=mon.3 client_cookie=b81ec179f5e82551 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:33:28.358 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:28.345+0000 7fb3667fc700 1 -- 172.21.15.88:0/1271020827 <== mon.3 v2:172.21.15.107:3301/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7fb35800e070 con 0x7fb3681311c0 2023-09-30T01:33:28.358 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:28.345+0000 7fb3667fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:33:28.358 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:28.345+0000 7fb3667fc700 10 monclient: got monmap 1 from mon.noname-g (according to old e1) 2023-09-30T01:33:28.359 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:28.345+0000 7fb3667fc700 10 monclient: dump: 2023-09-30T01:33:28.359 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:33:28.359 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:33:28.359 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:28.359 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:28.359 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:33:28.359 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:33:28.359 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:33:28.359 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:33:28.359 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:33:28.359 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:33:28.359 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:33:28.359 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:33:28.360 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:33:28.360 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:33:28.360 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:33:28.360 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:33:28.360 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:28.345+0000 7fb3667fc700 10 monclient: _finish_auth 0 2023-09-30T01:33:28.360 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:28.345+0000 7fb3667fc700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:33:28.360 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:28.345+0000 7fb36ce3c700 5 monclient: authenticate success, global_id 4359 2023-09-30T01:33:28.360 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:28.345+0000 7fb36ce3c700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:33:28.360 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:28.345+0000 7fb3667fc700 1 -- 172.21.15.88:0/1271020827 <== mon.3 v2:172.21.15.107:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb35800bdf0 con 0x7fb3681311c0 2023-09-30T01:33:28.360 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:28.345+0000 7fb3667fc700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:33:28.360 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:28.345+0000 7fb36ce3c700 10 monclient: get_monmap_and_config success 2023-09-30T01:33:28.360 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:28.345+0000 7fb3667fc700 1 -- 172.21.15.88:0/1271020827 <== mon.3 v2:172.21.15.107:3301/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7fb3580092e0 con 0x7fb3681311c0 2023-09-30T01:33:28.360 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:28.345+0000 7fb36ce3c700 4 set_mon_vals no callback set 2023-09-30T01:33:28.361 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:28.345+0000 7fb3667fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:33:28.361 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:28.345+0000 7fb3667fc700 10 monclient: got monmap 1 from mon.g (according to old e1) 2023-09-30T01:33:28.361 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:28.345+0000 7fb3667fc700 10 monclient: dump: 2023-09-30T01:33:28.361 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:33:28.361 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:33:28.361 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:28.361 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:28.361 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:33:28.361 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:33:28.361 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:33:28.361 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:33:28.361 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:33:28.361 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:33:28.362 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:33:28.362 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:33:28.362 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:33:28.362 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:33:28.362 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:33:28.362 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:33:28.362 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:28.345+0000 7fb36ce3c700 10 monclient: shutdown 2023-09-30T01:33:28.362 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:28.345+0000 7fb36ce3c700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:33:28.362 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:28.345+0000 7fb36ce3c700 1 -- 172.21.15.88:0/1271020827 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7fb3681311c0 msgr2=0x7fb368132c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:33:28.362 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:28.345+0000 7fb36ce3c700 1 --2- 172.21.15.88:0/1271020827 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7fb3681311c0 0x7fb368132c40 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fb358009f20 tx=0x7fb358009840 comp rx=0 tx=0).stop 2023-09-30T01:33:28.362 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:28.345+0000 7fb36ce3c700 1 -- 172.21.15.88:0/1271020827 shutdown_connections 2023-09-30T01:33:28.362 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:28.345+0000 7fb36ce3c700 1 --2- 172.21.15.88:0/1271020827 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7fb3681332e0 0x7fb368133740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:28.363 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:28.345+0000 7fb36ce3c700 1 --2- 172.21.15.88:0/1271020827 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7fb3681311c0 0x7fb368132c40 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:28.363 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:28.345+0000 7fb36ce3c700 1 --2- 172.21.15.88:0/1271020827 >> v2:172.21.15.107:3300/0 conn(0x7fb368097a40 0x7fb368132700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:28.363 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:28.345+0000 7fb36ce3c700 1 -- 172.21.15.88:0/1271020827 >> 172.21.15.88:0/1271020827 conn(0x7fb36806e5e0 msgr2=0x7fb36806e9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:33:28.363 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:28.345+0000 7fb36ce3c700 1 -- 172.21.15.88:0/1271020827 shutdown_connections 2023-09-30T01:33:28.363 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:28.345+0000 7fb36ce3c700 1 -- 172.21.15.88:0/1271020827 wait complete. 2023-09-30T01:33:28.363 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:28.345+0000 7fb36ce3c700 10 monclient: build_initial_monmap 2023-09-30T01:33:28.363 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:28.345+0000 7fb36ce3c700 10 monclient: monmap: 2023-09-30T01:33:28.363 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2023-09-30T01:33:28.363 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:33:28.363 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 0.000000 2023-09-30T01:33:28.363 INFO:teuthology.orchestra.run.smithi088.stderr:created 0.000000 2023-09-30T01:33:28.363 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2023-09-30T01:33:28.363 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 1 2023-09-30T01:33:28.364 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.noname-a 2023-09-30T01:33:28.364 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.noname-f 2023-09-30T01:33:28.364 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.noname-b 2023-09-30T01:33:28.364 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.noname-g 2023-09-30T01:33:28.364 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.noname-c 2023-09-30T01:33:28.364 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.noname-h 2023-09-30T01:33:28.364 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.noname-d 2023-09-30T01:33:28.364 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.noname-i 2023-09-30T01:33:28.364 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.noname-e 2023-09-30T01:33:28.364 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:33:28.364 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:28.345+0000 7fb36ce3c700 1 Processor -- start 2023-09-30T01:33:28.364 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:28.345+0000 7fb36ce3c700 1 -- start start 2023-09-30T01:33:28.364 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:28.345+0000 7fb36ce3c700 10 monclient: init 2023-09-30T01:33:28.364 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:28.345+0000 7fb36ce3c700 10 monclient: _reopen_session rank -1 2023-09-30T01:33:28.365 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:28.345+0000 7fb36ce3c700 10 monclient: _add_conns ranks=[5,7,4,1,0,3,6,8,2] 2023-09-30T01:33:28.365 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:28.345+0000 7fb36ce3c700 1 --2- >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7fb368097a40 0x7fb3680a1e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:33:28.365 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:28.345+0000 7fb36ce3c700 10 monclient(hunting): picked mon.noname-h con 0x7fb368097a40 addr [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] 2023-09-30T01:33:28.365 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:28.345+0000 7fb36ce3c700 1 --2- >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7fb3681311c0 0x7fb3680a2380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:33:28.365 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:28.345+0000 7fb36ce3c700 10 monclient(hunting): picked mon.noname-i con 0x7fb3681311c0 addr [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] 2023-09-30T01:33:28.365 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:28.345+0000 7fb36ce3c700 1 --2- >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7fb3681332e0 0x7fb3680a28c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:33:28.365 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:28.345+0000 7fb36ce3c700 10 monclient(hunting): picked mon.noname-c con 0x7fb3681332e0 addr [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] 2023-09-30T01:33:28.365 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:28.345+0000 7fb36ce3c700 10 monclient(hunting): start opening mon connection 2023-09-30T01:33:28.365 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:28.345+0000 7fb36ce3c700 1 -- --> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] -- mon_getmap magic: 0 v1 -- 0x7fb368099c30 con 0x7fb3681332e0 2023-09-30T01:33:28.365 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:28.345+0000 7fb36ce3c700 10 monclient(hunting): start opening mon connection 2023-09-30T01:33:28.365 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:28.345+0000 7fb36ce3c700 1 -- --> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] -- mon_getmap magic: 0 v1 -- 0x7fb3680996b0 con 0x7fb368097a40 2023-09-30T01:33:28.365 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:28.345+0000 7fb36ce3c700 10 monclient(hunting): start opening mon connection 2023-09-30T01:33:28.365 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:28.345+0000 7fb36ce3c700 1 -- --> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] -- mon_getmap magic: 0 v1 -- 0x7fb36811ee80 con 0x7fb3681311c0 2023-09-30T01:33:28.366 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:28.345+0000 7fb36ce3c700 10 monclient(hunting): _renew_subs 2023-09-30T01:33:28.366 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:28.345+0000 7fb36ce3c700 10 monclient(hunting): authenticate will time out at 880.387837s 2023-09-30T01:33:28.366 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:28.349+0000 7fb366ffd700 1 --2- >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7fb3681311c0 0x7fb3680a2380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:33:28.366 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:28.349+0000 7fb367fff700 1 --2- >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7fb3681332e0 0x7fb3680a28c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:33:28.366 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:28.349+0000 7fb366ffd700 1 --2- >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7fb3681311c0 0x7fb3680a2380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3303/0 says I am v2:172.21.15.88:37662/0 (socket says 172.21.15.88:37662) 2023-09-30T01:33:28.366 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:28.349+0000 7fb366ffd700 1 -- 172.21.15.88:0/456759762 learned_addr learned my addr 172.21.15.88:0/456759762 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-09-30T01:33:28.366 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:28.349+0000 7fb366ffd700 10 monclient(hunting): get_auth_request con 0x7fb3681311c0 auth_method 0 2023-09-30T01:33:28.366 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:28.349+0000 7fb366ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:33:28.366 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:28.349+0000 7fb366ffd700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:33:28.366 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:28.349+0000 7fb366ffd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:33:28.366 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:28.349+0000 7fb3677fe700 1 --2- 172.21.15.88:0/456759762 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7fb368097a40 0x7fb3680a1e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:33:28.366 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:28.349+0000 7fb3677fe700 10 monclient(hunting): get_auth_request con 0x7fb368097a40 auth_method 0 2023-09-30T01:33:28.367 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:28.349+0000 7fb3677fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:33:28.367 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:28.349+0000 7fb3677fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:33:28.367 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:28.349+0000 7fb3677fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:33:28.367 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:28.349+0000 7fb367fff700 10 monclient(hunting): get_auth_request con 0x7fb3681332e0 auth_method 0 2023-09-30T01:33:28.367 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:28.349+0000 7fb367fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:33:28.367 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:28.349+0000 7fb367fff700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:33:28.367 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:28.349+0000 7fb367fff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:33:28.367 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:28.349+0000 7fb366ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:33:28.367 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:28.349+0000 7fb366ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:33:28.367 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:28.349+0000 7fb366ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:33:28.367 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:28.349+0000 7fb3677fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:33:28.367 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:28.349+0000 7fb3677fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:33:28.367 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:28.349+0000 7fb3677fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:33:28.367 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:28.349+0000 7fb367fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:33:28.368 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:28.349+0000 7fb367fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:33:28.368 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:28.349+0000 7fb367fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:33:28.368 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:28.349+0000 7fb3677fe700 10 monclient(hunting): handle_auth_done global_id 4352 payload 824 2023-09-30T01:33:28.368 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:28.349+0000 7fb3677fe700 1 -- 172.21.15.88:0/456759762 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7fb3681311c0 msgr2=0x7fb3680a2380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:33:28.368 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:28.349+0000 7fb3677fe700 1 --2- 172.21.15.88:0/456759762 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7fb3681311c0 0x7fb3680a2380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:28.368 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:28.349+0000 7fb3677fe700 1 -- 172.21.15.88:0/456759762 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7fb3681332e0 msgr2=0x7fb3680a28c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:33:28.368 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:28.349+0000 7fb3677fe700 1 --2- 172.21.15.88:0/456759762 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7fb3681332e0 0x7fb3680a28c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:28.368 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:28.349+0000 7fb3677fe700 10 monclient: _finish_hunting 0 2023-09-30T01:33:28.368 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:28.349+0000 7fb3677fe700 1 monclient: found mon.noname-h 2023-09-30T01:33:28.368 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:28.349+0000 7fb3677fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:33:28.368 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:28.349+0000 7fb3677fe700 10 monclient: _send_mon_message to mon.noname-h at v2:172.21.15.107:3302/0 2023-09-30T01:33:28.368 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:28.349+0000 7fb3677fe700 1 -- 172.21.15.88:0/456759762 --> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb358009ae0 con 0x7fb368097a40 2023-09-30T01:33:28.368 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:28.349+0000 7fb3677fe700 1 --2- 172.21.15.88:0/456759762 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7fb368097a40 0x7fb3680a1e40 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fb35c00ea50 tx=0x7fb35c0091a0 comp rx=0 tx=0).ready entity=mon.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:33:28.369 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:28.349+0000 7fb34f7fe700 1 -- 172.21.15.88:0/456759762 <== mon.5 v2:172.21.15.107:3302/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7fb35c0095d0 con 0x7fb368097a40 2023-09-30T01:33:28.369 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:28.349+0000 7fb34f7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:33:28.369 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:28.349+0000 7fb34f7fe700 10 monclient: got monmap 1 from mon.noname-h (according to old e1) 2023-09-30T01:33:28.369 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:28.349+0000 7fb34f7fe700 10 monclient: dump: 2023-09-30T01:33:28.369 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:33:28.369 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:33:28.369 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:28.369 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:28.369 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:33:28.369 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:33:28.369 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:33:28.369 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:33:28.369 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:33:28.369 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:33:28.369 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:33:28.370 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:33:28.370 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:33:28.370 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:33:28.370 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:33:28.370 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:33:28.370 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:28.349+0000 7fb34f7fe700 10 monclient: _finish_auth 0 2023-09-30T01:33:28.370 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:28.349+0000 7fb34f7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:33:28.370 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:28.349+0000 7fb34f7fe700 1 -- 172.21.15.88:0/456759762 <== mon.5 v2:172.21.15.107:3302/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb35c00cc70 con 0x7fb368097a40 2023-09-30T01:33:28.370 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:28.349+0000 7fb36ce3c700 5 monclient: authenticate success, global_id 4352 2023-09-30T01:33:28.370 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:28.349+0000 7fb36ce3c700 10 monclient: _renew_subs 2023-09-30T01:33:28.370 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:28.349+0000 7fb36ce3c700 10 monclient: _send_mon_message to mon.h at v2:172.21.15.107:3302/0 2023-09-30T01:33:28.370 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:28.349+0000 7fb36ce3c700 1 -- 172.21.15.88:0/456759762 --> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb36811f3b0 con 0x7fb368097a40 2023-09-30T01:33:28.370 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:28.349+0000 7fb34f7fe700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:33:28.371 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:28.349+0000 7fb34f7fe700 1 -- 172.21.15.88:0/456759762 <== mon.5 v2:172.21.15.107:3302/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7fb35c00cde0 con 0x7fb368097a40 2023-09-30T01:33:28.371 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:28.349+0000 7fb34f7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:33:28.371 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:28.349+0000 7fb34f7fe700 10 monclient: got monmap 1 from mon.h (according to old e1) 2023-09-30T01:33:28.371 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:28.349+0000 7fb34f7fe700 10 monclient: dump: 2023-09-30T01:33:28.371 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:33:28.371 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:33:28.371 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:28.371 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:28.371 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:33:28.371 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:33:28.371 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:33:28.371 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:33:28.371 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:33:28.372 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:33:28.372 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:33:28.372 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:33:28.372 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:33:28.372 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:33:28.372 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:33:28.372 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:33:28.372 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:28.349+0000 7fb36ce3c700 10 monclient: _renew_subs 2023-09-30T01:33:28.372 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:28.349+0000 7fb36ce3c700 10 monclient: _send_mon_message to mon.h at v2:172.21.15.107:3302/0 2023-09-30T01:33:28.372 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:28.349+0000 7fb36ce3c700 1 -- 172.21.15.88:0/456759762 --> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb36811d2c0 con 0x7fb368097a40 2023-09-30T01:33:28.372 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:28.349+0000 7fb365ffb700 4 set_mon_vals no callback set 2023-09-30T01:33:28.372 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:28.349+0000 7fb36ce3c700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:33:28.373 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:28.349+0000 7fb36ce3c700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:33:28.373 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:28.349+0000 7fb36ce3c700 10 monclient: _send_mon_message to mon.h at v2:172.21.15.107:3302/0 2023-09-30T01:33:28.373 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:28.349+0000 7fb36ce3c700 1 -- 172.21.15.88:0/456759762 --> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb3680a64a0 con 0x7fb368097a40 2023-09-30T01:33:28.373 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:28.349+0000 7fb34f7fe700 1 -- 172.21.15.88:0/456759762 <== mon.5 v2:172.21.15.107:3302/0 4 ==== mgrmap(e 5) v1 ==== 40144+0+0 (secure 0 0 0) 0x7fb35c00c7f0 con 0x7fb368097a40 2023-09-30T01:33:28.373 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:28.349+0000 7fb34f7fe700 1 --2- 172.21.15.88:0/456759762 >> v2:172.21.15.107:6812/29926 conn(0x7fb354033360 0x7fb354035810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:33:28.373 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:28.349+0000 7fb34f7fe700 1 -- 172.21.15.88:0/456759762 <== mon.5 v2:172.21.15.107:3302/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 3502+0+0 (secure 0 0 0) 0x7fb35c043630 con 0x7fb368097a40 2023-09-30T01:33:28.374 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:28.349+0000 7fb366ffd700 1 --2- 172.21.15.88:0/456759762 >> v2:172.21.15.107:6812/29926 conn(0x7fb354033360 0x7fb354035810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:33:28.374 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:28.349+0000 7fb366ffd700 10 monclient: get_auth_request con 0x7fb354033360 auth_method 0 2023-09-30T01:33:28.374 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:28.353+0000 7fb366ffd700 1 --2- 172.21.15.88:0/456759762 >> v2:172.21.15.107:6812/29926 conn(0x7fb354033360 0x7fb354035810 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fb358003440 tx=0x7fb358018000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:33:28.374 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:28.361+0000 7fb34f7fe700 1 -- 172.21.15.88:0/456759762 <== mon.5 v2:172.21.15.107:3302/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fb35c01be10 con 0x7fb368097a40 2023-09-30T01:33:28.374 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:28.361+0000 7fb34f7fe700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:33:28.374 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:28.361+0000 7fb34f7fe700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:33:28.565 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:28.561+0000 7fb36ce3c700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T01:33:28.565 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:28.561+0000 7fb36ce3c700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:33:28.565 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:28.561+0000 7fb36ce3c700 10 monclient: _send_mon_message to mon.h at v2:172.21.15.107:3302/0 2023-09-30T01:33:28.565 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:28.561+0000 7fb36ce3c700 1 -- 172.21.15.88:0/456759762 --> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fb368071960 con 0x7fb368097a40 2023-09-30T01:33:28.566 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:28.561+0000 7fb34f7fe700 1 -- 172.21.15.88:0/456759762 <== mon.5 v2:172.21.15.107:3302/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7fb35c01b3b0 con 0x7fb368097a40 2023-09-30T01:33:28.566 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:28.561+0000 7fb34f7fe700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:33:28.566 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:28.561+0000 7fb34f7fe700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:33:28.566 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-09-30T01:33:28.566 INFO:teuthology.orchestra.run.smithi088.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:33:28.568 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:28.565+0000 7fb36ce3c700 1 -- 172.21.15.88:0/456759762 >> v2:172.21.15.107:6812/29926 conn(0x7fb354033360 msgr2=0x7fb354035810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:33:28.568 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:28.565+0000 7fb36ce3c700 1 --2- 172.21.15.88:0/456759762 >> v2:172.21.15.107:6812/29926 conn(0x7fb354033360 0x7fb354035810 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fb358003440 tx=0x7fb358018000 comp rx=0 tx=0).stop 2023-09-30T01:33:28.568 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:28.565+0000 7fb36ce3c700 10 monclient: shutdown 2023-09-30T01:33:28.568 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:28.565+0000 7fb36ce3c700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:33:28.568 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:28.565+0000 7fb36ce3c700 1 -- 172.21.15.88:0/456759762 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7fb368097a40 msgr2=0x7fb3680a1e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:33:28.568 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:28.565+0000 7fb36ce3c700 1 --2- 172.21.15.88:0/456759762 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7fb368097a40 0x7fb3680a1e40 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fb35c00ea50 tx=0x7fb35c0091a0 comp rx=0 tx=0).stop 2023-09-30T01:33:28.568 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:28.565+0000 7fb36ce3c700 1 -- 172.21.15.88:0/456759762 shutdown_connections 2023-09-30T01:33:28.569 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:28.565+0000 7fb36ce3c700 1 --2- 172.21.15.88:0/456759762 >> v2:172.21.15.107:6812/29926 conn(0x7fb354033360 0x7fb354035810 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:28.569 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:28.565+0000 7fb36ce3c700 1 --2- 172.21.15.88:0/456759762 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7fb3681332e0 0x7fb3680a28c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:28.569 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:28.565+0000 7fb36ce3c700 1 --2- 172.21.15.88:0/456759762 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7fb3681311c0 0x7fb3680a2380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:28.569 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:28.565+0000 7fb36ce3c700 1 --2- 172.21.15.88:0/456759762 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7fb368097a40 0x7fb3680a1e40 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:28.569 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:28.565+0000 7fb36ce3c700 1 -- 172.21.15.88:0/456759762 >> 172.21.15.88:0/456759762 conn(0x7fb36806e5e0 msgr2=0x7fb3681328e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:33:28.569 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:28.565+0000 7fb36ce3c700 1 -- 172.21.15.88:0/456759762 shutdown_connections 2023-09-30T01:33:28.569 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:28.565+0000 7fb36ce3c700 1 -- 172.21.15.88:0/456759762 wait complete. 2023-09-30T01:33:31.580 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:33:31.692 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:31.689+0000 7fbe06012700 10 monclient: get_monmap_and_config 2023-09-30T01:33:31.693 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:31.689+0000 7fbe06012700 10 monclient: build_initial_monmap 2023-09-30T01:33:31.693 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:31.689+0000 7fbe06012700 10 monclient: monmap: 2023-09-30T01:33:31.693 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2023-09-30T01:33:31.693 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:33:31.693 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:33:31.693346+0000 2023-09-30T01:33:31.693 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:33:31.693346+0000 2023-09-30T01:33:31.693 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2023-09-30T01:33:31.693 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:33:31.693 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a-legacy 2023-09-30T01:33:31.694 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.107:6789/0 mon.noname-f-legacy 2023-09-30T01:33:31.694 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.noname-b 2023-09-30T01:33:31.694 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.noname-g 2023-09-30T01:33:31.694 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.noname-c 2023-09-30T01:33:31.694 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.noname-h 2023-09-30T01:33:31.694 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.noname-d 2023-09-30T01:33:31.694 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.noname-i 2023-09-30T01:33:31.694 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.noname-e 2023-09-30T01:33:31.694 INFO:teuthology.orchestra.run.smithi088.stderr:9: v2:172.21.15.88:3300/0 mon.noname-a 2023-09-30T01:33:31.694 INFO:teuthology.orchestra.run.smithi088.stderr:10: v2:172.21.15.107:3300/0 mon.noname-f 2023-09-30T01:33:31.695 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:33:31.704 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:31.689+0000 7fbe06012700 1 Processor -- start 2023-09-30T01:33:31.704 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:31.689+0000 7fbe06012700 1 -- start start 2023-09-30T01:33:31.704 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:31.689+0000 7fbe06012700 10 monclient: init 2023-09-30T01:33:31.704 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:31.689+0000 7fbe06012700 10 monclient: _reopen_session rank -1 2023-09-30T01:33:31.704 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:31.689+0000 7fbe06012700 10 monclient: _add_conns ranks=[6,2,10,0,9,8,5,1,3,7,4] 2023-09-30T01:33:31.704 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:31.689+0000 7fbe06012700 1 --2- >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7fbe000a3270 0x7fbe000a0160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:33:31.704 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:31.689+0000 7fbe06012700 10 monclient(hunting): picked mon.noname-d con 0x7fbe000a3270 addr [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] 2023-09-30T01:33:31.704 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:31.689+0000 7fbe06012700 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fbe000a4b00 0x7fbe000a06a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:33:31.704 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:31.689+0000 7fbe06012700 10 monclient(hunting): picked mon.noname-b con 0x7fbe000a4b00 addr [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] 2023-09-30T01:33:31.705 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:31.689+0000 7fbe06012700 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fbe001333c0 0x7fbe001337b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:33:31.705 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:31.689+0000 7fbe06012700 10 monclient(hunting): picked mon.noname-f con 0x7fbe001333c0 addr v2:172.21.15.107:3300/0 2023-09-30T01:33:31.705 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:31.689+0000 7fbe06012700 10 monclient(hunting): start opening mon connection 2023-09-30T01:33:31.705 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:31.689+0000 7fbe06012700 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbe000a4ed0 con 0x7fbe001333c0 2023-09-30T01:33:31.705 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:31.689+0000 7fbe06012700 10 monclient(hunting): start opening mon connection 2023-09-30T01:33:31.705 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:31.689+0000 7fbe06012700 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe000a6330 con 0x7fbe000a4b00 2023-09-30T01:33:31.705 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:31.689+0000 7fbe06012700 10 monclient(hunting): start opening mon connection 2023-09-30T01:33:31.705 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:31.689+0000 7fbe06012700 1 -- --> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] -- mon_getmap magic: 0 v1 -- 0x7fbe00098610 con 0x7fbe000a3270 2023-09-30T01:33:31.705 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:31.689+0000 7fbe06012700 10 monclient(hunting): _renew_subs 2023-09-30T01:33:31.705 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:31.689+0000 7fbe06012700 10 monclient(hunting): authenticate will time out at 883.731137s 2023-09-30T01:33:31.705 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:31.689+0000 7fbdff7fe700 1 --2- >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7fbe000a3270 0x7fbe000a0160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:33:31.705 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:31.689+0000 7fbdfeffd700 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fbe000a4b00 0x7fbe000a06a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:33:31.706 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:31.689+0000 7fbdff7fe700 1 --2- >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7fbe000a3270 0x7fbe000a0160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3303/0 says I am v2:172.21.15.88:44446/0 (socket says 172.21.15.88:44446) 2023-09-30T01:33:31.706 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:31.689+0000 7fbdfeffd700 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fbe000a4b00 0x7fbe000a06a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3301/0 says I am v2:172.21.15.88:45028/0 (socket says 172.21.15.88:45028) 2023-09-30T01:33:31.706 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:31.689+0000 7fbdff7fe700 1 -- 172.21.15.88:0/912983270 learned_addr learned my addr 172.21.15.88:0/912983270 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-09-30T01:33:31.706 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:31.689+0000 7fbdff7fe700 10 monclient(hunting): get_auth_request con 0x7fbe000a3270 auth_method 0 2023-09-30T01:33:31.706 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:31.689+0000 7fbdff7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:33:31.706 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:31.689+0000 7fbdff7fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:33:31.706 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:31.689+0000 7fbdff7fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:33:31.706 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:31.689+0000 7fbdfeffd700 10 monclient(hunting): get_auth_request con 0x7fbe000a4b00 auth_method 0 2023-09-30T01:33:31.706 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:31.689+0000 7fbdfeffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:33:31.706 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:31.689+0000 7fbdfeffd700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:33:31.706 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:31.689+0000 7fbdfeffd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:33:31.707 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:31.689+0000 7fbdfffff700 1 --2- 172.21.15.88:0/912983270 >> v2:172.21.15.107:3300/0 conn(0x7fbe001333c0 0x7fbe001337b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:33:31.707 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:31.689+0000 7fbdfffff700 10 monclient(hunting): get_auth_request con 0x7fbe001333c0 auth_method 0 2023-09-30T01:33:31.707 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:31.689+0000 7fbdfffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:33:31.707 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:31.689+0000 7fbdfffff700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:33:31.707 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:31.689+0000 7fbdfffff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:33:31.707 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:31.689+0000 7fbdff7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:33:31.707 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:31.689+0000 7fbdff7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:33:31.707 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:31.689+0000 7fbdff7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:33:31.707 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:31.689+0000 7fbdfeffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:33:31.707 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:31.689+0000 7fbdfeffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:33:31.707 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:31.689+0000 7fbdfeffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:33:31.708 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:31.689+0000 7fbdfffff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:33:31.708 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:31.689+0000 7fbdfffff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:33:31.708 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:31.689+0000 7fbdfffff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:33:31.708 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:31.693+0000 7fbdfeffd700 10 monclient(hunting): handle_auth_done global_id 4367 payload 274 2023-09-30T01:33:31.708 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:31.693+0000 7fbdfeffd700 1 -- 172.21.15.88:0/912983270 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7fbe000a3270 msgr2=0x7fbe000a0160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:33:31.708 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:31.693+0000 7fbdfeffd700 1 --2- 172.21.15.88:0/912983270 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7fbe000a3270 0x7fbe000a0160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:31.708 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:31.693+0000 7fbdfeffd700 1 -- 172.21.15.88:0/912983270 >> v2:172.21.15.107:3300/0 conn(0x7fbe001333c0 msgr2=0x7fbe001337b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:33:31.708 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:31.693+0000 7fbdfeffd700 1 --2- 172.21.15.88:0/912983270 >> v2:172.21.15.107:3300/0 conn(0x7fbe001333c0 0x7fbe001337b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:31.708 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:31.693+0000 7fbdfeffd700 10 monclient: _finish_hunting 0 2023-09-30T01:33:31.708 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:31.693+0000 7fbdfeffd700 1 monclient: found mon.noname-b 2023-09-30T01:33:31.708 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:31.693+0000 7fbdfeffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:33:31.709 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:31.693+0000 7fbdfeffd700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.88:3301/0 2023-09-30T01:33:31.709 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:31.693+0000 7fbdfeffd700 1 -- 172.21.15.88:0/912983270 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe0005ecb0 con 0x7fbe000a4b00 2023-09-30T01:33:31.709 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:31.693+0000 7fbdff7fe700 1 --2- 172.21.15.88:0/912983270 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7fbe000a3270 0x7fbe000a0160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:33:31.709 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:31.693+0000 7fbdfeffd700 1 --2- 172.21.15.88:0/912983270 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fbe000a4b00 0x7fbe000a06a0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fbdf40098e0 tx=0x7fbdf40091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6fd1f461c2c08546 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:33:31.709 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:31.693+0000 7fbdfe7fc700 1 -- 172.21.15.88:0/912983270 <== mon.2 v2:172.21.15.88:3301/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7fbdf4011ab0 con 0x7fbe000a4b00 2023-09-30T01:33:31.709 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:31.693+0000 7fbdfe7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:33:31.709 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:31.693+0000 7fbdfe7fc700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T01:33:31.709 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:31.693+0000 7fbdfe7fc700 10 monclient: dump: 2023-09-30T01:33:31.709 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:33:31.709 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:33:31.710 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:31.710 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:31.710 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:33:31.710 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:33:31.710 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:33:31.710 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:33:31.710 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:33:31.710 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:33:31.710 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:33:31.710 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:33:31.711 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:33:31.711 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:33:31.711 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:33:31.711 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:33:31.711 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:31.693+0000 7fbdfe7fc700 10 monclient: _finish_auth 0 2023-09-30T01:33:31.711 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:31.693+0000 7fbdfe7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:33:31.711 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:31.693+0000 7fbdfe7fc700 1 -- 172.21.15.88:0/912983270 <== mon.2 v2:172.21.15.88:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbdf4011c30 con 0x7fbe000a4b00 2023-09-30T01:33:31.711 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:31.693+0000 7fbdfe7fc700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:33:31.711 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:31.693+0000 7fbdfe7fc700 1 -- 172.21.15.88:0/912983270 <== mon.2 v2:172.21.15.88:3301/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7fbdf4011da0 con 0x7fbe000a4b00 2023-09-30T01:33:31.711 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:31.693+0000 7fbdfe7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:33:31.711 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:31.693+0000 7fbdfe7fc700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T01:33:31.712 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:31.693+0000 7fbdfe7fc700 10 monclient: dump: 2023-09-30T01:33:31.712 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:33:31.712 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:33:31.712 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:31.712 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:31.712 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:33:31.712 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:33:31.712 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:33:31.712 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:33:31.712 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:33:31.712 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:33:31.713 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:33:31.713 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:33:31.713 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:33:31.713 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:33:31.713 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:33:31.713 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:33:31.713 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:31.693+0000 7fbe06012700 5 monclient: authenticate success, global_id 4367 2023-09-30T01:33:31.713 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:31.693+0000 7fbe06012700 10 monclient: get_monmap_and_config success 2023-09-30T01:33:31.713 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:31.693+0000 7fbe06012700 4 set_mon_vals no callback set 2023-09-30T01:33:31.713 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:31.693+0000 7fbe06012700 10 monclient: shutdown 2023-09-30T01:33:31.714 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:31.693+0000 7fbe06012700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:33:31.714 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:31.693+0000 7fbe06012700 1 -- 172.21.15.88:0/912983270 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fbe000a4b00 msgr2=0x7fbe000a06a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:33:31.714 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:31.693+0000 7fbe06012700 1 --2- 172.21.15.88:0/912983270 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fbe000a4b00 0x7fbe000a06a0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fbdf40098e0 tx=0x7fbdf40091a0 comp rx=0 tx=0).stop 2023-09-30T01:33:31.714 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:31.693+0000 7fbe06012700 1 -- 172.21.15.88:0/912983270 shutdown_connections 2023-09-30T01:33:31.714 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:31.693+0000 7fbe06012700 1 --2- 172.21.15.88:0/912983270 >> v2:172.21.15.107:3300/0 conn(0x7fbe001333c0 0x7fbe001337b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:31.714 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:31.693+0000 7fbe06012700 1 --2- 172.21.15.88:0/912983270 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fbe000a4b00 0x7fbe000a06a0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:31.714 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:31.693+0000 7fbe06012700 1 --2- 172.21.15.88:0/912983270 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7fbe000a3270 0x7fbe000a0160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:31.714 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:31.693+0000 7fbe06012700 1 -- 172.21.15.88:0/912983270 >> 172.21.15.88:0/912983270 conn(0x7fbe000a67f0 msgr2=0x7fbe000a6bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:33:31.714 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:31.693+0000 7fbe06012700 1 -- 172.21.15.88:0/912983270 shutdown_connections 2023-09-30T01:33:31.714 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:31.693+0000 7fbe06012700 1 -- 172.21.15.88:0/912983270 wait complete. 2023-09-30T01:33:31.715 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:31.693+0000 7fbe06012700 10 monclient: build_initial_monmap 2023-09-30T01:33:31.715 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:31.693+0000 7fbe06012700 10 monclient: monmap: 2023-09-30T01:33:31.715 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2023-09-30T01:33:31.715 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:33:31.715 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 0.000000 2023-09-30T01:33:31.715 INFO:teuthology.orchestra.run.smithi088.stderr:created 0.000000 2023-09-30T01:33:31.715 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2023-09-30T01:33:31.715 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 1 2023-09-30T01:33:31.715 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.noname-a 2023-09-30T01:33:31.715 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.noname-f 2023-09-30T01:33:31.715 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.noname-b 2023-09-30T01:33:31.716 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.noname-g 2023-09-30T01:33:31.716 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.noname-c 2023-09-30T01:33:31.716 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.noname-h 2023-09-30T01:33:31.716 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.noname-d 2023-09-30T01:33:31.716 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.noname-i 2023-09-30T01:33:31.716 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.noname-e 2023-09-30T01:33:31.716 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:33:31.716 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:31.693+0000 7fbe06012700 1 Processor -- start 2023-09-30T01:33:31.716 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:31.693+0000 7fbe06012700 1 -- start start 2023-09-30T01:33:31.716 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:31.693+0000 7fbe06012700 10 monclient: init 2023-09-30T01:33:31.716 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:31.693+0000 7fbe06012700 10 monclient: _reopen_session rank -1 2023-09-30T01:33:31.716 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:31.693+0000 7fbe06012700 10 monclient: _add_conns ranks=[7,4,1,3,2,8,0,5,6] 2023-09-30T01:33:31.717 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:31.693+0000 7fbe06012700 1 --2- >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7fbe000a3270 0x7fbe0011dc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:33:31.717 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:31.693+0000 7fbe06012700 10 monclient(hunting): picked mon.noname-i con 0x7fbe000a3270 addr [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] 2023-09-30T01:33:31.717 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:31.693+0000 7fbe06012700 1 --2- >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7fbe000a4b00 0x7fbe0011e1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:33:31.717 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:31.693+0000 7fbe06012700 10 monclient(hunting): picked mon.noname-c con 0x7fbe000a4b00 addr [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] 2023-09-30T01:33:31.717 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:31.693+0000 7fbe06012700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbe001333c0 0x7fbe0011e710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:33:31.717 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:31.693+0000 7fbe06012700 10 monclient(hunting): picked mon.noname-f con 0x7fbe001333c0 addr [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] 2023-09-30T01:33:31.717 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:31.693+0000 7fbe06012700 10 monclient(hunting): start opening mon connection 2023-09-30T01:33:31.717 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:31.693+0000 7fbe06012700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe00131120 con 0x7fbe001333c0 2023-09-30T01:33:31.717 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:31.693+0000 7fbe06012700 10 monclient(hunting): start opening mon connection 2023-09-30T01:33:31.717 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:31.693+0000 7fbe06012700 1 -- --> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] -- mon_getmap magic: 0 v1 -- 0x7fbe00097c70 con 0x7fbe000a4b00 2023-09-30T01:33:31.717 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:31.693+0000 7fbe06012700 10 monclient(hunting): start opening mon connection 2023-09-30T01:33:31.717 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:31.693+0000 7fbe06012700 1 -- --> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] -- mon_getmap magic: 0 v1 -- 0x7fbe000a0e80 con 0x7fbe000a3270 2023-09-30T01:33:31.718 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:31.693+0000 7fbe06012700 10 monclient(hunting): _renew_subs 2023-09-30T01:33:31.718 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:31.693+0000 7fbe06012700 10 monclient(hunting): authenticate will time out at 883.735770s 2023-09-30T01:33:31.718 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:31.693+0000 7fbdfeffd700 1 --2- >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7fbe000a4b00 0x7fbe0011e1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:33:31.718 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:31.697+0000 7fbdfeffd700 1 --2- >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7fbe000a4b00 0x7fbe0011e1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3302/0 says I am v2:172.21.15.88:49118/0 (socket says 172.21.15.88:49118) 2023-09-30T01:33:31.718 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:31.697+0000 7fbdfeffd700 1 -- 172.21.15.88:0/1235590094 learned_addr learned my addr 172.21.15.88:0/1235590094 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-09-30T01:33:31.718 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:31.697+0000 7fbdfeffd700 10 monclient(hunting): get_auth_request con 0x7fbe000a4b00 auth_method 0 2023-09-30T01:33:31.718 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:31.697+0000 7fbdfeffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:33:31.718 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:31.697+0000 7fbdfeffd700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:33:31.718 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:31.697+0000 7fbdfeffd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:33:31.718 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:31.697+0000 7fbdff7fe700 1 --2- 172.21.15.88:0/1235590094 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7fbe000a3270 0x7fbe0011dc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:33:31.718 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:31.697+0000 7fbdfffff700 1 --2- 172.21.15.88:0/1235590094 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbe001333c0 0x7fbe0011e710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:33:31.719 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:31.697+0000 7fbdfeffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:33:31.719 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:31.697+0000 7fbdfeffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:33:31.719 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:31.697+0000 7fbdfeffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:33:31.719 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:31.697+0000 7fbdff7fe700 10 monclient(hunting): get_auth_request con 0x7fbe000a3270 auth_method 0 2023-09-30T01:33:31.719 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:31.697+0000 7fbdff7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:33:31.719 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:31.697+0000 7fbdff7fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:33:31.719 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:31.697+0000 7fbdff7fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:33:31.719 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:31.697+0000 7fbdfffff700 10 monclient(hunting): get_auth_request con 0x7fbe001333c0 auth_method 0 2023-09-30T01:33:31.719 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:31.697+0000 7fbdfffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:33:31.719 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:31.697+0000 7fbdfffff700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:33:31.719 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:31.697+0000 7fbdfffff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:33:31.719 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:31.697+0000 7fbdfffff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:33:31.720 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:31.697+0000 7fbdfffff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:33:31.720 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:31.697+0000 7fbdfffff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:33:31.720 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:31.697+0000 7fbdff7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:33:31.720 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:31.697+0000 7fbdff7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:33:31.720 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:31.697+0000 7fbdff7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:33:31.720 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:31.697+0000 7fbdfeffd700 10 monclient(hunting): handle_auth_done global_id 4423 payload 824 2023-09-30T01:33:31.720 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:31.697+0000 7fbdfeffd700 1 -- 172.21.15.88:0/1235590094 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7fbe000a3270 msgr2=0x7fbe0011dc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:33:31.720 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:31.697+0000 7fbdfeffd700 1 --2- 172.21.15.88:0/1235590094 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7fbe000a3270 0x7fbe0011dc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:31.720 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:31.697+0000 7fbdfeffd700 1 -- 172.21.15.88:0/1235590094 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbe001333c0 msgr2=0x7fbe0011e710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:33:31.720 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:31.697+0000 7fbdfeffd700 1 --2- 172.21.15.88:0/1235590094 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbe001333c0 0x7fbe0011e710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:31.720 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:31.697+0000 7fbdfeffd700 10 monclient: _finish_hunting 0 2023-09-30T01:33:31.721 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:31.697+0000 7fbdfeffd700 1 monclient: found mon.noname-c 2023-09-30T01:33:31.721 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:31.697+0000 7fbdfeffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:33:31.721 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:31.697+0000 7fbdfeffd700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.88:3302/0 2023-09-30T01:33:31.721 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:31.697+0000 7fbdfeffd700 1 -- 172.21.15.88:0/1235590094 --> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbdf0009ae0 con 0x7fbe000a4b00 2023-09-30T01:33:31.721 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:31.697+0000 7fbdfeffd700 1 --2- 172.21.15.88:0/1235590094 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7fbe000a4b00 0x7fbe0011e1d0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fbdf4013000 tx=0x7fbdf4017000 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:33:31.721 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:31.697+0000 7fbde77fe700 1 -- 172.21.15.88:0/1235590094 <== mon.4 v2:172.21.15.88:3302/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7fbdf4006ad0 con 0x7fbe000a4b00 2023-09-30T01:33:31.721 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:31.697+0000 7fbde77fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:33:31.721 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:31.697+0000 7fbde77fe700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T01:33:31.721 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:31.697+0000 7fbde77fe700 10 monclient: dump: 2023-09-30T01:33:31.721 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:33:31.721 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:33:31.721 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:31.722 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:31.722 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:33:31.722 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:33:31.722 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:33:31.722 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:33:31.722 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:33:31.722 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:33:31.722 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:33:31.722 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:33:31.722 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:33:31.722 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:33:31.722 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:33:31.723 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:33:31.723 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:31.697+0000 7fbde77fe700 10 monclient: _finish_auth 0 2023-09-30T01:33:31.723 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:31.697+0000 7fbde77fe700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:33:31.723 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:31.697+0000 7fbde77fe700 1 -- 172.21.15.88:0/1235590094 <== mon.4 v2:172.21.15.88:3302/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbdf401a430 con 0x7fbe000a4b00 2023-09-30T01:33:31.723 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:31.697+0000 7fbe06012700 5 monclient: authenticate success, global_id 4423 2023-09-30T01:33:31.723 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:31.697+0000 7fbe06012700 10 monclient: _renew_subs 2023-09-30T01:33:31.723 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:31.697+0000 7fbe06012700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.88:3302/0 2023-09-30T01:33:31.723 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:31.697+0000 7fbe06012700 1 -- 172.21.15.88:0/1235590094 --> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbdf4009420 con 0x7fbe000a4b00 2023-09-30T01:33:31.723 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:31.697+0000 7fbde77fe700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:33:31.723 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:31.697+0000 7fbde77fe700 1 -- 172.21.15.88:0/1235590094 <== mon.4 v2:172.21.15.88:3302/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7fbdf401aa20 con 0x7fbe000a4b00 2023-09-30T01:33:31.723 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:31.697+0000 7fbde77fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:33:31.723 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:31.697+0000 7fbde77fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T01:33:31.724 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:31.697+0000 7fbde77fe700 10 monclient: dump: 2023-09-30T01:33:31.724 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:33:31.724 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:33:31.724 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:31.724 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:31.724 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:33:31.724 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:33:31.724 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:33:31.724 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:33:31.724 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:33:31.724 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:33:31.725 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:33:31.725 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:33:31.725 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:33:31.725 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:33:31.725 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:33:31.725 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:33:31.725 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:31.697+0000 7fbe06012700 10 monclient: _renew_subs 2023-09-30T01:33:31.725 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:31.697+0000 7fbe06012700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.88:3302/0 2023-09-30T01:33:31.725 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:31.697+0000 7fbdfdffb700 4 set_mon_vals no callback set 2023-09-30T01:33:31.725 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:31.697+0000 7fbe06012700 1 -- 172.21.15.88:0/1235590094 --> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe00121090 con 0x7fbe000a4b00 2023-09-30T01:33:31.725 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:31.697+0000 7fbde77fe700 1 -- 172.21.15.88:0/1235590094 <== mon.4 v2:172.21.15.88:3302/0 4 ==== mgrmap(e 5) v1 ==== 40144+0+0 (secure 0 0 0) 0x7fbdf401ab80 con 0x7fbe000a4b00 2023-09-30T01:33:31.725 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:31.697+0000 7fbde77fe700 1 --2- 172.21.15.88:0/1235590094 >> v2:172.21.15.107:6812/29926 conn(0x7fbdec033360 0x7fbdec035810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:33:31.726 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:31.697+0000 7fbe06012700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:33:31.726 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:31.697+0000 7fbe06012700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:33:31.726 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:31.697+0000 7fbe06012700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.88:3302/0 2023-09-30T01:33:31.726 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:31.697+0000 7fbde77fe700 1 -- 172.21.15.88:0/1235590094 <== mon.4 v2:172.21.15.88:3302/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 3502+0+0 (secure 0 0 0) 0x7fbdf4044eb0 con 0x7fbe000a4b00 2023-09-30T01:33:31.726 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:31.697+0000 7fbe06012700 1 -- 172.21.15.88:0/1235590094 --> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbe00071960 con 0x7fbe000a4b00 2023-09-30T01:33:31.726 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:31.697+0000 7fbdff7fe700 1 --2- 172.21.15.88:0/1235590094 >> v2:172.21.15.107:6812/29926 conn(0x7fbdec033360 0x7fbdec035810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:33:31.726 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:31.697+0000 7fbdff7fe700 10 monclient: get_auth_request con 0x7fbdec033360 auth_method 0 2023-09-30T01:33:31.726 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:31.701+0000 7fbdff7fe700 1 --2- 172.21.15.88:0/1235590094 >> v2:172.21.15.107:6812/29926 conn(0x7fbdec033360 0x7fbdec035810 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fbdf00072d0 tx=0x7fbdf000b040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:33:31.727 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:31.705+0000 7fbde77fe700 1 -- 172.21.15.88:0/1235590094 <== mon.4 v2:172.21.15.88:3302/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fbdf4015070 con 0x7fbe000a4b00 2023-09-30T01:33:31.727 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:31.705+0000 7fbde77fe700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:33:31.727 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:31.705+0000 7fbde77fe700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:33:31.899 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:31.897+0000 7fbe06012700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T01:33:31.899 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:31.897+0000 7fbe06012700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:33:31.899 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:31.897+0000 7fbe06012700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.88:3302/0 2023-09-30T01:33:31.899 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:31.897+0000 7fbe06012700 1 -- 172.21.15.88:0/1235590094 --> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fbe00064b80 con 0x7fbe000a4b00 2023-09-30T01:33:31.901 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:31.897+0000 7fbde77fe700 1 -- 172.21.15.88:0/1235590094 <== mon.4 v2:172.21.15.88:3302/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7fbdf401a5a0 con 0x7fbe000a4b00 2023-09-30T01:33:31.901 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:31.897+0000 7fbde77fe700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:33:31.901 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:31.897+0000 7fbde77fe700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:33:31.901 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-09-30T01:33:31.901 INFO:teuthology.orchestra.run.smithi088.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:33:31.904 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:31.901+0000 7fbe06012700 1 -- 172.21.15.88:0/1235590094 >> v2:172.21.15.107:6812/29926 conn(0x7fbdec033360 msgr2=0x7fbdec035810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:33:31.904 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:31.901+0000 7fbe06012700 1 --2- 172.21.15.88:0/1235590094 >> v2:172.21.15.107:6812/29926 conn(0x7fbdec033360 0x7fbdec035810 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fbdf00072d0 tx=0x7fbdf000b040 comp rx=0 tx=0).stop 2023-09-30T01:33:31.904 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:31.901+0000 7fbe06012700 10 monclient: shutdown 2023-09-30T01:33:31.904 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:31.901+0000 7fbe06012700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:33:31.904 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:31.901+0000 7fbe06012700 1 -- 172.21.15.88:0/1235590094 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7fbe000a4b00 msgr2=0x7fbe0011e1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:33:31.904 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:31.901+0000 7fbe06012700 1 --2- 172.21.15.88:0/1235590094 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7fbe000a4b00 0x7fbe0011e1d0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fbdf4013000 tx=0x7fbdf4017000 comp rx=0 tx=0).stop 2023-09-30T01:33:31.904 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:31.901+0000 7fbe06012700 1 -- 172.21.15.88:0/1235590094 shutdown_connections 2023-09-30T01:33:31.904 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:31.901+0000 7fbe06012700 1 --2- 172.21.15.88:0/1235590094 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbe001333c0 0x7fbe0011e710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:31.904 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:31.901+0000 7fbe06012700 1 --2- 172.21.15.88:0/1235590094 >> v2:172.21.15.107:6812/29926 conn(0x7fbdec033360 0x7fbdec035810 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:31.905 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:31.901+0000 7fbe06012700 1 --2- 172.21.15.88:0/1235590094 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7fbe000a4b00 0x7fbe0011e1d0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:31.905 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:31.901+0000 7fbe06012700 1 --2- 172.21.15.88:0/1235590094 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7fbe000a3270 0x7fbe0011dc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:31.905 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:31.901+0000 7fbe06012700 1 -- 172.21.15.88:0/1235590094 >> 172.21.15.88:0/1235590094 conn(0x7fbe000a67f0 msgr2=0x7fbe000a20e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:33:31.905 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:31.901+0000 7fbe06012700 1 -- 172.21.15.88:0/1235590094 shutdown_connections 2023-09-30T01:33:31.905 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:31.901+0000 7fbe06012700 1 -- 172.21.15.88:0/1235590094 wait complete. 2023-09-30T01:33:34.921 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:33:35.032 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:35.029+0000 7f86d759e700 10 monclient: get_monmap_and_config 2023-09-30T01:33:35.032 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:35.029+0000 7f86d759e700 10 monclient: build_initial_monmap 2023-09-30T01:33:35.032 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:35.029+0000 7f86d759e700 10 monclient: monmap: 2023-09-30T01:33:35.032 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2023-09-30T01:33:35.032 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:33:35.032 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:33:35.032468+0000 2023-09-30T01:33:35.032 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:33:35.032468+0000 2023-09-30T01:33:35.032 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2023-09-30T01:33:35.033 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:33:35.033 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a-legacy 2023-09-30T01:33:35.033 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.107:6789/0 mon.noname-f-legacy 2023-09-30T01:33:35.033 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.noname-b 2023-09-30T01:33:35.033 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.noname-g 2023-09-30T01:33:35.033 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.noname-c 2023-09-30T01:33:35.033 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.noname-h 2023-09-30T01:33:35.033 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.noname-d 2023-09-30T01:33:35.033 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.noname-i 2023-09-30T01:33:35.033 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.noname-e 2023-09-30T01:33:35.033 INFO:teuthology.orchestra.run.smithi088.stderr:9: v2:172.21.15.88:3300/0 mon.noname-a 2023-09-30T01:33:35.034 INFO:teuthology.orchestra.run.smithi088.stderr:10: v2:172.21.15.107:3300/0 mon.noname-f 2023-09-30T01:33:35.034 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:33:35.044 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:35.029+0000 7f86d759e700 1 Processor -- start 2023-09-30T01:33:35.044 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:35.029+0000 7f86d759e700 1 -- start start 2023-09-30T01:33:35.044 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:35.029+0000 7f86d759e700 10 monclient: init 2023-09-30T01:33:35.044 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:35.029+0000 7f86d759e700 10 monclient: _reopen_session rank -1 2023-09-30T01:33:35.044 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:35.029+0000 7f86d759e700 10 monclient: _add_conns ranks=[0,10,2,4,3,1,9,5,8,6,7] 2023-09-30T01:33:35.044 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:35.029+0000 7f86d759e700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f86d80a3380 addr v1:172.21.15.88:6789/0 2023-09-30T01:33:35.044 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:35.029+0000 7f86d759e700 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f86d809ffd0 0x7f86d80a03a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:33:35.044 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:35.029+0000 7f86d759e700 10 monclient(hunting): picked mon.noname-f con 0x7f86d809ffd0 addr v2:172.21.15.107:3300/0 2023-09-30T01:33:35.045 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:35.029+0000 7f86d759e700 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f86d81386e0 0x7f86d813aac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:33:35.045 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:35.029+0000 7f86d759e700 10 monclient(hunting): picked mon.noname-b con 0x7f86d81386e0 addr [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] 2023-09-30T01:33:35.045 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:35.029+0000 7f86d759e700 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86d804bb20 con 0x7f86d80a3380 2023-09-30T01:33:35.045 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:35.029+0000 7f86d759e700 10 monclient(hunting): start opening mon connection 2023-09-30T01:33:35.045 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:35.029+0000 7f86d759e700 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f86d8097ab0 con 0x7f86d809ffd0 2023-09-30T01:33:35.045 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:35.029+0000 7f86d759e700 10 monclient(hunting): start opening mon connection 2023-09-30T01:33:35.045 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:35.029+0000 7f86d759e700 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86d80a08e0 con 0x7f86d81386e0 2023-09-30T01:33:35.045 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:35.029+0000 7f86d759e700 10 monclient(hunting): _renew_subs 2023-09-30T01:33:35.045 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:35.029+0000 7f86d759e700 10 monclient(hunting): authenticate will time out at 887.069924s 2023-09-30T01:33:35.045 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:35.029+0000 7f86d6d9d700 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f86d81386e0 0x7f86d813aac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:33:35.046 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:35.029+0000 7f86d6d9d700 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f86d81386e0 0x7f86d813aac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3301/0 says I am v2:172.21.15.88:45044/0 (socket says 172.21.15.88:45044) 2023-09-30T01:33:35.046 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:35.029+0000 7f86d6d9d700 1 -- 172.21.15.88:0/1670904275 learned_addr learned my addr 172.21.15.88:0/1670904275 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-09-30T01:33:35.046 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:35.029+0000 7f86d6d9d700 10 monclient(hunting): get_auth_request con 0x7f86d81386e0 auth_method 0 2023-09-30T01:33:35.046 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:35.029+0000 7f86d6d9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:33:35.046 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:35.029+0000 7f86d6d9d700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:33:35.046 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:35.029+0000 7f86d6d9d700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:33:35.046 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:35.029+0000 7f86d6d9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:33:35.046 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:35.029+0000 7f86d6d9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:33:35.046 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:35.029+0000 7f86d5d9b700 1 --2- 172.21.15.88:0/1670904275 >> v2:172.21.15.107:3300/0 conn(0x7f86d809ffd0 0x7f86d80a03a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:33:35.046 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:35.029+0000 7f86d6d9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:33:35.046 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:35.029+0000 7f86d5d9b700 10 monclient(hunting): get_auth_request con 0x7f86d809ffd0 auth_method 0 2023-09-30T01:33:35.047 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:35.029+0000 7f86d5d9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:33:35.047 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:35.029+0000 7f86d5d9b700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:33:35.047 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:35.029+0000 7f86d5d9b700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:33:35.047 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:35.029+0000 7f86d5d9b700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:33:35.047 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:35.029+0000 7f86d5d9b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:33:35.047 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:35.029+0000 7f86d5d9b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:33:35.047 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:35.029+0000 7f86d559a700 1 -- 172.21.15.88:0/1670904275 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2246206822 0 0) 0x7f86d804bb20 con 0x7f86d80a3380 2023-09-30T01:33:35.047 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:35.029+0000 7f86d559a700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:33:35.047 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:35.029+0000 7f86d559a700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:33:35.047 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:35.029+0000 7f86d559a700 10 monclient(hunting): my global_id is 4491 2023-09-30T01:33:35.047 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:35.029+0000 7f86d559a700 1 -- 172.21.15.88:0/1670904275 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f86c0003570 con 0x7f86d80a3380 2023-09-30T01:33:35.047 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:35.029+0000 7f86d6d9d700 10 monclient(hunting): handle_auth_done global_id 4376 payload 274 2023-09-30T01:33:35.048 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:35.029+0000 7f86d6d9d700 1 -- 172.21.15.88:0/1670904275 >> v2:172.21.15.107:3300/0 conn(0x7f86d809ffd0 msgr2=0x7f86d80a03a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:33:35.048 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:35.029+0000 7f86d6d9d700 1 --2- 172.21.15.88:0/1670904275 >> v2:172.21.15.107:3300/0 conn(0x7f86d809ffd0 0x7f86d80a03a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:35.048 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:35.029+0000 7f86d6d9d700 1 -- 172.21.15.88:0/1670904275 >> v1:172.21.15.88:6789/0 conn(0x7f86d80a3380 legacy=0x7f86d80a48b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:33:35.048 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:35.029+0000 7f86d6d9d700 10 monclient: _finish_hunting 0 2023-09-30T01:33:35.048 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:35.029+0000 7f86d6d9d700 1 monclient: found mon.noname-b 2023-09-30T01:33:35.048 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:35.029+0000 7f86d6d9d700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:33:35.048 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:35.029+0000 7f86d6d9d700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.88:3301/0 2023-09-30T01:33:35.048 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:35.029+0000 7f86d6d9d700 1 -- 172.21.15.88:0/1670904275 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86d8098480 con 0x7f86d81386e0 2023-09-30T01:33:35.049 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:35.029+0000 7f86d6d9d700 1 --2- 172.21.15.88:0/1670904275 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f86d81386e0 0x7f86d813aac0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f86d0009f20 tx=0x7f86d0009840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=302536fc75e5cf2c server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:33:35.049 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:35.029+0000 7f86d559a700 1 -- 172.21.15.88:0/1670904275 <== mon.2 v2:172.21.15.88:3301/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f86d000e070 con 0x7f86d81386e0 2023-09-30T01:33:35.049 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:35.029+0000 7f86d559a700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:33:35.049 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:35.029+0000 7f86d559a700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T01:33:35.049 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:35.029+0000 7f86d559a700 10 monclient: dump: 2023-09-30T01:33:35.049 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:33:35.049 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:33:35.049 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:35.049 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:35.049 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:33:35.049 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:33:35.049 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:33:35.050 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:33:35.050 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:33:35.050 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:33:35.050 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:33:35.050 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:33:35.050 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:33:35.050 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:33:35.050 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:33:35.050 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:33:35.050 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:35.029+0000 7f86d559a700 10 monclient: _finish_auth 0 2023-09-30T01:33:35.050 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:35.029+0000 7f86d559a700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:33:35.051 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:35.029+0000 7f86d759e700 5 monclient: authenticate success, global_id 4376 2023-09-30T01:33:35.051 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:35.029+0000 7f86d759e700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:33:35.051 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:35.029+0000 7f86d559a700 1 -- 172.21.15.88:0/1670904275 <== mon.2 v2:172.21.15.88:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f86d000bdf0 con 0x7f86d81386e0 2023-09-30T01:33:35.051 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:35.029+0000 7f86d559a700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:33:35.051 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:35.029+0000 7f86d759e700 10 monclient: get_monmap_and_config success 2023-09-30T01:33:35.051 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:35.029+0000 7f86d759e700 4 set_mon_vals no callback set 2023-09-30T01:33:35.051 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:35.029+0000 7f86d559a700 1 -- 172.21.15.88:0/1670904275 <== mon.2 v2:172.21.15.88:3301/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f86d00092e0 con 0x7f86d81386e0 2023-09-30T01:33:35.051 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:35.029+0000 7f86d559a700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:33:35.051 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:35.029+0000 7f86d559a700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T01:33:35.051 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:35.029+0000 7f86d559a700 10 monclient: dump: 2023-09-30T01:33:35.051 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:33:35.052 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:33:35.052 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:35.052 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:35.052 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:33:35.052 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:33:35.052 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:33:35.052 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:33:35.052 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:33:35.052 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:33:35.052 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:33:35.052 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:33:35.053 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:33:35.053 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:33:35.053 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:33:35.053 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:33:35.053 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:35.033+0000 7f86d759e700 10 monclient: shutdown 2023-09-30T01:33:35.053 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:35.033+0000 7f86d759e700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:33:35.053 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:35.033+0000 7f86d759e700 1 -- 172.21.15.88:0/1670904275 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f86d81386e0 msgr2=0x7f86d813aac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:33:35.053 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:35.033+0000 7f86d759e700 1 --2- 172.21.15.88:0/1670904275 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f86d81386e0 0x7f86d813aac0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f86d0009f20 tx=0x7f86d0009840 comp rx=0 tx=0).stop 2023-09-30T01:33:35.053 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:35.033+0000 7f86d759e700 1 -- 172.21.15.88:0/1670904275 shutdown_connections 2023-09-30T01:33:35.053 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:35.033+0000 7f86d759e700 1 --2- 172.21.15.88:0/1670904275 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f86d81386e0 0x7f86d813aac0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:35.053 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:35.033+0000 7f86d759e700 1 --2- 172.21.15.88:0/1670904275 >> v2:172.21.15.107:3300/0 conn(0x7f86d809ffd0 0x7f86d80a03a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:35.053 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:35.033+0000 7f86d759e700 1 -- 172.21.15.88:0/1670904275 >> 172.21.15.88:0/1670904275 conn(0x7f86d806e5e0 msgr2=0x7f86d806e9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:33:35.054 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:35.033+0000 7f86d759e700 1 -- 172.21.15.88:0/1670904275 shutdown_connections 2023-09-30T01:33:35.054 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:35.033+0000 7f86d759e700 1 -- 172.21.15.88:0/1670904275 wait complete. 2023-09-30T01:33:35.054 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:35.033+0000 7f86d759e700 10 monclient: build_initial_monmap 2023-09-30T01:33:35.054 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:35.033+0000 7f86d759e700 10 monclient: monmap: 2023-09-30T01:33:35.054 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2023-09-30T01:33:35.054 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:33:35.054 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 0.000000 2023-09-30T01:33:35.054 INFO:teuthology.orchestra.run.smithi088.stderr:created 0.000000 2023-09-30T01:33:35.054 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2023-09-30T01:33:35.054 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 1 2023-09-30T01:33:35.054 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.noname-a 2023-09-30T01:33:35.055 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.noname-f 2023-09-30T01:33:35.055 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.noname-b 2023-09-30T01:33:35.055 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.noname-g 2023-09-30T01:33:35.055 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.noname-c 2023-09-30T01:33:35.055 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.noname-h 2023-09-30T01:33:35.055 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.noname-d 2023-09-30T01:33:35.055 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.noname-i 2023-09-30T01:33:35.055 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.noname-e 2023-09-30T01:33:35.055 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:33:35.055 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:35.033+0000 7f86d759e700 1 Processor -- start 2023-09-30T01:33:35.055 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:35.033+0000 7f86d759e700 1 -- start start 2023-09-30T01:33:35.056 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:35.033+0000 7f86d759e700 10 monclient: init 2023-09-30T01:33:35.056 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:35.037+0000 7f86d759e700 10 monclient: _reopen_session rank -1 2023-09-30T01:33:35.056 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:35.037+0000 7f86d759e700 10 monclient: _add_conns ranks=[6,1,5,7,3,0,2,4,8] 2023-09-30T01:33:35.056 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:35.037+0000 7f86d759e700 1 --2- >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7f86d809ffd0 0x7f86d809df90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:33:35.056 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:35.037+0000 7f86d759e700 10 monclient(hunting): picked mon.noname-d con 0x7f86d809ffd0 addr [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] 2023-09-30T01:33:35.056 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:35.037+0000 7f86d759e700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f86d80a3380 0x7f86d81378b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:33:35.056 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:35.037+0000 7f86d759e700 10 monclient(hunting): picked mon.noname-f con 0x7f86d80a3380 addr [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] 2023-09-30T01:33:35.056 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:35.037+0000 7f86d759e700 1 --2- >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7f86d81386e0 0x7f86d8141f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:33:35.056 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:35.037+0000 7f86d759e700 10 monclient(hunting): picked mon.noname-h con 0x7f86d81386e0 addr [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] 2023-09-30T01:33:35.056 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:35.037+0000 7f86d759e700 10 monclient(hunting): start opening mon connection 2023-09-30T01:33:35.056 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:35.037+0000 7f86d759e700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86d8131da0 con 0x7f86d80a3380 2023-09-30T01:33:35.056 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:35.037+0000 7f86d759e700 10 monclient(hunting): start opening mon connection 2023-09-30T01:33:35.057 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:35.037+0000 7f86d759e700 1 -- --> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] -- mon_getmap magic: 0 v1 -- 0x7f86d8130c60 con 0x7f86d81386e0 2023-09-30T01:33:35.057 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:35.037+0000 7f86d759e700 10 monclient(hunting): start opening mon connection 2023-09-30T01:33:35.057 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:35.037+0000 7f86d759e700 1 -- --> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] -- mon_getmap magic: 0 v1 -- 0x7f86d8131ac0 con 0x7f86d809ffd0 2023-09-30T01:33:35.057 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:35.037+0000 7f86d759e700 10 monclient(hunting): _renew_subs 2023-09-30T01:33:35.057 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:35.037+0000 7f86d759e700 10 monclient(hunting): authenticate will time out at 887.076380s 2023-09-30T01:33:35.057 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:35.037+0000 7f86d659c700 1 --2- >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7f86d809ffd0 0x7f86d809df90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:33:35.057 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:35.037+0000 7f86d659c700 1 --2- >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7f86d809ffd0 0x7f86d809df90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3303/0 says I am v2:172.21.15.88:44462/0 (socket says 172.21.15.88:44462) 2023-09-30T01:33:35.057 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:35.037+0000 7f86d659c700 1 -- 172.21.15.88:0/1148477412 learned_addr learned my addr 172.21.15.88:0/1148477412 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-09-30T01:33:35.057 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:35.037+0000 7f86d659c700 10 monclient(hunting): get_auth_request con 0x7f86d809ffd0 auth_method 0 2023-09-30T01:33:35.057 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:35.037+0000 7f86d659c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:33:35.057 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:35.037+0000 7f86d659c700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:33:35.058 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:35.037+0000 7f86d659c700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:33:35.058 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:35.037+0000 7f86d659c700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:33:35.058 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:35.037+0000 7f86d659c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:33:35.058 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:35.037+0000 7f86d659c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:33:35.058 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:35.037+0000 7f86d5d9b700 1 --2- 172.21.15.88:0/1148477412 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f86d80a3380 0x7f86d81378b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:33:35.058 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:35.037+0000 7f86d6d9d700 1 --2- 172.21.15.88:0/1148477412 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7f86d81386e0 0x7f86d8141f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:33:35.058 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:35.037+0000 7f86d5d9b700 10 monclient(hunting): get_auth_request con 0x7f86d80a3380 auth_method 0 2023-09-30T01:33:35.058 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:35.037+0000 7f86d5d9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:33:35.058 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:35.037+0000 7f86d5d9b700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:33:35.058 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:35.037+0000 7f86d5d9b700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:33:35.058 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:35.037+0000 7f86d6d9d700 10 monclient(hunting): get_auth_request con 0x7f86d81386e0 auth_method 0 2023-09-30T01:33:35.058 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:35.037+0000 7f86d6d9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:33:35.059 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:35.037+0000 7f86d6d9d700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:33:35.059 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:35.037+0000 7f86d6d9d700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:33:35.059 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:35.037+0000 7f86d5d9b700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:33:35.059 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:35.037+0000 7f86d5d9b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:33:35.059 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:35.037+0000 7f86d5d9b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:33:35.059 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:35.037+0000 7f86d6d9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:33:35.059 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:35.037+0000 7f86d6d9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:33:35.059 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:35.037+0000 7f86d6d9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:33:35.059 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:35.037+0000 7f86d659c700 10 monclient(hunting): handle_auth_done global_id 4353 payload 824 2023-09-30T01:33:35.059 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:35.037+0000 7f86d659c700 1 -- 172.21.15.88:0/1148477412 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7f86d81386e0 msgr2=0x7f86d8141f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:33:35.059 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:35.037+0000 7f86d659c700 1 --2- 172.21.15.88:0/1148477412 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7f86d81386e0 0x7f86d8141f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:35.059 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:35.037+0000 7f86d659c700 1 -- 172.21.15.88:0/1148477412 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f86d80a3380 msgr2=0x7f86d81378b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:33:35.060 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:35.037+0000 7f86d659c700 1 --2- 172.21.15.88:0/1148477412 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f86d80a3380 0x7f86d81378b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:35.060 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:35.037+0000 7f86d659c700 10 monclient: _finish_hunting 0 2023-09-30T01:33:35.060 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:35.037+0000 7f86d659c700 1 monclient: found mon.noname-d 2023-09-30T01:33:35.060 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:35.037+0000 7f86d659c700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:33:35.060 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:35.037+0000 7f86d659c700 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.88:3303/0 2023-09-30T01:33:35.060 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:35.037+0000 7f86d659c700 1 -- 172.21.15.88:0/1148477412 --> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86d0009ae0 con 0x7f86d809ffd0 2023-09-30T01:33:35.060 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:35.037+0000 7f86d659c700 1 --2- 172.21.15.88:0/1148477412 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7f86d809ffd0 0x7f86d809df90 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f86cc00c7c0 tx=0x7f86cc007dc0 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:33:35.060 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:35.037+0000 7f86c67fc700 1 -- 172.21.15.88:0/1148477412 <== mon.6 v2:172.21.15.88:3303/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f86cc002ea0 con 0x7f86d809ffd0 2023-09-30T01:33:35.060 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:35.037+0000 7f86c67fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:33:35.060 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:35.037+0000 7f86c67fc700 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2023-09-30T01:33:35.060 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:35.037+0000 7f86c67fc700 10 monclient: dump: 2023-09-30T01:33:35.060 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:33:35.061 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:33:35.061 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:35.061 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:35.061 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:33:35.061 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:33:35.061 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:33:35.061 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:33:35.061 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:33:35.061 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:33:35.061 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:33:35.061 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:33:35.061 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:33:35.061 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:33:35.062 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:33:35.062 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:33:35.062 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:35.037+0000 7f86c67fc700 10 monclient: _finish_auth 0 2023-09-30T01:33:35.062 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:35.037+0000 7f86c67fc700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:33:35.062 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:35.037+0000 7f86d759e700 5 monclient: authenticate success, global_id 4353 2023-09-30T01:33:35.062 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:35.037+0000 7f86d759e700 10 monclient: _renew_subs 2023-09-30T01:33:35.062 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:35.037+0000 7f86c67fc700 1 -- 172.21.15.88:0/1148477412 <== mon.6 v2:172.21.15.88:3303/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f86cc010990 con 0x7f86d809ffd0 2023-09-30T01:33:35.062 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:35.037+0000 7f86d759e700 10 monclient: _send_mon_message to mon.d at v2:172.21.15.88:3303/0 2023-09-30T01:33:35.062 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:35.037+0000 7f86d759e700 1 -- 172.21.15.88:0/1148477412 --> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f86d8132080 con 0x7f86d809ffd0 2023-09-30T01:33:35.062 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:35.037+0000 7f86c67fc700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:33:35.062 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:35.037+0000 7f86c67fc700 1 -- 172.21.15.88:0/1148477412 <== mon.6 v2:172.21.15.88:3303/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f86cc0113a0 con 0x7f86d809ffd0 2023-09-30T01:33:35.062 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:35.037+0000 7f86c67fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:33:35.063 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:35.037+0000 7f86d4d99700 4 set_mon_vals no callback set 2023-09-30T01:33:35.063 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:35.037+0000 7f86c67fc700 10 monclient: got monmap 1 from mon.d (according to old e1) 2023-09-30T01:33:35.063 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:35.037+0000 7f86c67fc700 10 monclient: dump: 2023-09-30T01:33:35.063 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:33:35.063 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:33:35.063 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:35.063 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:35.063 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:33:35.063 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:33:35.063 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:33:35.063 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:33:35.063 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:33:35.064 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:33:35.064 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:33:35.064 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:33:35.064 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:33:35.064 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:33:35.064 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:33:35.064 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:33:35.064 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:35.037+0000 7f86d759e700 10 monclient: _renew_subs 2023-09-30T01:33:35.064 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:35.037+0000 7f86d759e700 10 monclient: _send_mon_message to mon.d at v2:172.21.15.88:3303/0 2023-09-30T01:33:35.064 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:35.037+0000 7f86d759e700 1 -- 172.21.15.88:0/1148477412 --> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f86d809ead0 con 0x7f86d809ffd0 2023-09-30T01:33:35.064 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:35.037+0000 7f86d759e700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:33:35.064 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:35.037+0000 7f86c67fc700 1 -- 172.21.15.88:0/1148477412 <== mon.6 v2:172.21.15.88:3303/0 4 ==== mgrmap(e 5) v1 ==== 40144+0+0 (secure 0 0 0) 0x7f86cc010510 con 0x7f86d809ffd0 2023-09-30T01:33:35.064 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:35.037+0000 7f86d759e700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:33:35.065 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:35.037+0000 7f86d759e700 10 monclient: _send_mon_message to mon.d at v2:172.21.15.88:3303/0 2023-09-30T01:33:35.065 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:35.037+0000 7f86d759e700 1 -- 172.21.15.88:0/1148477412 --> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f86d80a61b0 con 0x7f86d809ffd0 2023-09-30T01:33:35.065 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:35.037+0000 7f86c67fc700 1 --2- 172.21.15.88:0/1148477412 >> v2:172.21.15.107:6812/29926 conn(0x7f86b80332c0 0x7f86b8035770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:33:35.065 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:35.037+0000 7f86c67fc700 1 -- 172.21.15.88:0/1148477412 <== mon.6 v2:172.21.15.88:3303/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 3502+0+0 (secure 0 0 0) 0x7f86cc042600 con 0x7f86d809ffd0 2023-09-30T01:33:35.065 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:35.041+0000 7f86d5d9b700 1 --2- 172.21.15.88:0/1148477412 >> v2:172.21.15.107:6812/29926 conn(0x7f86b80332c0 0x7f86b8035770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:33:35.065 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:35.041+0000 7f86d5d9b700 10 monclient: get_auth_request con 0x7f86b80332c0 auth_method 0 2023-09-30T01:33:35.065 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:35.041+0000 7f86d5d9b700 1 --2- 172.21.15.88:0/1148477412 >> v2:172.21.15.107:6812/29926 conn(0x7f86b80332c0 0x7f86b8035770 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f86d809ea10 tx=0x7f86c80091a0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:33:35.066 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:35.045+0000 7f86c67fc700 1 -- 172.21.15.88:0/1148477412 <== mon.6 v2:172.21.15.88:3303/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f86cc017620 con 0x7f86d809ffd0 2023-09-30T01:33:35.066 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:35.045+0000 7f86c67fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:33:35.066 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:35.045+0000 7f86c67fc700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:33:35.249 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:35.245+0000 7f86d759e700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T01:33:35.249 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:35.245+0000 7f86d759e700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:33:35.249 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:35.245+0000 7f86d759e700 10 monclient: _send_mon_message to mon.d at v2:172.21.15.88:3303/0 2023-09-30T01:33:35.249 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:35.245+0000 7f86d759e700 1 -- 172.21.15.88:0/1148477412 --> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f86d80710d0 con 0x7f86d809ffd0 2023-09-30T01:33:35.250 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:35.245+0000 7f86c67fc700 1 -- 172.21.15.88:0/1148477412 <== mon.6 v2:172.21.15.88:3303/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f86cc017030 con 0x7f86d809ffd0 2023-09-30T01:33:35.250 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:35.245+0000 7f86c67fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:33:35.250 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:35.245+0000 7f86c67fc700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:33:35.250 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-09-30T01:33:35.250 INFO:teuthology.orchestra.run.smithi088.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:33:35.252 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:35.249+0000 7f86d759e700 1 -- 172.21.15.88:0/1148477412 >> v2:172.21.15.107:6812/29926 conn(0x7f86b80332c0 msgr2=0x7f86b8035770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:33:35.252 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:35.249+0000 7f86d759e700 1 --2- 172.21.15.88:0/1148477412 >> v2:172.21.15.107:6812/29926 conn(0x7f86b80332c0 0x7f86b8035770 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f86d809ea10 tx=0x7f86c80091a0 comp rx=0 tx=0).stop 2023-09-30T01:33:35.252 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:35.249+0000 7f86d759e700 10 monclient: shutdown 2023-09-30T01:33:35.252 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:35.249+0000 7f86d759e700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:33:35.252 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:35.249+0000 7f86d759e700 1 -- 172.21.15.88:0/1148477412 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7f86d809ffd0 msgr2=0x7f86d809df90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:33:35.252 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:35.249+0000 7f86d759e700 1 --2- 172.21.15.88:0/1148477412 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7f86d809ffd0 0x7f86d809df90 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f86cc00c7c0 tx=0x7f86cc007dc0 comp rx=0 tx=0).stop 2023-09-30T01:33:35.252 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:35.249+0000 7f86d759e700 1 -- 172.21.15.88:0/1148477412 shutdown_connections 2023-09-30T01:33:35.252 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:35.249+0000 7f86d759e700 1 --2- 172.21.15.88:0/1148477412 >> v2:172.21.15.107:6812/29926 conn(0x7f86b80332c0 0x7f86b8035770 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:35.253 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:35.249+0000 7f86d759e700 1 --2- 172.21.15.88:0/1148477412 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f86d80a3380 0x7f86d81378b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:35.253 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:35.249+0000 7f86d759e700 1 --2- 172.21.15.88:0/1148477412 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7f86d81386e0 0x7f86d8141f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:35.253 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:35.249+0000 7f86d759e700 1 --2- 172.21.15.88:0/1148477412 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7f86d809ffd0 0x7f86d809df90 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:35.253 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:35.249+0000 7f86d759e700 1 -- 172.21.15.88:0/1148477412 >> 172.21.15.88:0/1148477412 conn(0x7f86d806e5e0 msgr2=0x7f86d812e4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:33:35.253 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:35.249+0000 7f86d759e700 1 -- 172.21.15.88:0/1148477412 shutdown_connections 2023-09-30T01:33:35.253 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:35.249+0000 7f86d759e700 1 -- 172.21.15.88:0/1148477412 wait complete. 2023-09-30T01:33:38.265 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:33:38.377 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:38.373+0000 7fbeb608b700 10 monclient: get_monmap_and_config 2023-09-30T01:33:38.377 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:38.373+0000 7fbeb608b700 10 monclient: build_initial_monmap 2023-09-30T01:33:38.377 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:38.373+0000 7fbeb608b700 10 monclient: monmap: 2023-09-30T01:33:38.377 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2023-09-30T01:33:38.377 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:33:38.378 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:33:38.377814+0000 2023-09-30T01:33:38.378 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:33:38.377814+0000 2023-09-30T01:33:38.378 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2023-09-30T01:33:38.378 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:33:38.378 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a-legacy 2023-09-30T01:33:38.378 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.107:6789/0 mon.noname-f-legacy 2023-09-30T01:33:38.378 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.noname-b 2023-09-30T01:33:38.378 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.noname-g 2023-09-30T01:33:38.378 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.noname-c 2023-09-30T01:33:38.378 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.noname-h 2023-09-30T01:33:38.378 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.noname-d 2023-09-30T01:33:38.378 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.noname-i 2023-09-30T01:33:38.379 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.noname-e 2023-09-30T01:33:38.379 INFO:teuthology.orchestra.run.smithi088.stderr:9: v2:172.21.15.88:3300/0 mon.noname-a 2023-09-30T01:33:38.379 INFO:teuthology.orchestra.run.smithi088.stderr:10: v2:172.21.15.107:3300/0 mon.noname-f 2023-09-30T01:33:38.379 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:33:38.382 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:38.373+0000 7fbeb608b700 1 Processor -- start 2023-09-30T01:33:38.382 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:38.373+0000 7fbeb608b700 1 -- start start 2023-09-30T01:33:38.382 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:38.373+0000 7fbeb608b700 10 monclient: init 2023-09-30T01:33:38.382 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:38.373+0000 7fbeb608b700 10 monclient: _reopen_session rank -1 2023-09-30T01:33:38.383 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:38.373+0000 7fbeb608b700 10 monclient: _add_conns ranks=[2,0,5,8,10,3,9,1,7,6,4] 2023-09-30T01:33:38.383 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:38.373+0000 7fbeb608b700 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fbeb0097a40 0x7fbeb0132700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:33:38.383 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:38.373+0000 7fbeb608b700 10 monclient(hunting): picked mon.noname-b con 0x7fbeb0097a40 addr [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] 2023-09-30T01:33:38.383 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:38.373+0000 7fbeb608b700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fbeb01311c0 addr v1:172.21.15.88:6789/0 2023-09-30T01:33:38.383 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:38.373+0000 7fbeb608b700 1 --2- >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7fbeb0132c40 0x7fbeb01339f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:33:38.383 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:38.373+0000 7fbeb608b700 10 monclient(hunting): picked mon.noname-h con 0x7fbeb0132c40 addr [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] 2023-09-30T01:33:38.383 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:38.373+0000 7fbeb608b700 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbeb004bb20 con 0x7fbeb01311c0 2023-09-30T01:33:38.383 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:38.373+0000 7fbeb608b700 10 monclient(hunting): start opening mon connection 2023-09-30T01:33:38.383 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:38.373+0000 7fbeb608b700 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbeb0131590 con 0x7fbeb0097a40 2023-09-30T01:33:38.383 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:38.373+0000 7fbeb608b700 10 monclient(hunting): start opening mon connection 2023-09-30T01:33:38.383 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:38.373+0000 7fbeb608b700 1 -- --> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] -- mon_getmap magic: 0 v1 -- 0x7fbeb009a5d0 con 0x7fbeb0132c40 2023-09-30T01:33:38.384 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:38.373+0000 7fbeb608b700 10 monclient(hunting): _renew_subs 2023-09-30T01:33:38.384 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:38.373+0000 7fbeb608b700 10 monclient(hunting): authenticate will time out at 890.415280s 2023-09-30T01:33:38.384 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:38.373+0000 7fbeb4888700 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7fbeb01311c0 0x7fbeb01332e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:43746/0 (socket says 172.21.15.88:43746) 2023-09-30T01:33:38.384 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:38.373+0000 7fbeb4888700 1 -- 172.21.15.88:0/1750249833 learned_addr learned my addr 172.21.15.88:0/1750249833 (peer_addr_for_me v1:172.21.15.88:0/0) 2023-09-30T01:33:38.384 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:38.373+0000 7fbeb5089700 1 --2- 172.21.15.88:0/1750249833 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fbeb0097a40 0x7fbeb0132700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:33:38.384 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:38.373+0000 7fbeb5089700 10 monclient(hunting): get_auth_request con 0x7fbeb0097a40 auth_method 0 2023-09-30T01:33:38.384 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:38.373+0000 7fbeb5089700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:33:38.384 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:38.373+0000 7fbeb5089700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:33:38.384 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:38.373+0000 7fbeb5089700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:33:38.384 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:38.373+0000 7fbeb588a700 1 --2- 172.21.15.88:0/1750249833 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7fbeb0132c40 0x7fbeb01339f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:33:38.384 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:38.373+0000 7fbeb5089700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:33:38.385 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:38.373+0000 7fbeb5089700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:33:38.385 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:38.373+0000 7fbeb5089700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:33:38.385 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:38.373+0000 7fbeb588a700 10 monclient(hunting): get_auth_request con 0x7fbeb0132c40 auth_method 0 2023-09-30T01:33:38.385 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:38.377+0000 7fbeb588a700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:33:38.385 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:38.377+0000 7fbeb588a700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:33:38.385 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:38.377+0000 7fbeb588a700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:33:38.385 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:38.377+0000 7fbea3fff700 1 -- 172.21.15.88:0/1750249833 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2455557720 0 0) 0x7fbeb004bb20 con 0x7fbeb01311c0 2023-09-30T01:33:38.385 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:38.377+0000 7fbea3fff700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:33:38.385 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:38.377+0000 7fbea3fff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:33:38.385 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:38.377+0000 7fbea3fff700 10 monclient(hunting): my global_id is 4500 2023-09-30T01:33:38.385 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:38.377+0000 7fbea3fff700 1 -- 172.21.15.88:0/1750249833 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbe9c003570 con 0x7fbeb01311c0 2023-09-30T01:33:38.385 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:38.377+0000 7fbeb588a700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:33:38.386 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:38.377+0000 7fbeb588a700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:33:38.386 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:38.377+0000 7fbeb588a700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:33:38.386 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:38.377+0000 7fbeb5089700 10 monclient(hunting): handle_auth_done global_id 4385 payload 274 2023-09-30T01:33:38.386 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:38.377+0000 7fbeb5089700 1 -- 172.21.15.88:0/1750249833 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7fbeb0132c40 msgr2=0x7fbeb01339f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:33:38.386 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:38.377+0000 7fbeb5089700 1 --2- 172.21.15.88:0/1750249833 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7fbeb0132c40 0x7fbeb01339f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:38.386 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:38.377+0000 7fbeb5089700 1 -- 172.21.15.88:0/1750249833 >> v1:172.21.15.88:6789/0 conn(0x7fbeb01311c0 legacy=0x7fbeb01332e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:33:38.386 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:38.377+0000 7fbeb5089700 10 monclient: _finish_hunting 0 2023-09-30T01:33:38.386 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:38.377+0000 7fbeb5089700 1 monclient: found mon.noname-b 2023-09-30T01:33:38.386 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:38.377+0000 7fbeb5089700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:33:38.386 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:38.377+0000 7fbeb5089700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.88:3301/0 2023-09-30T01:33:38.386 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:38.377+0000 7fbeb5089700 1 -- 172.21.15.88:0/1750249833 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbeb01330a0 con 0x7fbeb0097a40 2023-09-30T01:33:38.386 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:38.377+0000 7fbeb5089700 1 --2- 172.21.15.88:0/1750249833 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fbeb0097a40 0x7fbeb0132700 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fbeac009910 tx=0x7fbeac0091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f6b678feb0ad45ea server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:33:38.387 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:38.377+0000 7fbea3fff700 1 -- 172.21.15.88:0/1750249833 <== mon.2 v2:172.21.15.88:3301/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7fbeac011ab0 con 0x7fbeb0097a40 2023-09-30T01:33:38.387 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:38.377+0000 7fbea3fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:33:38.387 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:38.377+0000 7fbea3fff700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T01:33:38.387 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:38.377+0000 7fbea3fff700 10 monclient: dump: 2023-09-30T01:33:38.387 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:33:38.387 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:33:38.387 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:38.387 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:38.387 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:33:38.387 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:33:38.387 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:33:38.387 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:33:38.387 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:33:38.387 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:33:38.388 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:33:38.388 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:33:38.388 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:33:38.388 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:33:38.388 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:33:38.388 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:33:38.388 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:38.377+0000 7fbea3fff700 10 monclient: _finish_auth 0 2023-09-30T01:33:38.388 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:38.377+0000 7fbea3fff700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:33:38.388 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:38.377+0000 7fbea3fff700 1 -- 172.21.15.88:0/1750249833 <== mon.2 v2:172.21.15.88:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbeac011c30 con 0x7fbeb0097a40 2023-09-30T01:33:38.388 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:38.377+0000 7fbeb608b700 5 monclient: authenticate success, global_id 4385 2023-09-30T01:33:38.388 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:38.377+0000 7fbeb608b700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:33:38.388 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:38.377+0000 7fbea3fff700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:33:38.388 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:38.377+0000 7fbea3fff700 1 -- 172.21.15.88:0/1750249833 <== mon.2 v2:172.21.15.88:3301/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7fbeac011da0 con 0x7fbeb0097a40 2023-09-30T01:33:38.389 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:38.377+0000 7fbea3fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:33:38.389 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:38.377+0000 7fbea3fff700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T01:33:38.389 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:38.377+0000 7fbea3fff700 10 monclient: dump: 2023-09-30T01:33:38.389 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:33:38.389 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:33:38.389 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:38.389 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:38.389 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:33:38.389 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:33:38.389 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:33:38.389 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:33:38.389 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:33:38.390 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:33:38.390 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:33:38.390 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:33:38.390 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:33:38.390 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:33:38.390 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:33:38.390 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:33:38.390 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:38.377+0000 7fbeb608b700 10 monclient: get_monmap_and_config success 2023-09-30T01:33:38.390 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:38.377+0000 7fbeb608b700 4 set_mon_vals no callback set 2023-09-30T01:33:38.390 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:38.377+0000 7fbeb608b700 10 monclient: shutdown 2023-09-30T01:33:38.390 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:38.377+0000 7fbeb608b700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:33:38.390 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:38.377+0000 7fbeb608b700 1 -- 172.21.15.88:0/1750249833 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fbeb0097a40 msgr2=0x7fbeb0132700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:33:38.391 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:38.377+0000 7fbeb608b700 1 --2- 172.21.15.88:0/1750249833 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fbeb0097a40 0x7fbeb0132700 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fbeac009910 tx=0x7fbeac0091a0 comp rx=0 tx=0).stop 2023-09-30T01:33:38.391 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:38.377+0000 7fbeb608b700 1 -- 172.21.15.88:0/1750249833 shutdown_connections 2023-09-30T01:33:38.391 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:38.377+0000 7fbeb608b700 1 --2- 172.21.15.88:0/1750249833 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7fbeb0132c40 0x7fbeb01339f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:38.391 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:38.377+0000 7fbeb608b700 1 --2- 172.21.15.88:0/1750249833 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fbeb0097a40 0x7fbeb0132700 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:38.391 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:38.377+0000 7fbeb608b700 1 -- 172.21.15.88:0/1750249833 >> 172.21.15.88:0/1750249833 conn(0x7fbeb006e5e0 msgr2=0x7fbeb006e9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:33:38.391 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:38.377+0000 7fbeb608b700 1 -- 172.21.15.88:0/1750249833 shutdown_connections 2023-09-30T01:33:38.391 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:38.377+0000 7fbeb608b700 1 -- 172.21.15.88:0/1750249833 wait complete. 2023-09-30T01:33:38.396 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:38.381+0000 7fbeb608b700 10 monclient: build_initial_monmap 2023-09-30T01:33:38.396 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:38.381+0000 7fbeb608b700 10 monclient: monmap: 2023-09-30T01:33:38.396 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2023-09-30T01:33:38.396 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:33:38.396 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 0.000000 2023-09-30T01:33:38.396 INFO:teuthology.orchestra.run.smithi088.stderr:created 0.000000 2023-09-30T01:33:38.396 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2023-09-30T01:33:38.396 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 1 2023-09-30T01:33:38.396 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.noname-a 2023-09-30T01:33:38.397 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.noname-f 2023-09-30T01:33:38.397 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.noname-b 2023-09-30T01:33:38.397 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.noname-g 2023-09-30T01:33:38.397 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.noname-c 2023-09-30T01:33:38.397 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.noname-h 2023-09-30T01:33:38.397 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.noname-d 2023-09-30T01:33:38.397 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.noname-i 2023-09-30T01:33:38.397 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.noname-e 2023-09-30T01:33:38.397 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:33:38.397 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:38.381+0000 7fbeb608b700 1 Processor -- start 2023-09-30T01:33:38.397 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:38.381+0000 7fbeb608b700 1 -- start start 2023-09-30T01:33:38.397 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:38.381+0000 7fbeb608b700 10 monclient: init 2023-09-30T01:33:38.398 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:38.381+0000 7fbeb608b700 10 monclient: _reopen_session rank -1 2023-09-30T01:33:38.398 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:38.381+0000 7fbeb608b700 10 monclient: _add_conns ranks=[8,7,3,4,6,2,0,1,5] 2023-09-30T01:33:38.398 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:38.381+0000 7fbeb608b700 1 --2- >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7fbeb0097a40 0x7fbeb012fff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:33:38.398 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:38.381+0000 7fbeb608b700 10 monclient(hunting): picked mon.noname-e con 0x7fbeb0097a40 addr [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] 2023-09-30T01:33:38.398 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:38.381+0000 7fbeb608b700 1 --2- >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7fbeb01311c0 0x7fbeb011cd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:33:38.398 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:38.381+0000 7fbeb608b700 10 monclient(hunting): picked mon.noname-i con 0x7fbeb01311c0 addr [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] 2023-09-30T01:33:38.398 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:38.381+0000 7fbeb608b700 1 --2- >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7fbeb0132c40 0x7fbeb011d290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:33:38.398 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:38.381+0000 7fbeb608b700 10 monclient(hunting): picked mon.noname-g con 0x7fbeb0132c40 addr [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] 2023-09-30T01:33:38.398 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:38.381+0000 7fbeb608b700 10 monclient(hunting): start opening mon connection 2023-09-30T01:33:38.398 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:38.381+0000 7fbeb608b700 1 -- --> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbeb0120240 con 0x7fbeb0132c40 2023-09-30T01:33:38.398 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:38.381+0000 7fbeb608b700 10 monclient(hunting): start opening mon connection 2023-09-30T01:33:38.398 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:38.381+0000 7fbeb608b700 1 -- --> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] -- mon_getmap magic: 0 v1 -- 0x7fbeb0120770 con 0x7fbeb01311c0 2023-09-30T01:33:38.399 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:38.381+0000 7fbeb608b700 10 monclient(hunting): start opening mon connection 2023-09-30T01:33:38.399 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:38.381+0000 7fbeb608b700 1 -- --> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] -- mon_getmap magic: 0 v1 -- 0x7fbeb0099c30 con 0x7fbeb0097a40 2023-09-30T01:33:38.399 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:38.381+0000 7fbeb608b700 10 monclient(hunting): _renew_subs 2023-09-30T01:33:38.399 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:38.381+0000 7fbeb608b700 10 monclient(hunting): authenticate will time out at 890.422477s 2023-09-30T01:33:38.399 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:38.381+0000 7fbeb5089700 1 --2- >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7fbeb0097a40 0x7fbeb012fff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:33:38.399 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:38.381+0000 7fbeb5089700 1 --2- >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7fbeb0097a40 0x7fbeb012fff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3304/0 says I am v2:172.21.15.88:43430/0 (socket says 172.21.15.88:43430) 2023-09-30T01:33:38.399 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:38.381+0000 7fbeb5089700 1 -- 172.21.15.88:0/3166588025 learned_addr learned my addr 172.21.15.88:0/3166588025 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-09-30T01:33:38.399 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:38.381+0000 7fbeb5089700 10 monclient(hunting): get_auth_request con 0x7fbeb0097a40 auth_method 0 2023-09-30T01:33:38.399 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:38.381+0000 7fbeb5089700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:33:38.399 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:38.381+0000 7fbeb5089700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:33:38.399 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:38.381+0000 7fbeb5089700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:33:38.400 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:38.381+0000 7fbeb5089700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:33:38.400 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:38.381+0000 7fbeb5089700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:33:38.400 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:38.381+0000 7fbeb4888700 1 --2- 172.21.15.88:0/3166588025 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7fbeb01311c0 0x7fbeb011cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:33:38.400 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:38.381+0000 7fbeb5089700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:33:38.400 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:38.381+0000 7fbeb4888700 10 monclient(hunting): get_auth_request con 0x7fbeb01311c0 auth_method 0 2023-09-30T01:33:38.400 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:38.381+0000 7fbeb4888700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:33:38.400 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:38.381+0000 7fbeb4888700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:33:38.400 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:38.381+0000 7fbeb4888700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:33:38.400 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:38.381+0000 7fbeb588a700 1 --2- 172.21.15.88:0/3166588025 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7fbeb0132c40 0x7fbeb011d290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:33:38.400 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:38.381+0000 7fbeb4888700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:33:38.400 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:38.381+0000 7fbeb4888700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:33:38.401 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:38.381+0000 7fbeb4888700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:33:38.401 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:38.381+0000 7fbeb588a700 10 monclient(hunting): get_auth_request con 0x7fbeb0132c40 auth_method 0 2023-09-30T01:33:38.401 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:38.381+0000 7fbeb588a700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:33:38.401 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:38.381+0000 7fbeb588a700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:33:38.401 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:38.381+0000 7fbeb588a700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:33:38.401 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:38.381+0000 7fbeb5089700 10 monclient(hunting): handle_auth_done global_id 4445 payload 824 2023-09-30T01:33:38.401 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:38.381+0000 7fbeb5089700 1 -- 172.21.15.88:0/3166588025 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7fbeb01311c0 msgr2=0x7fbeb011cd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:33:38.401 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:38.381+0000 7fbeb5089700 1 --2- 172.21.15.88:0/3166588025 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7fbeb01311c0 0x7fbeb011cd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:38.401 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:38.381+0000 7fbeb5089700 1 -- 172.21.15.88:0/3166588025 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7fbeb0132c40 msgr2=0x7fbeb011d290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:33:38.401 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:38.381+0000 7fbeb5089700 1 --2- 172.21.15.88:0/3166588025 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7fbeb0132c40 0x7fbeb011d290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:38.401 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:38.381+0000 7fbeb5089700 10 monclient: _finish_hunting 0 2023-09-30T01:33:38.401 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:38.381+0000 7fbeb5089700 1 monclient: found mon.noname-e 2023-09-30T01:33:38.402 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:38.381+0000 7fbeb5089700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:33:38.402 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:38.381+0000 7fbeb5089700 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.88:3304/0 2023-09-30T01:33:38.402 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:38.381+0000 7fbeb5089700 1 -- 172.21.15.88:0/3166588025 --> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbeac009420 con 0x7fbeb0097a40 2023-09-30T01:33:38.402 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:38.381+0000 7fbeb588a700 1 --2- 172.21.15.88:0/3166588025 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7fbeb0132c40 0x7fbeb011d290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-30T01:33:38.402 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:38.381+0000 7fbeb5089700 1 --2- 172.21.15.88:0/3166588025 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7fbeb0097a40 0x7fbeb012fff0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fbeac011fd0 tx=0x7fbeac017000 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:33:38.402 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:38.381+0000 7fbea17fa700 1 -- 172.21.15.88:0/3166588025 <== mon.8 v2:172.21.15.88:3304/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7fbeac01f070 con 0x7fbeb0097a40 2023-09-30T01:33:38.402 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:38.381+0000 7fbea17fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:33:38.402 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:38.381+0000 7fbea17fa700 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2023-09-30T01:33:38.402 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:38.381+0000 7fbea17fa700 10 monclient: dump: 2023-09-30T01:33:38.402 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:33:38.402 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:33:38.403 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:38.403 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:38.403 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:33:38.403 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:33:38.403 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:33:38.403 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:33:38.403 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:33:38.403 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:33:38.403 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:33:38.403 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:33:38.403 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:33:38.403 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:33:38.404 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:33:38.404 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:33:38.407 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:38.381+0000 7fbea17fa700 10 monclient: _finish_auth 0 2023-09-30T01:33:38.407 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:38.381+0000 7fbea17fa700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:33:38.407 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:38.385+0000 7fbeb608b700 5 monclient: authenticate success, global_id 4445 2023-09-30T01:33:38.407 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:38.385+0000 7fbea17fa700 1 -- 172.21.15.88:0/3166588025 <== mon.8 v2:172.21.15.88:3304/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbeac006ab0 con 0x7fbeb0097a40 2023-09-30T01:33:38.407 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:38.385+0000 7fbea17fa700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:33:38.407 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:38.385+0000 7fbeb608b700 10 monclient: _renew_subs 2023-09-30T01:33:38.407 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:38.385+0000 7fbea17fa700 1 -- 172.21.15.88:0/3166588025 <== mon.8 v2:172.21.15.88:3304/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7fbeac01ae00 con 0x7fbeb0097a40 2023-09-30T01:33:38.408 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:38.385+0000 7fbeb608b700 10 monclient: _send_mon_message to mon.e at v2:172.21.15.88:3304/0 2023-09-30T01:33:38.408 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:38.385+0000 7fbeb608b700 1 -- 172.21.15.88:0/3166588025 --> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbeb011f250 con 0x7fbeb0097a40 2023-09-30T01:33:38.408 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:38.385+0000 7fbea2ffd700 4 set_mon_vals no callback set 2023-09-30T01:33:38.408 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:38.385+0000 7fbea17fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:33:38.408 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:38.385+0000 7fbea17fa700 10 monclient: got monmap 1 from mon.e (according to old e1) 2023-09-30T01:33:38.408 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:38.385+0000 7fbea17fa700 10 monclient: dump: 2023-09-30T01:33:38.408 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:33:38.408 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:33:38.408 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:38.408 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:38.408 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:33:38.408 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:33:38.409 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:33:38.409 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:33:38.409 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:33:38.409 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:33:38.409 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:33:38.409 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:33:38.409 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:33:38.409 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:33:38.409 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:33:38.409 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:33:38.409 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:38.385+0000 7fbeb608b700 10 monclient: _renew_subs 2023-09-30T01:33:38.409 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:38.385+0000 7fbeb608b700 10 monclient: _send_mon_message to mon.e at v2:172.21.15.88:3304/0 2023-09-30T01:33:38.410 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:38.385+0000 7fbeb608b700 1 -- 172.21.15.88:0/3166588025 --> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbeb0134840 con 0x7fbeb0097a40 2023-09-30T01:33:38.410 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:38.385+0000 7fbea17fa700 1 -- 172.21.15.88:0/3166588025 <== mon.8 v2:172.21.15.88:3304/0 4 ==== mgrmap(e 5) v1 ==== 40144+0+0 (secure 0 0 0) 0x7fbeac023070 con 0x7fbeb0097a40 2023-09-30T01:33:38.410 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:38.385+0000 7fbeb608b700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:33:38.410 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:38.385+0000 7fbeb608b700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:33:38.410 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:38.385+0000 7fbeb608b700 10 monclient: _send_mon_message to mon.e at v2:172.21.15.88:3304/0 2023-09-30T01:33:38.410 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:38.385+0000 7fbeb608b700 1 -- 172.21.15.88:0/3166588025 --> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbeb0071960 con 0x7fbeb0097a40 2023-09-30T01:33:38.410 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:38.385+0000 7fbea17fa700 1 --2- 172.21.15.88:0/3166588025 >> v2:172.21.15.107:6812/29926 conn(0x7fbe94033360 0x7fbe94035810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:33:38.410 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:38.385+0000 7fbea17fa700 1 -- 172.21.15.88:0/3166588025 <== mon.8 v2:172.21.15.88:3304/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 3502+0+0 (secure 0 0 0) 0x7fbeac044680 con 0x7fbeb0097a40 2023-09-30T01:33:38.410 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:38.385+0000 7fbeb4888700 1 --2- 172.21.15.88:0/3166588025 >> v2:172.21.15.107:6812/29926 conn(0x7fbe94033360 0x7fbe94035810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:33:38.410 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:38.385+0000 7fbeb4888700 10 monclient: get_auth_request con 0x7fbe94033360 auth_method 0 2023-09-30T01:33:38.410 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:38.385+0000 7fbeb4888700 1 --2- 172.21.15.88:0/3166588025 >> v2:172.21.15.107:6812/29926 conn(0x7fbe94033360 0x7fbe94035810 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fbea4007fd0 tx=0x7fbea4009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:33:38.411 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:38.393+0000 7fbea17fa700 1 -- 172.21.15.88:0/3166588025 <== mon.8 v2:172.21.15.88:3304/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fbeac02f760 con 0x7fbeb0097a40 2023-09-30T01:33:38.411 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:38.393+0000 7fbea17fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:33:38.411 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:38.393+0000 7fbea17fa700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:33:38.591 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:38.585+0000 7fbeb608b700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T01:33:38.591 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:38.585+0000 7fbeb608b700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:33:38.591 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:38.585+0000 7fbeb608b700 10 monclient: _send_mon_message to mon.e at v2:172.21.15.88:3304/0 2023-09-30T01:33:38.591 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:38.585+0000 7fbeb608b700 1 -- 172.21.15.88:0/3166588025 --> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fbeb00996b0 con 0x7fbeb0097a40 2023-09-30T01:33:38.591 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:38.589+0000 7fbea17fa700 1 -- 172.21.15.88:0/3166588025 <== mon.8 v2:172.21.15.88:3304/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7fbeac015080 con 0x7fbeb0097a40 2023-09-30T01:33:38.592 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:38.589+0000 7fbea17fa700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:33:38.592 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:38.589+0000 7fbea17fa700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:33:38.592 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-09-30T01:33:38.592 INFO:teuthology.orchestra.run.smithi088.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:33:38.594 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:38.589+0000 7fbeb608b700 1 -- 172.21.15.88:0/3166588025 >> v2:172.21.15.107:6812/29926 conn(0x7fbe94033360 msgr2=0x7fbe94035810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:33:38.594 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:38.589+0000 7fbeb608b700 1 --2- 172.21.15.88:0/3166588025 >> v2:172.21.15.107:6812/29926 conn(0x7fbe94033360 0x7fbe94035810 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fbea4007fd0 tx=0x7fbea4009040 comp rx=0 tx=0).stop 2023-09-30T01:33:38.594 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:38.589+0000 7fbeb608b700 10 monclient: shutdown 2023-09-30T01:33:38.594 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:38.589+0000 7fbeb608b700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:33:38.594 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:38.589+0000 7fbeb608b700 1 -- 172.21.15.88:0/3166588025 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7fbeb0097a40 msgr2=0x7fbeb012fff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:33:38.594 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:38.589+0000 7fbeb608b700 1 --2- 172.21.15.88:0/3166588025 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7fbeb0097a40 0x7fbeb012fff0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fbeac011fd0 tx=0x7fbeac017000 comp rx=0 tx=0).stop 2023-09-30T01:33:38.594 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:38.589+0000 7fbeb608b700 1 -- 172.21.15.88:0/3166588025 shutdown_connections 2023-09-30T01:33:38.594 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:38.589+0000 7fbeb608b700 1 --2- 172.21.15.88:0/3166588025 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7fbeb0132c40 0x7fbeb011d290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:38.594 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:38.589+0000 7fbeb608b700 1 --2- 172.21.15.88:0/3166588025 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7fbeb01311c0 0x7fbeb011cd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:38.595 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:38.589+0000 7fbeb608b700 1 --2- 172.21.15.88:0/3166588025 >> v2:172.21.15.107:6812/29926 conn(0x7fbe94033360 0x7fbe94035810 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:38.595 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:38.589+0000 7fbeb608b700 1 --2- 172.21.15.88:0/3166588025 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7fbeb0097a40 0x7fbeb012fff0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:38.595 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:38.589+0000 7fbeb608b700 1 -- 172.21.15.88:0/3166588025 >> 172.21.15.88:0/3166588025 conn(0x7fbeb006e5e0 msgr2=0x7fbeb00a53e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:33:38.595 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:38.589+0000 7fbeb608b700 1 -- 172.21.15.88:0/3166588025 shutdown_connections 2023-09-30T01:33:38.595 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:38.589+0000 7fbeb608b700 1 -- 172.21.15.88:0/3166588025 wait complete. 2023-09-30T01:33:41.606 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:33:41.717 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:41.713+0000 7f23a54d0700 10 monclient: get_monmap_and_config 2023-09-30T01:33:41.717 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:41.713+0000 7f23a54d0700 10 monclient: build_initial_monmap 2023-09-30T01:33:41.717 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:41.713+0000 7f23a54d0700 10 monclient: monmap: 2023-09-30T01:33:41.717 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2023-09-30T01:33:41.718 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:33:41.718 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:33:41.717874+0000 2023-09-30T01:33:41.718 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:33:41.717874+0000 2023-09-30T01:33:41.718 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2023-09-30T01:33:41.718 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:33:41.718 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a-legacy 2023-09-30T01:33:41.718 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.107:6789/0 mon.noname-f-legacy 2023-09-30T01:33:41.718 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.noname-b 2023-09-30T01:33:41.718 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.noname-g 2023-09-30T01:33:41.718 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.noname-c 2023-09-30T01:33:41.718 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.noname-h 2023-09-30T01:33:41.719 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.noname-d 2023-09-30T01:33:41.719 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.noname-i 2023-09-30T01:33:41.719 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.noname-e 2023-09-30T01:33:41.719 INFO:teuthology.orchestra.run.smithi088.stderr:9: v2:172.21.15.88:3300/0 mon.noname-a 2023-09-30T01:33:41.719 INFO:teuthology.orchestra.run.smithi088.stderr:10: v2:172.21.15.107:3300/0 mon.noname-f 2023-09-30T01:33:41.719 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:33:41.723 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:41.713+0000 7f23a54d0700 1 Processor -- start 2023-09-30T01:33:41.723 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:41.713+0000 7f23a54d0700 1 -- start start 2023-09-30T01:33:41.723 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:41.713+0000 7f23a54d0700 10 monclient: init 2023-09-30T01:33:41.723 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:41.713+0000 7f23a54d0700 10 monclient: _reopen_session rank -1 2023-09-30T01:33:41.724 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:41.713+0000 7f23a54d0700 10 monclient: _add_conns ranks=[1,7,2,4,6,3,9,8,10,0,5] 2023-09-30T01:33:41.724 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:41.717+0000 7f23a54d0700 10 monclient(hunting): picked mon.noname-f-legacy con 0x7f23a00966a0 addr v1:172.21.15.107:6789/0 2023-09-30T01:33:41.724 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:41.717+0000 7f23a54d0700 1 --2- >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7f23a0097e40 0x7f23a0138650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:33:41.724 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:41.717+0000 7f23a54d0700 10 monclient(hunting): picked mon.noname-i con 0x7f23a0097e40 addr [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] 2023-09-30T01:33:41.724 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:41.717+0000 7f23a54d0700 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f23a0098210 0x7f23a013aba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:33:41.724 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:41.717+0000 7f23a54d0700 10 monclient(hunting): picked mon.noname-b con 0x7f23a0098210 addr [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] 2023-09-30T01:33:41.724 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:41.717+0000 7f23a54d0700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23a005ecb0 con 0x7f23a00966a0 2023-09-30T01:33:41.724 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:41.717+0000 7f23a54d0700 10 monclient(hunting): start opening mon connection 2023-09-30T01:33:41.724 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:41.717+0000 7f23a54d0700 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23a0097180 con 0x7f23a0098210 2023-09-30T01:33:41.724 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:41.717+0000 7f23a54d0700 10 monclient(hunting): start opening mon connection 2023-09-30T01:33:41.724 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:41.717+0000 7f23a54d0700 1 -- --> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] -- mon_getmap magic: 0 v1 -- 0x7f23a0097740 con 0x7f23a0097e40 2023-09-30T01:33:41.725 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:41.717+0000 7f23a54d0700 10 monclient(hunting): _renew_subs 2023-09-30T01:33:41.725 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:41.717+0000 7f23a54d0700 10 monclient(hunting): authenticate will time out at 893.756100s 2023-09-30T01:33:41.725 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:41.717+0000 7f239f7fe700 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f23a0098210 0x7f23a013aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:33:41.725 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:41.717+0000 7f239f7fe700 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f23a0098210 0x7f23a013aba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3301/0 says I am v2:172.21.15.88:42236/0 (socket says 172.21.15.88:42236) 2023-09-30T01:33:41.725 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:41.717+0000 7f239f7fe700 1 -- 172.21.15.88:0/143552823 learned_addr learned my addr 172.21.15.88:0/143552823 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-09-30T01:33:41.725 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:41.717+0000 7f239f7fe700 10 monclient(hunting): get_auth_request con 0x7f23a0098210 auth_method 0 2023-09-30T01:33:41.725 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:41.717+0000 7f239f7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:33:41.725 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:41.717+0000 7f239f7fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:33:41.725 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:41.717+0000 7f239f7fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:33:41.725 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:41.717+0000 7f239f7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:33:41.725 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:41.717+0000 7f239f7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:33:41.725 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:41.717+0000 7f239f7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:33:41.726 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:41.717+0000 7f239e7fc700 1 --2- 172.21.15.88:0/143552823 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7f23a0097e40 0x7f23a0138650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:33:41.726 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:41.717+0000 7f239e7fc700 10 monclient(hunting): get_auth_request con 0x7f23a0097e40 auth_method 0 2023-09-30T01:33:41.726 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:41.717+0000 7f239e7fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:33:41.726 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:41.717+0000 7f239e7fc700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:33:41.726 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:41.717+0000 7f239e7fc700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:33:41.726 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:41.717+0000 7f239f7fe700 10 monclient(hunting): handle_auth_done global_id 4394 payload 274 2023-09-30T01:33:41.726 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:41.717+0000 7f239f7fe700 1 -- 172.21.15.88:0/143552823 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7f23a0097e40 msgr2=0x7f23a0138650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:33:41.726 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:41.717+0000 7f239f7fe700 1 --2- 172.21.15.88:0/143552823 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7f23a0097e40 0x7f23a0138650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:41.726 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:41.717+0000 7f239f7fe700 1 -- 172.21.15.88:0/143552823 >> v1:172.21.15.107:6789/0 conn(0x7f23a00966a0 legacy=0x7f23a0096a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:33:41.726 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:41.717+0000 7f239f7fe700 10 monclient: _finish_hunting 0 2023-09-30T01:33:41.726 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:41.717+0000 7f239f7fe700 1 monclient: found mon.noname-b 2023-09-30T01:33:41.727 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:41.717+0000 7f239f7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:33:41.727 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:41.717+0000 7f239f7fe700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.88:3301/0 2023-09-30T01:33:41.727 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:41.717+0000 7f239f7fe700 1 -- 172.21.15.88:0/143552823 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23a0063510 con 0x7f23a0098210 2023-09-30T01:33:41.727 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:41.717+0000 7f239f7fe700 1 --2- 172.21.15.88:0/143552823 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f23a0098210 0x7f23a013aba0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f23980099e0 tx=0x7f2398009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=569336765b6dc128 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:33:41.727 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:41.717+0000 7f239dffb700 1 -- 172.21.15.88:0/143552823 <== mon.2 v2:172.21.15.88:3301/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f239800bce0 con 0x7f23a0098210 2023-09-30T01:33:41.727 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:41.717+0000 7f239dffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:33:41.727 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:41.717+0000 7f239dffb700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T01:33:41.727 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:41.717+0000 7f239dffb700 10 monclient: dump: 2023-09-30T01:33:41.727 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:33:41.727 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:33:41.727 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:41.727 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:41.728 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:33:41.728 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:33:41.728 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:33:41.728 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:33:41.728 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:33:41.728 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:33:41.728 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:33:41.728 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:33:41.728 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:33:41.728 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:33:41.728 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:33:41.728 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:33:41.729 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:41.717+0000 7f239dffb700 10 monclient: _finish_auth 0 2023-09-30T01:33:41.729 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:41.717+0000 7f239dffb700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:33:41.729 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:41.717+0000 7f239dffb700 1 -- 172.21.15.88:0/143552823 <== mon.2 v2:172.21.15.88:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f239800be40 con 0x7f23a0098210 2023-09-30T01:33:41.729 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:41.717+0000 7f23a54d0700 5 monclient: authenticate success, global_id 4394 2023-09-30T01:33:41.729 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:41.717+0000 7f23a54d0700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:33:41.729 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:41.717+0000 7f239dffb700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:33:41.729 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:41.717+0000 7f23a54d0700 10 monclient: get_monmap_and_config success 2023-09-30T01:33:41.729 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:41.717+0000 7f239dffb700 1 -- 172.21.15.88:0/143552823 <== mon.2 v2:172.21.15.88:3301/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f23980118a0 con 0x7f23a0098210 2023-09-30T01:33:41.729 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:41.717+0000 7f23a54d0700 4 set_mon_vals no callback set 2023-09-30T01:33:41.729 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:41.717+0000 7f239dffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:33:41.729 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:41.717+0000 7f239dffb700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T01:33:41.729 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:41.717+0000 7f239dffb700 10 monclient: dump: 2023-09-30T01:33:41.730 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:33:41.730 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:33:41.730 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:41.730 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:41.730 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:33:41.730 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:33:41.730 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:33:41.730 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:33:41.730 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:33:41.730 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:33:41.730 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:33:41.730 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:33:41.731 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:33:41.731 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:33:41.731 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:33:41.731 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:33:41.731 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:41.717+0000 7f23a54d0700 10 monclient: shutdown 2023-09-30T01:33:41.731 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:41.717+0000 7f23a54d0700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:33:41.731 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:41.717+0000 7f23a54d0700 1 -- 172.21.15.88:0/143552823 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f23a0098210 msgr2=0x7f23a013aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:33:41.731 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:41.717+0000 7f23a54d0700 1 --2- 172.21.15.88:0/143552823 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f23a0098210 0x7f23a013aba0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f23980099e0 tx=0x7f2398009250 comp rx=0 tx=0).stop 2023-09-30T01:33:41.731 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:41.717+0000 7f23a54d0700 1 -- 172.21.15.88:0/143552823 shutdown_connections 2023-09-30T01:33:41.731 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:41.717+0000 7f23a54d0700 1 --2- 172.21.15.88:0/143552823 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7f23a0097e40 0x7f23a0138650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:41.731 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:41.717+0000 7f23a54d0700 1 --2- 172.21.15.88:0/143552823 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f23a0098210 0x7f23a013aba0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:41.731 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:41.717+0000 7f23a54d0700 1 -- 172.21.15.88:0/143552823 >> 172.21.15.88:0/143552823 conn(0x7f23a00a3440 msgr2=0x7f23a00a3840 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:33:41.732 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:41.717+0000 7f23a54d0700 1 -- 172.21.15.88:0/143552823 shutdown_connections 2023-09-30T01:33:41.732 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:41.717+0000 7f23a54d0700 1 -- 172.21.15.88:0/143552823 wait complete. 2023-09-30T01:33:41.732 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:41.721+0000 7f23a54d0700 10 monclient: build_initial_monmap 2023-09-30T01:33:41.732 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:41.721+0000 7f23a54d0700 10 monclient: monmap: 2023-09-30T01:33:41.732 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2023-09-30T01:33:41.732 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:33:41.732 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 0.000000 2023-09-30T01:33:41.732 INFO:teuthology.orchestra.run.smithi088.stderr:created 0.000000 2023-09-30T01:33:41.732 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2023-09-30T01:33:41.732 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 1 2023-09-30T01:33:41.732 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.noname-a 2023-09-30T01:33:41.732 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.noname-f 2023-09-30T01:33:41.733 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.noname-b 2023-09-30T01:33:41.733 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.noname-g 2023-09-30T01:33:41.733 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.noname-c 2023-09-30T01:33:41.733 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.noname-h 2023-09-30T01:33:41.733 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.noname-d 2023-09-30T01:33:41.733 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.noname-i 2023-09-30T01:33:41.733 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.noname-e 2023-09-30T01:33:41.733 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:33:41.738 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:41.721+0000 7f23a54d0700 1 Processor -- start 2023-09-30T01:33:41.738 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:41.721+0000 7f23a54d0700 1 -- start start 2023-09-30T01:33:41.738 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:41.721+0000 7f23a54d0700 10 monclient: init 2023-09-30T01:33:41.738 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:41.721+0000 7f23a54d0700 10 monclient: _reopen_session rank -1 2023-09-30T01:33:41.738 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:41.721+0000 7f23a54d0700 10 monclient: _add_conns ranks=[2,5,8,4,7,0,6,1,3] 2023-09-30T01:33:41.738 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:41.721+0000 7f23a54d0700 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f23a00966a0 0x7f23a0136850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:33:41.738 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:41.721+0000 7f23a54d0700 10 monclient(hunting): picked mon.noname-b con 0x7f23a00966a0 addr [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] 2023-09-30T01:33:41.738 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:41.721+0000 7f23a54d0700 1 --2- >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7f23a0097e40 0x7f23a0136d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:33:41.738 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:41.721+0000 7f23a54d0700 10 monclient(hunting): picked mon.noname-h con 0x7f23a0097e40 addr [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] 2023-09-30T01:33:41.738 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:41.721+0000 7f23a54d0700 1 --2- >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f23a0098210 0x7f23a008e3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:33:41.739 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:41.721+0000 7f23a54d0700 10 monclient(hunting): picked mon.noname-e con 0x7f23a0098210 addr [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] 2023-09-30T01:33:41.739 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:41.721+0000 7f23a54d0700 10 monclient(hunting): start opening mon connection 2023-09-30T01:33:41.739 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:41.721+0000 7f23a54d0700 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23a0091640 con 0x7f23a00966a0 2023-09-30T01:33:41.739 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:41.721+0000 7f23a54d0700 10 monclient(hunting): start opening mon connection 2023-09-30T01:33:41.739 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:41.721+0000 7f23a54d0700 1 -- --> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] -- mon_getmap magic: 0 v1 -- 0x7f23a0091c90 con 0x7f23a0097e40 2023-09-30T01:33:41.739 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:41.721+0000 7f23a54d0700 10 monclient(hunting): start opening mon connection 2023-09-30T01:33:41.739 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:41.721+0000 7f23a54d0700 1 -- --> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] -- mon_getmap magic: 0 v1 -- 0x7f23a009f780 con 0x7f23a0098210 2023-09-30T01:33:41.739 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:41.721+0000 7f23a54d0700 10 monclient(hunting): _renew_subs 2023-09-30T01:33:41.739 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:41.721+0000 7f23a54d0700 10 monclient(hunting): authenticate will time out at 893.762816s 2023-09-30T01:33:41.739 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:41.721+0000 7f239effd700 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f23a00966a0 0x7f23a0136850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:33:41.739 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:41.721+0000 7f239f7fe700 1 --2- >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f23a0098210 0x7f23a008e3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:33:41.740 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:41.721+0000 7f239effd700 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f23a00966a0 0x7f23a0136850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3301/0 says I am v2:172.21.15.88:42248/0 (socket says 172.21.15.88:42248) 2023-09-30T01:33:41.740 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:41.721+0000 7f239effd700 1 -- 172.21.15.88:0/3005748834 learned_addr learned my addr 172.21.15.88:0/3005748834 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-09-30T01:33:41.740 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:41.721+0000 7f239effd700 10 monclient(hunting): get_auth_request con 0x7f23a00966a0 auth_method 0 2023-09-30T01:33:41.740 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:41.721+0000 7f239effd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:33:41.740 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:41.721+0000 7f239effd700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:33:41.740 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:41.721+0000 7f239effd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:33:41.740 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:41.721+0000 7f239e7fc700 1 --2- 172.21.15.88:0/3005748834 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7f23a0097e40 0x7f23a0136d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:33:41.740 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:41.721+0000 7f239f7fe700 10 monclient(hunting): get_auth_request con 0x7f23a0098210 auth_method 0 2023-09-30T01:33:41.740 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:41.721+0000 7f239f7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:33:41.740 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:41.721+0000 7f239f7fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:33:41.740 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:41.721+0000 7f239f7fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:33:41.740 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:41.721+0000 7f239e7fc700 10 monclient(hunting): get_auth_request con 0x7f23a0097e40 auth_method 0 2023-09-30T01:33:41.741 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:41.721+0000 7f239e7fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:33:41.741 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:41.721+0000 7f239e7fc700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:33:41.741 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:41.721+0000 7f239e7fc700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:33:41.741 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:41.721+0000 7f239effd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:33:41.741 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:41.721+0000 7f239effd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:33:41.741 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:41.721+0000 7f239effd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:33:41.741 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:41.721+0000 7f239f7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:33:41.741 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:41.721+0000 7f239f7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:33:41.741 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:41.721+0000 7f239f7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:33:41.741 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:41.721+0000 7f239e7fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:33:41.741 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:41.721+0000 7f239e7fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:33:41.741 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:41.721+0000 7f239e7fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:33:41.742 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:41.721+0000 7f239effd700 10 monclient(hunting): handle_auth_done global_id 4403 payload 824 2023-09-30T01:33:41.742 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:41.721+0000 7f239effd700 1 -- 172.21.15.88:0/3005748834 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f23a0098210 msgr2=0x7f23a008e3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:33:41.742 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:41.721+0000 7f239effd700 1 --2- 172.21.15.88:0/3005748834 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f23a0098210 0x7f23a008e3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:41.742 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:41.721+0000 7f239effd700 1 -- 172.21.15.88:0/3005748834 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7f23a0097e40 msgr2=0x7f23a0136d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:33:41.742 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:41.721+0000 7f239effd700 1 --2- 172.21.15.88:0/3005748834 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7f23a0097e40 0x7f23a0136d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:41.742 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:41.721+0000 7f239effd700 10 monclient: _finish_hunting 0 2023-09-30T01:33:41.742 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:41.721+0000 7f239effd700 1 monclient: found mon.noname-b 2023-09-30T01:33:41.742 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:41.721+0000 7f239effd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:33:41.742 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:41.721+0000 7f239effd700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.88:3301/0 2023-09-30T01:33:41.742 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:41.721+0000 7f239effd700 1 -- 172.21.15.88:0/3005748834 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23980094f0 con 0x7f23a00966a0 2023-09-30T01:33:41.742 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:41.721+0000 7f239effd700 1 --2- 172.21.15.88:0/3005748834 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f23a00966a0 0x7f23a0136850 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f239000cb70 tx=0x7f2390001770 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:33:41.743 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:41.725+0000 7f2396ffd700 1 -- 172.21.15.88:0/3005748834 <== mon.2 v2:172.21.15.88:3301/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f239000ac20 con 0x7f23a00966a0 2023-09-30T01:33:41.743 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:41.725+0000 7f2396ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:33:41.743 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:41.725+0000 7f2396ffd700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T01:33:41.743 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:41.725+0000 7f2396ffd700 10 monclient: dump: 2023-09-30T01:33:41.743 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:33:41.743 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:33:41.743 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:41.743 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:41.743 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:33:41.743 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:33:41.743 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:33:41.743 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:33:41.743 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:33:41.744 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:33:41.744 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:33:41.744 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:33:41.744 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:33:41.744 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:33:41.744 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:33:41.744 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:33:41.744 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:41.725+0000 7f2396ffd700 10 monclient: _finish_auth 0 2023-09-30T01:33:41.744 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:41.725+0000 7f2396ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:33:41.744 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:41.725+0000 7f2396ffd700 1 -- 172.21.15.88:0/3005748834 <== mon.2 v2:172.21.15.88:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f239000adc0 con 0x7f23a00966a0 2023-09-30T01:33:41.744 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:41.725+0000 7f23a54d0700 5 monclient: authenticate success, global_id 4403 2023-09-30T01:33:41.744 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:41.725+0000 7f23a54d0700 10 monclient: _renew_subs 2023-09-30T01:33:41.745 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:41.725+0000 7f23a54d0700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.88:3301/0 2023-09-30T01:33:41.745 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:41.725+0000 7f23a54d0700 1 -- 172.21.15.88:0/3005748834 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23a0099830 con 0x7f23a00966a0 2023-09-30T01:33:41.745 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:41.725+0000 7f2396ffd700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:33:41.745 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:41.725+0000 7f2396ffd700 1 -- 172.21.15.88:0/3005748834 <== mon.2 v2:172.21.15.88:3301/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f23900128a0 con 0x7f23a00966a0 2023-09-30T01:33:41.745 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:41.725+0000 7f2396ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:33:41.745 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:41.725+0000 7f239d7fa700 4 set_mon_vals no callback set 2023-09-30T01:33:41.745 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:41.725+0000 7f2396ffd700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T01:33:41.745 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:41.725+0000 7f2396ffd700 10 monclient: dump: 2023-09-30T01:33:41.745 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:33:41.745 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:33:41.745 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:41.745 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:41.746 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:33:41.746 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:33:41.746 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:33:41.746 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:33:41.746 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:33:41.746 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:33:41.746 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:33:41.746 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:33:41.746 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:33:41.746 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:33:41.746 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:33:41.746 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:33:41.747 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:41.725+0000 7f23a54d0700 10 monclient: _renew_subs 2023-09-30T01:33:41.747 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:41.725+0000 7f23a54d0700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.88:3301/0 2023-09-30T01:33:41.747 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:41.725+0000 7f23a54d0700 1 -- 172.21.15.88:0/3005748834 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f23a0090750 con 0x7f23a00966a0 2023-09-30T01:33:41.747 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:41.725+0000 7f23a54d0700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:33:41.747 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:41.725+0000 7f23a54d0700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:33:41.747 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:41.725+0000 7f23a54d0700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.88:3301/0 2023-09-30T01:33:41.747 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:41.725+0000 7f23a54d0700 1 -- 172.21.15.88:0/3005748834 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f23a0062f80 con 0x7f23a00966a0 2023-09-30T01:33:41.747 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:41.725+0000 7f2396ffd700 1 -- 172.21.15.88:0/3005748834 <== mon.2 v2:172.21.15.88:3301/0 4 ==== mgrmap(e 5) v1 ==== 40144+0+0 (secure 0 0 0) 0x7f2390012420 con 0x7f23a00966a0 2023-09-30T01:33:41.747 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:41.725+0000 7f2396ffd700 1 --2- 172.21.15.88:0/3005748834 >> v2:172.21.15.107:6812/29926 conn(0x7f2384033360 0x7f2384035810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:33:41.747 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:41.725+0000 7f2396ffd700 1 -- 172.21.15.88:0/3005748834 <== mon.2 v2:172.21.15.88:3301/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 3502+0+0 (secure 0 0 0) 0x7f2390042be0 con 0x7f23a00966a0 2023-09-30T01:33:41.747 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:41.725+0000 7f239e7fc700 1 --2- 172.21.15.88:0/3005748834 >> v2:172.21.15.107:6812/29926 conn(0x7f2384033360 0x7f2384035810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:33:41.747 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:41.725+0000 7f239e7fc700 10 monclient: get_auth_request con 0x7f2384033360 auth_method 0 2023-09-30T01:33:41.748 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:41.725+0000 7f239e7fc700 1 --2- 172.21.15.88:0/3005748834 >> v2:172.21.15.107:6812/29926 conn(0x7f2384033360 0x7f2384035810 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f2388006fd0 tx=0x7f2388008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:33:41.748 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:41.733+0000 7f2396ffd700 1 -- 172.21.15.88:0/3005748834 <== mon.2 v2:172.21.15.88:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f23900040d0 con 0x7f23a00966a0 2023-09-30T01:33:41.748 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:41.733+0000 7f2396ffd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:33:41.748 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:41.733+0000 7f2396ffd700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:33:41.934 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:41.929+0000 7f23a54d0700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T01:33:41.934 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:41.929+0000 7f23a54d0700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:33:41.934 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:41.929+0000 7f23a54d0700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.88:3301/0 2023-09-30T01:33:41.934 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:41.929+0000 7f23a54d0700 1 -- 172.21.15.88:0/3005748834 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f23a0071960 con 0x7f23a00966a0 2023-09-30T01:33:41.935 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:41.929+0000 7f2396ffd700 1 -- 172.21.15.88:0/3005748834 <== mon.2 v2:172.21.15.88:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f2390016070 con 0x7f23a00966a0 2023-09-30T01:33:41.935 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:41.929+0000 7f2396ffd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:33:41.935 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:41.929+0000 7f2396ffd700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:33:41.935 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-09-30T01:33:41.935 INFO:teuthology.orchestra.run.smithi088.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:33:41.937 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:41.933+0000 7f23a54d0700 1 -- 172.21.15.88:0/3005748834 >> v2:172.21.15.107:6812/29926 conn(0x7f2384033360 msgr2=0x7f2384035810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:33:41.937 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:41.933+0000 7f23a54d0700 1 --2- 172.21.15.88:0/3005748834 >> v2:172.21.15.107:6812/29926 conn(0x7f2384033360 0x7f2384035810 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f2388006fd0 tx=0x7f2388008040 comp rx=0 tx=0).stop 2023-09-30T01:33:41.937 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:41.933+0000 7f23a54d0700 10 monclient: shutdown 2023-09-30T01:33:41.937 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:41.933+0000 7f23a54d0700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:33:41.937 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:41.933+0000 7f23a54d0700 1 -- 172.21.15.88:0/3005748834 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f23a00966a0 msgr2=0x7f23a0136850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:33:41.937 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:41.933+0000 7f23a54d0700 1 --2- 172.21.15.88:0/3005748834 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f23a00966a0 0x7f23a0136850 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f239000cb70 tx=0x7f2390001770 comp rx=0 tx=0).stop 2023-09-30T01:33:41.937 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:41.933+0000 7f23a54d0700 1 -- 172.21.15.88:0/3005748834 shutdown_connections 2023-09-30T01:33:41.937 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:41.933+0000 7f23a54d0700 1 --2- 172.21.15.88:0/3005748834 >> v2:172.21.15.107:6812/29926 conn(0x7f2384033360 0x7f2384035810 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:41.937 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:41.933+0000 7f23a54d0700 1 --2- 172.21.15.88:0/3005748834 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f23a0098210 0x7f23a008e3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:41.938 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:41.933+0000 7f23a54d0700 1 --2- 172.21.15.88:0/3005748834 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7f23a0097e40 0x7f23a0136d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:41.938 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:41.933+0000 7f23a54d0700 1 --2- 172.21.15.88:0/3005748834 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f23a00966a0 0x7f23a0136850 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:41.938 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:41.933+0000 7f23a54d0700 1 -- 172.21.15.88:0/3005748834 >> 172.21.15.88:0/3005748834 conn(0x7f23a00a3440 msgr2=0x7f23a00a3e30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:33:41.938 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:41.933+0000 7f23a54d0700 1 -- 172.21.15.88:0/3005748834 shutdown_connections 2023-09-30T01:33:41.938 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:41.933+0000 7f23a54d0700 1 -- 172.21.15.88:0/3005748834 wait complete. 2023-09-30T01:33:44.948 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:33:45.059 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:45.057+0000 7f5254f2b700 10 monclient: get_monmap_and_config 2023-09-30T01:33:45.060 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:45.057+0000 7f5254f2b700 10 monclient: build_initial_monmap 2023-09-30T01:33:45.060 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:45.057+0000 7f5254f2b700 10 monclient: monmap: 2023-09-30T01:33:45.060 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2023-09-30T01:33:45.060 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:33:45.060 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:33:45.060139+0000 2023-09-30T01:33:45.060 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:33:45.060139+0000 2023-09-30T01:33:45.060 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2023-09-30T01:33:45.060 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:33:45.060 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a-legacy 2023-09-30T01:33:45.060 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.107:6789/0 mon.noname-f-legacy 2023-09-30T01:33:45.060 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.noname-b 2023-09-30T01:33:45.060 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.noname-g 2023-09-30T01:33:45.061 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.noname-c 2023-09-30T01:33:45.061 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.noname-h 2023-09-30T01:33:45.061 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.noname-d 2023-09-30T01:33:45.061 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.noname-i 2023-09-30T01:33:45.061 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.noname-e 2023-09-30T01:33:45.061 INFO:teuthology.orchestra.run.smithi088.stderr:9: v2:172.21.15.88:3300/0 mon.noname-a 2023-09-30T01:33:45.061 INFO:teuthology.orchestra.run.smithi088.stderr:10: v2:172.21.15.107:3300/0 mon.noname-f 2023-09-30T01:33:45.061 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:33:45.065 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:45.057+0000 7f5254f2b700 1 Processor -- start 2023-09-30T01:33:45.066 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:45.057+0000 7f5254f2b700 1 -- start start 2023-09-30T01:33:45.066 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:45.057+0000 7f5254f2b700 10 monclient: init 2023-09-30T01:33:45.066 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:45.057+0000 7f5254f2b700 10 monclient: _reopen_session rank -1 2023-09-30T01:33:45.066 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:45.057+0000 7f5254f2b700 10 monclient: _add_conns ranks=[8,3,4,0,7,2,1,10,9,5,6] 2023-09-30T01:33:45.066 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:45.057+0000 7f5254f2b700 1 --2- >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f5250097a40 0x7f5250132700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:33:45.066 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:45.057+0000 7f5254f2b700 10 monclient(hunting): picked mon.noname-e con 0x7f5250097a40 addr [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] 2023-09-30T01:33:45.066 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:45.057+0000 7f5254f2b700 1 --2- >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7f52501311c0 0x7f5250132c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:33:45.066 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:45.057+0000 7f5254f2b700 10 monclient(hunting): picked mon.noname-g con 0x7f52501311c0 addr [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] 2023-09-30T01:33:45.066 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:45.057+0000 7f5254f2b700 1 --2- >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f52501332e0 0x7f5250133710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:33:45.066 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:45.057+0000 7f5254f2b700 10 monclient(hunting): picked mon.noname-c con 0x7f52501332e0 addr [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] 2023-09-30T01:33:45.066 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:45.057+0000 7f5254f2b700 10 monclient(hunting): start opening mon connection 2023-09-30T01:33:45.066 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:45.057+0000 7f5254f2b700 1 -- --> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5250131590 con 0x7f52501311c0 2023-09-30T01:33:45.067 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:45.057+0000 7f5254f2b700 10 monclient(hunting): start opening mon connection 2023-09-30T01:33:45.067 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:45.057+0000 7f5254f2b700 1 -- --> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] -- mon_getmap magic: 0 v1 -- 0x7f525009a5d0 con 0x7f52501332e0 2023-09-30T01:33:45.067 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:45.057+0000 7f5254f2b700 10 monclient(hunting): start opening mon connection 2023-09-30T01:33:45.067 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:45.057+0000 7f5254f2b700 1 -- --> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] -- mon_getmap magic: 0 v1 -- 0x7f5250097e10 con 0x7f5250097a40 2023-09-30T01:33:45.067 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:45.057+0000 7f5254f2b700 10 monclient(hunting): _renew_subs 2023-09-30T01:33:45.067 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:45.057+0000 7f5254f2b700 10 monclient(hunting): authenticate will time out at 897.098091s 2023-09-30T01:33:45.069 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:45.057+0000 7f524ffff700 1 --2- >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f52501332e0 0x7f5250133710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:33:45.069 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:45.057+0000 7f524f7fe700 1 --2- >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f5250097a40 0x7f5250132700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:33:45.069 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:45.057+0000 7f524ffff700 1 --2- >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f52501332e0 0x7f5250133710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3302/0 says I am v2:172.21.15.88:50122/0 (socket says 172.21.15.88:50122) 2023-09-30T01:33:45.069 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:45.057+0000 7f524ffff700 1 -- 172.21.15.88:0/3474700607 learned_addr learned my addr 172.21.15.88:0/3474700607 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-09-30T01:33:45.069 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:45.057+0000 7f524ffff700 10 monclient(hunting): get_auth_request con 0x7f52501332e0 auth_method 0 2023-09-30T01:33:45.069 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:45.057+0000 7f524ffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:33:45.069 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:45.057+0000 7f524ffff700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:33:45.069 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:45.057+0000 7f524ffff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:33:45.070 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:45.057+0000 7f524f7fe700 10 monclient(hunting): get_auth_request con 0x7f5250097a40 auth_method 0 2023-09-30T01:33:45.070 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:45.057+0000 7f524f7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:33:45.070 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:45.057+0000 7f524f7fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:33:45.070 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:45.057+0000 7f524f7fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:33:45.070 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:45.057+0000 7f524ffff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:33:45.070 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:45.057+0000 7f524ffff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:33:45.070 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:45.057+0000 7f524effd700 1 --2- 172.21.15.88:0/3474700607 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7f52501311c0 0x7f5250132c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:33:45.070 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:45.057+0000 7f524ffff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:33:45.070 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:45.057+0000 7f524effd700 10 monclient(hunting): get_auth_request con 0x7f52501311c0 auth_method 0 2023-09-30T01:33:45.070 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:45.057+0000 7f524effd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:33:45.070 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:45.057+0000 7f524effd700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:33:45.070 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:45.057+0000 7f524effd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:33:45.070 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:45.057+0000 7f524f7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:33:45.071 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:45.057+0000 7f524f7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:33:45.071 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:45.057+0000 7f524f7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:33:45.071 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:45.057+0000 7f524effd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:33:45.071 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:45.057+0000 7f524effd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:33:45.071 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:45.057+0000 7f524effd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:33:45.071 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:45.057+0000 7f524ffff700 10 monclient(hunting): handle_auth_done global_id 4432 payload 274 2023-09-30T01:33:45.071 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:45.057+0000 7f524ffff700 1 -- 172.21.15.88:0/3474700607 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f5250097a40 msgr2=0x7f5250132700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:33:45.071 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:45.057+0000 7f524ffff700 1 --2- 172.21.15.88:0/3474700607 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f5250097a40 0x7f5250132700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:45.071 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:45.057+0000 7f524ffff700 1 -- 172.21.15.88:0/3474700607 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7f52501311c0 msgr2=0x7f5250132c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:33:45.071 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:45.057+0000 7f524ffff700 1 --2- 172.21.15.88:0/3474700607 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7f52501311c0 0x7f5250132c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:45.071 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:45.057+0000 7f524ffff700 10 monclient: _finish_hunting 0 2023-09-30T01:33:45.071 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:45.057+0000 7f524ffff700 1 monclient: found mon.noname-c 2023-09-30T01:33:45.072 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:45.057+0000 7f524ffff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:33:45.072 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:45.057+0000 7f524ffff700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.88:3302/0 2023-09-30T01:33:45.072 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:45.057+0000 7f524ffff700 1 -- 172.21.15.88:0/3474700607 --> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f525004bb20 con 0x7f52501332e0 2023-09-30T01:33:45.072 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:45.057+0000 7f524ffff700 1 --2- 172.21.15.88:0/3474700607 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f52501332e0 0x7f5250133710 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f5248009f50 tx=0x7f5248009840 comp rx=0 tx=0).ready entity=mon.4 client_cookie=779470308653026c server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:33:45.072 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:45.057+0000 7f524e7fc700 1 -- 172.21.15.88:0/3474700607 <== mon.4 v2:172.21.15.88:3302/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f524800e070 con 0x7f52501332e0 2023-09-30T01:33:45.072 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:45.057+0000 7f524e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:33:45.072 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:45.057+0000 7f524e7fc700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T01:33:45.072 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:45.057+0000 7f524e7fc700 10 monclient: dump: 2023-09-30T01:33:45.072 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:33:45.072 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:33:45.072 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:45.072 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:45.073 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:33:45.073 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:33:45.073 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:33:45.073 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:33:45.073 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:33:45.073 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:33:45.073 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:33:45.073 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:33:45.073 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:33:45.073 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:33:45.073 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:33:45.073 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:33:45.073 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:45.057+0000 7f524e7fc700 10 monclient: _finish_auth 0 2023-09-30T01:33:45.074 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:45.057+0000 7f524e7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:33:45.074 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:45.057+0000 7f524e7fc700 1 -- 172.21.15.88:0/3474700607 <== mon.4 v2:172.21.15.88:3302/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f524800bdd0 con 0x7f52501332e0 2023-09-30T01:33:45.074 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:45.057+0000 7f5254f2b700 5 monclient: authenticate success, global_id 4432 2023-09-30T01:33:45.074 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:45.057+0000 7f5254f2b700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:33:45.074 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:45.057+0000 7f524e7fc700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:33:45.074 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:45.057+0000 7f5254f2b700 10 monclient: get_monmap_and_config success 2023-09-30T01:33:45.074 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:45.057+0000 7f524e7fc700 1 -- 172.21.15.88:0/3474700607 <== mon.4 v2:172.21.15.88:3302/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f52480092e0 con 0x7f52501332e0 2023-09-30T01:33:45.074 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:45.057+0000 7f524e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:33:45.074 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:45.057+0000 7f5254f2b700 4 set_mon_vals no callback set 2023-09-30T01:33:45.074 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:45.057+0000 7f524e7fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T01:33:45.074 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:45.057+0000 7f524e7fc700 10 monclient: dump: 2023-09-30T01:33:45.074 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:33:45.075 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:33:45.075 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:45.075 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:45.075 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:33:45.075 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:33:45.075 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:33:45.075 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:33:45.075 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:33:45.075 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:33:45.075 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:33:45.075 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:33:45.075 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:33:45.075 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:33:45.076 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:33:45.076 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:33:45.076 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:45.061+0000 7f5254f2b700 10 monclient: shutdown 2023-09-30T01:33:45.076 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:45.061+0000 7f5254f2b700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:33:45.076 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:45.061+0000 7f5254f2b700 1 -- 172.21.15.88:0/3474700607 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f52501332e0 msgr2=0x7f5250133710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:33:45.076 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:45.061+0000 7f5254f2b700 1 --2- 172.21.15.88:0/3474700607 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f52501332e0 0x7f5250133710 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f5248009f50 tx=0x7f5248009840 comp rx=0 tx=0).stop 2023-09-30T01:33:45.076 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:45.061+0000 7f5254f2b700 1 -- 172.21.15.88:0/3474700607 shutdown_connections 2023-09-30T01:33:45.076 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:45.061+0000 7f5254f2b700 1 --2- 172.21.15.88:0/3474700607 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7f52501311c0 0x7f5250132c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:45.076 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:45.061+0000 7f5254f2b700 1 --2- 172.21.15.88:0/3474700607 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f52501332e0 0x7f5250133710 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:45.076 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:45.061+0000 7f5254f2b700 1 --2- 172.21.15.88:0/3474700607 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f5250097a40 0x7f5250132700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:45.076 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:45.061+0000 7f5254f2b700 1 -- 172.21.15.88:0/3474700607 >> 172.21.15.88:0/3474700607 conn(0x7f525006e5e0 msgr2=0x7f525006e9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:33:45.076 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:45.061+0000 7f5254f2b700 1 -- 172.21.15.88:0/3474700607 shutdown_connections 2023-09-30T01:33:45.076 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:45.061+0000 7f5254f2b700 1 -- 172.21.15.88:0/3474700607 wait complete. 2023-09-30T01:33:45.077 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:45.061+0000 7f5254f2b700 10 monclient: build_initial_monmap 2023-09-30T01:33:45.077 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:45.061+0000 7f5254f2b700 10 monclient: monmap: 2023-09-30T01:33:45.077 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2023-09-30T01:33:45.077 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:33:45.077 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 0.000000 2023-09-30T01:33:45.077 INFO:teuthology.orchestra.run.smithi088.stderr:created 0.000000 2023-09-30T01:33:45.077 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2023-09-30T01:33:45.077 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 1 2023-09-30T01:33:45.077 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.noname-a 2023-09-30T01:33:45.077 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.noname-f 2023-09-30T01:33:45.077 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.noname-b 2023-09-30T01:33:45.077 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.noname-g 2023-09-30T01:33:45.077 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.noname-c 2023-09-30T01:33:45.078 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.noname-h 2023-09-30T01:33:45.078 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.noname-d 2023-09-30T01:33:45.078 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.noname-i 2023-09-30T01:33:45.078 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.noname-e 2023-09-30T01:33:45.078 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:33:45.082 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:45.061+0000 7f5254f2b700 1 Processor -- start 2023-09-30T01:33:45.082 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:45.061+0000 7f5254f2b700 1 -- start start 2023-09-30T01:33:45.082 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:45.061+0000 7f5254f2b700 10 monclient: init 2023-09-30T01:33:45.082 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:45.065+0000 7f5254f2b700 10 monclient: _reopen_session rank -1 2023-09-30T01:33:45.082 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:45.065+0000 7f5254f2b700 10 monclient: _add_conns ranks=[8,2,4,0,7,6,5,3,1] 2023-09-30T01:33:45.082 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:45.065+0000 7f5254f2b700 1 --2- >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f5250097a40 0x7f525011cf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:33:45.082 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:45.065+0000 7f5254f2b700 10 monclient(hunting): picked mon.noname-e con 0x7f5250097a40 addr [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] 2023-09-30T01:33:45.083 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:45.065+0000 7f5254f2b700 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f52501311c0 0x7f525011d4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:33:45.083 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:45.065+0000 7f5254f2b700 10 monclient(hunting): picked mon.noname-b con 0x7f52501311c0 addr [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] 2023-09-30T01:33:45.083 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:45.065+0000 7f5254f2b700 1 --2- >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f52501332e0 0x7f525011da00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:33:45.083 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:45.065+0000 7f5254f2b700 10 monclient(hunting): picked mon.noname-c con 0x7f52501332e0 addr [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] 2023-09-30T01:33:45.083 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:45.065+0000 7f5254f2b700 10 monclient(hunting): start opening mon connection 2023-09-30T01:33:45.083 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:45.065+0000 7f5254f2b700 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5250099c30 con 0x7f52501311c0 2023-09-30T01:33:45.083 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:45.065+0000 7f5254f2b700 10 monclient(hunting): start opening mon connection 2023-09-30T01:33:45.083 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:45.065+0000 7f5254f2b700 1 -- --> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] -- mon_getmap magic: 0 v1 -- 0x7f52500996b0 con 0x7f52501332e0 2023-09-30T01:33:45.083 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:45.065+0000 7f5254f2b700 10 monclient(hunting): start opening mon connection 2023-09-30T01:33:45.083 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:45.065+0000 7f5254f2b700 1 -- --> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] -- mon_getmap magic: 0 v1 -- 0x7f5250121160 con 0x7f5250097a40 2023-09-30T01:33:45.083 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:45.065+0000 7f5254f2b700 10 monclient(hunting): _renew_subs 2023-09-30T01:33:45.083 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:45.065+0000 7f5254f2b700 10 monclient(hunting): authenticate will time out at 897.104635s 2023-09-30T01:33:45.084 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:45.065+0000 7f524effd700 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f52501311c0 0x7f525011d4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:33:45.084 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:45.065+0000 7f524f7fe700 1 --2- >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f5250097a40 0x7f525011cf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:33:45.084 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:45.065+0000 7f524f7fe700 1 --2- >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f5250097a40 0x7f525011cf80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3304/0 says I am v2:172.21.15.88:43146/0 (socket says 172.21.15.88:43146) 2023-09-30T01:33:45.084 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:45.065+0000 7f524effd700 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f52501311c0 0x7f525011d4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3301/0 says I am v2:172.21.15.88:42258/0 (socket says 172.21.15.88:42258) 2023-09-30T01:33:45.084 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:45.065+0000 7f524ffff700 1 --2- >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f52501332e0 0x7f525011da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:33:45.084 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:45.065+0000 7f524f7fe700 1 -- 172.21.15.88:0/846699708 learned_addr learned my addr 172.21.15.88:0/846699708 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-09-30T01:33:45.084 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:45.065+0000 7f524f7fe700 10 monclient(hunting): get_auth_request con 0x7f5250097a40 auth_method 0 2023-09-30T01:33:45.084 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:45.065+0000 7f524f7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:33:45.084 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:45.065+0000 7f524f7fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:33:45.084 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:45.065+0000 7f524f7fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:33:45.084 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:45.065+0000 7f524effd700 10 monclient(hunting): get_auth_request con 0x7f52501311c0 auth_method 0 2023-09-30T01:33:45.084 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:45.065+0000 7f524effd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:33:45.085 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:45.065+0000 7f524effd700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:33:45.085 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:45.065+0000 7f524effd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:33:45.085 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:45.065+0000 7f524ffff700 10 monclient(hunting): get_auth_request con 0x7f52501332e0 auth_method 0 2023-09-30T01:33:45.085 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:45.065+0000 7f524ffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:33:45.085 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:45.065+0000 7f524ffff700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:33:45.085 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:45.065+0000 7f524ffff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:33:45.085 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:45.065+0000 7f524effd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:33:45.085 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:45.065+0000 7f524effd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:33:45.085 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:45.065+0000 7f524effd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:33:45.085 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:45.065+0000 7f524ffff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:33:45.085 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:45.065+0000 7f524ffff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:33:45.085 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:45.065+0000 7f524ffff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:33:45.085 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:45.065+0000 7f524f7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:33:45.086 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:45.065+0000 7f524f7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:33:45.086 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:45.065+0000 7f524f7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:33:45.086 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:45.065+0000 7f524effd700 10 monclient(hunting): handle_auth_done global_id 4412 payload 824 2023-09-30T01:33:45.086 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:45.065+0000 7f524effd700 1 -- 172.21.15.88:0/846699708 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f5250097a40 msgr2=0x7f525011cf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:33:45.086 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:45.065+0000 7f524effd700 1 --2- 172.21.15.88:0/846699708 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f5250097a40 0x7f525011cf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:45.086 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:45.065+0000 7f524effd700 1 -- 172.21.15.88:0/846699708 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f52501332e0 msgr2=0x7f525011da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:33:45.086 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:45.065+0000 7f524effd700 1 --2- 172.21.15.88:0/846699708 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f52501332e0 0x7f525011da00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:45.086 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:45.065+0000 7f524effd700 10 monclient: _finish_hunting 0 2023-09-30T01:33:45.086 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:45.065+0000 7f524effd700 1 monclient: found mon.noname-b 2023-09-30T01:33:45.086 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:45.065+0000 7f524effd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:33:45.086 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:45.065+0000 7f524effd700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.88:3301/0 2023-09-30T01:33:45.086 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:45.065+0000 7f524effd700 1 -- 172.21.15.88:0/846699708 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5248009ae0 con 0x7f52501311c0 2023-09-30T01:33:45.087 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:45.065+0000 7f524ffff700 1 --2- 172.21.15.88:0/846699708 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f52501332e0 0x7f525011da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:33:45.087 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:45.065+0000 7f524f7fe700 1 --2- 172.21.15.88:0/846699708 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f5250097a40 0x7f525011cf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:33:45.087 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:45.065+0000 7f524effd700 1 --2- 172.21.15.88:0/846699708 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f52501311c0 0x7f525011d4c0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f524000ea50 tx=0x7f52400091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:33:45.087 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:45.065+0000 7f52377fe700 1 -- 172.21.15.88:0/846699708 <== mon.2 v2:172.21.15.88:3301/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f52400095d0 con 0x7f52501311c0 2023-09-30T01:33:45.087 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:45.065+0000 7f52377fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:33:45.087 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:45.065+0000 7f52377fe700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T01:33:45.087 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:45.065+0000 7f52377fe700 10 monclient: dump: 2023-09-30T01:33:45.087 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:33:45.087 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:33:45.087 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:45.087 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:45.087 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:33:45.088 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:33:45.088 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:33:45.088 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:33:45.088 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:33:45.088 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:33:45.088 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:33:45.088 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:33:45.088 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:33:45.088 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:33:45.088 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:33:45.088 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:33:45.088 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:45.065+0000 7f52377fe700 10 monclient: _finish_auth 0 2023-09-30T01:33:45.088 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:45.065+0000 7f52377fe700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:33:45.089 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:45.065+0000 7f52377fe700 1 -- 172.21.15.88:0/846699708 <== mon.2 v2:172.21.15.88:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f524000cc70 con 0x7f52501311c0 2023-09-30T01:33:45.089 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:45.065+0000 7f5254f2b700 5 monclient: authenticate success, global_id 4412 2023-09-30T01:33:45.089 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:45.065+0000 7f52377fe700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:33:45.089 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:45.065+0000 7f5254f2b700 10 monclient: _renew_subs 2023-09-30T01:33:45.089 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:45.065+0000 7f52377fe700 1 -- 172.21.15.88:0/846699708 <== mon.2 v2:172.21.15.88:3301/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f524000cde0 con 0x7f52501311c0 2023-09-30T01:33:45.089 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:45.065+0000 7f5254f2b700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.88:3301/0 2023-09-30T01:33:45.089 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:45.065+0000 7f5254f2b700 1 -- 172.21.15.88:0/846699708 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f525011fc30 con 0x7f52501311c0 2023-09-30T01:33:45.089 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:45.065+0000 7f52377fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:33:45.089 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:45.065+0000 7f524dffb700 4 set_mon_vals no callback set 2023-09-30T01:33:45.089 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:45.065+0000 7f52377fe700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T01:33:45.089 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:45.065+0000 7f52377fe700 10 monclient: dump: 2023-09-30T01:33:45.089 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:33:45.089 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:33:45.090 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:45.090 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:45.090 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:33:45.090 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:33:45.090 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:33:45.090 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:33:45.090 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:33:45.090 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:33:45.090 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:33:45.090 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:33:45.090 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:33:45.090 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:33:45.090 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:33:45.091 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:33:45.091 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:45.065+0000 7f5254f2b700 10 monclient: _renew_subs 2023-09-30T01:33:45.091 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:45.065+0000 7f5254f2b700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.88:3301/0 2023-09-30T01:33:45.091 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:45.065+0000 7f5254f2b700 1 -- 172.21.15.88:0/846699708 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f525011e3a0 con 0x7f52501311c0 2023-09-30T01:33:45.091 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:45.065+0000 7f52377fe700 1 -- 172.21.15.88:0/846699708 <== mon.2 v2:172.21.15.88:3301/0 4 ==== mgrmap(e 5) v1 ==== 40144+0+0 (secure 0 0 0) 0x7f524000c7f0 con 0x7f52501311c0 2023-09-30T01:33:45.091 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:45.069+0000 7f52377fe700 1 --2- 172.21.15.88:0/846699708 >> v2:172.21.15.107:6812/29926 conn(0x7f523c033360 0x7f523c035810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:33:45.091 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:45.069+0000 7f5254f2b700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:33:45.091 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:45.069+0000 7f5254f2b700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:33:45.091 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:45.069+0000 7f5254f2b700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.88:3301/0 2023-09-30T01:33:45.091 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:45.069+0000 7f52377fe700 1 -- 172.21.15.88:0/846699708 <== mon.2 v2:172.21.15.88:3301/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 3502+0+0 (secure 0 0 0) 0x7f5240043900 con 0x7f52501311c0 2023-09-30T01:33:45.091 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:45.069+0000 7f5254f2b700 1 -- 172.21.15.88:0/846699708 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f525009e7b0 con 0x7f52501311c0 2023-09-30T01:33:45.091 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:45.069+0000 7f524f7fe700 1 --2- 172.21.15.88:0/846699708 >> v2:172.21.15.107:6812/29926 conn(0x7f523c033360 0x7f523c035810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:33:45.092 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:45.069+0000 7f524f7fe700 10 monclient: get_auth_request con 0x7f523c033360 auth_method 0 2023-09-30T01:33:45.092 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:45.069+0000 7f524f7fe700 1 --2- 172.21.15.88:0/846699708 >> v2:172.21.15.107:6812/29926 conn(0x7f523c033360 0x7f523c035810 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f5244009fd0 tx=0x7f52440091a0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:33:45.092 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:45.073+0000 7f52377fe700 1 -- 172.21.15.88:0/846699708 <== mon.2 v2:172.21.15.88:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f524000caa0 con 0x7f52501311c0 2023-09-30T01:33:45.092 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:45.073+0000 7f52377fe700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:33:45.092 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:45.073+0000 7f52377fe700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:33:45.269 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:45.265+0000 7f5254f2b700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T01:33:45.269 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:45.265+0000 7f5254f2b700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:33:45.269 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:45.265+0000 7f5254f2b700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.88:3301/0 2023-09-30T01:33:45.269 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:45.265+0000 7f5254f2b700 1 -- 172.21.15.88:0/846699708 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f5250071960 con 0x7f52501311c0 2023-09-30T01:33:45.270 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:45.265+0000 7f52377fe700 1 -- 172.21.15.88:0/846699708 <== mon.2 v2:172.21.15.88:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f524002a080 con 0x7f52501311c0 2023-09-30T01:33:45.270 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:45.265+0000 7f52377fe700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:33:45.270 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:45.265+0000 7f52377fe700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:33:45.270 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-09-30T01:33:45.270 INFO:teuthology.orchestra.run.smithi088.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:33:45.272 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:45.269+0000 7f5254f2b700 1 -- 172.21.15.88:0/846699708 >> v2:172.21.15.107:6812/29926 conn(0x7f523c033360 msgr2=0x7f523c035810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:33:45.272 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:45.269+0000 7f5254f2b700 1 --2- 172.21.15.88:0/846699708 >> v2:172.21.15.107:6812/29926 conn(0x7f523c033360 0x7f523c035810 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f5244009fd0 tx=0x7f52440091a0 comp rx=0 tx=0).stop 2023-09-30T01:33:45.272 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:45.269+0000 7f5254f2b700 10 monclient: shutdown 2023-09-30T01:33:45.272 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:45.269+0000 7f5254f2b700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:33:45.272 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:45.269+0000 7f5254f2b700 1 -- 172.21.15.88:0/846699708 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f52501311c0 msgr2=0x7f525011d4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:33:45.272 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:45.269+0000 7f5254f2b700 1 --2- 172.21.15.88:0/846699708 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f52501311c0 0x7f525011d4c0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f524000ea50 tx=0x7f52400091a0 comp rx=0 tx=0).stop 2023-09-30T01:33:45.273 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:45.269+0000 7f5254f2b700 1 -- 172.21.15.88:0/846699708 shutdown_connections 2023-09-30T01:33:45.273 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:45.269+0000 7f5254f2b700 1 --2- 172.21.15.88:0/846699708 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f52501311c0 0x7f525011d4c0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:45.273 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:45.269+0000 7f5254f2b700 1 --2- 172.21.15.88:0/846699708 >> v2:172.21.15.107:6812/29926 conn(0x7f523c033360 0x7f523c035810 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:45.273 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:45.269+0000 7f5254f2b700 1 --2- 172.21.15.88:0/846699708 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f52501332e0 0x7f525011da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:45.273 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:45.269+0000 7f5254f2b700 1 --2- 172.21.15.88:0/846699708 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f5250097a40 0x7f525011cf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:45.273 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:45.269+0000 7f5254f2b700 1 -- 172.21.15.88:0/846699708 >> 172.21.15.88:0/846699708 conn(0x7f525006e5e0 msgr2=0x7f52500a5170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:33:45.273 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:45.269+0000 7f5254f2b700 1 -- 172.21.15.88:0/846699708 shutdown_connections 2023-09-30T01:33:45.273 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:45.269+0000 7f5254f2b700 1 -- 172.21.15.88:0/846699708 wait complete. 2023-09-30T01:33:48.284 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:33:48.399 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:48.393+0000 7fdb7f59e700 10 monclient: get_monmap_and_config 2023-09-30T01:33:48.399 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:48.397+0000 7fdb7f59e700 10 monclient: build_initial_monmap 2023-09-30T01:33:48.399 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:48.397+0000 7fdb7f59e700 10 monclient: monmap: 2023-09-30T01:33:48.399 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2023-09-30T01:33:48.400 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:33:48.400 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:33:48.399938+0000 2023-09-30T01:33:48.400 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:33:48.399938+0000 2023-09-30T01:33:48.400 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2023-09-30T01:33:48.400 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:33:48.400 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a-legacy 2023-09-30T01:33:48.400 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.107:6789/0 mon.noname-f-legacy 2023-09-30T01:33:48.400 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.noname-b 2023-09-30T01:33:48.400 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.noname-g 2023-09-30T01:33:48.400 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.noname-c 2023-09-30T01:33:48.400 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.noname-h 2023-09-30T01:33:48.400 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.noname-d 2023-09-30T01:33:48.401 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.noname-i 2023-09-30T01:33:48.401 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.noname-e 2023-09-30T01:33:48.401 INFO:teuthology.orchestra.run.smithi088.stderr:9: v2:172.21.15.88:3300/0 mon.noname-a 2023-09-30T01:33:48.401 INFO:teuthology.orchestra.run.smithi088.stderr:10: v2:172.21.15.107:3300/0 mon.noname-f 2023-09-30T01:33:48.401 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:33:48.403 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:48.397+0000 7fdb7f59e700 1 Processor -- start 2023-09-30T01:33:48.404 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:48.397+0000 7fdb7f59e700 1 -- start start 2023-09-30T01:33:48.404 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:48.397+0000 7fdb7f59e700 10 monclient: init 2023-09-30T01:33:48.404 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:48.397+0000 7fdb7f59e700 10 monclient: _reopen_session rank -1 2023-09-30T01:33:48.404 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:48.397+0000 7fdb7f59e700 10 monclient: _add_conns ranks=[8,2,6,9,5,0,1,4,3,10,7] 2023-09-30T01:33:48.404 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:48.397+0000 7fdb7f59e700 1 --2- >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7fdb800966a0 0x7fdb80096a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:33:48.404 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:48.397+0000 7fdb7f59e700 10 monclient(hunting): picked mon.noname-e con 0x7fdb800966a0 addr [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] 2023-09-30T01:33:48.404 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:48.397+0000 7fdb7f59e700 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fdb80097cd0 0x7fdb800980a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:33:48.404 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:48.397+0000 7fdb7f59e700 10 monclient(hunting): picked mon.noname-b con 0x7fdb80097cd0 addr [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] 2023-09-30T01:33:48.404 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:48.397+0000 7fdb7f59e700 1 --2- >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7fdb8009ed80 0x7fdb8009f1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:33:48.404 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:48.397+0000 7fdb7f59e700 10 monclient(hunting): picked mon.noname-d con 0x7fdb8009ed80 addr [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] 2023-09-30T01:33:48.405 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:48.397+0000 7fdb7f59e700 10 monclient(hunting): start opening mon connection 2023-09-30T01:33:48.405 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:48.397+0000 7fdb7f59e700 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb80097920 con 0x7fdb80097cd0 2023-09-30T01:33:48.405 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:48.397+0000 7fdb7f59e700 10 monclient(hunting): start opening mon connection 2023-09-30T01:33:48.405 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:48.397+0000 7fdb7f59e700 1 -- --> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] -- mon_getmap magic: 0 v1 -- 0x7fdb8009a340 con 0x7fdb8009ed80 2023-09-30T01:33:48.405 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:48.397+0000 7fdb7f59e700 10 monclient(hunting): start opening mon connection 2023-09-30T01:33:48.405 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:48.397+0000 7fdb7f59e700 1 -- --> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] -- mon_getmap magic: 0 v1 -- 0x7fdb800975d0 con 0x7fdb800966a0 2023-09-30T01:33:48.405 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:48.397+0000 7fdb7f59e700 10 monclient(hunting): _renew_subs 2023-09-30T01:33:48.405 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:48.397+0000 7fdb7f59e700 10 monclient(hunting): authenticate will time out at 900.437427s 2023-09-30T01:33:48.405 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:48.397+0000 7fdb7dd9b700 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fdb80097cd0 0x7fdb800980a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:33:48.405 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:48.397+0000 7fdb7e59c700 1 --2- >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7fdb800966a0 0x7fdb80096a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:33:48.405 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:48.397+0000 7fdb7ed9d700 1 --2- >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7fdb8009ed80 0x7fdb8009f1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:33:48.405 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:48.397+0000 7fdb7dd9b700 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fdb80097cd0 0x7fdb800980a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3301/0 says I am v2:172.21.15.88:42260/0 (socket says 172.21.15.88:42260) 2023-09-30T01:33:48.406 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:48.397+0000 7fdb7dd9b700 1 -- 172.21.15.88:0/1565188173 learned_addr learned my addr 172.21.15.88:0/1565188173 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-09-30T01:33:48.406 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:48.397+0000 7fdb7dd9b700 10 monclient(hunting): get_auth_request con 0x7fdb80097cd0 auth_method 0 2023-09-30T01:33:48.406 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:48.397+0000 7fdb7dd9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:33:48.406 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:48.397+0000 7fdb7dd9b700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:33:48.406 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:48.397+0000 7fdb7dd9b700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:33:48.406 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:48.397+0000 7fdb7e59c700 10 monclient(hunting): get_auth_request con 0x7fdb800966a0 auth_method 0 2023-09-30T01:33:48.406 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:48.397+0000 7fdb7e59c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:33:48.406 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:48.397+0000 7fdb7e59c700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:33:48.406 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:48.397+0000 7fdb7e59c700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:33:48.406 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:48.397+0000 7fdb7ed9d700 10 monclient(hunting): get_auth_request con 0x7fdb8009ed80 auth_method 0 2023-09-30T01:33:48.406 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:48.397+0000 7fdb7ed9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:33:48.406 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:48.397+0000 7fdb7ed9d700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:33:48.407 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:48.397+0000 7fdb7ed9d700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:33:48.407 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:48.397+0000 7fdb7e59c700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:33:48.407 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:48.397+0000 7fdb7e59c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:33:48.407 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:48.397+0000 7fdb7e59c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:33:48.407 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:48.397+0000 7fdb7dd9b700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:33:48.407 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:48.397+0000 7fdb7dd9b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:33:48.407 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:48.397+0000 7fdb7dd9b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:33:48.407 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:48.397+0000 7fdb7ed9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:33:48.407 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:48.397+0000 7fdb7ed9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:33:48.407 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:48.397+0000 7fdb7ed9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:33:48.407 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:48.397+0000 7fdb7e59c700 10 monclient(hunting): handle_auth_done global_id 4481 payload 274 2023-09-30T01:33:48.407 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:48.397+0000 7fdb7e59c700 1 -- 172.21.15.88:0/1565188173 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7fdb8009ed80 msgr2=0x7fdb8009f1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:33:48.408 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:48.397+0000 7fdb7e59c700 1 --2- 172.21.15.88:0/1565188173 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7fdb8009ed80 0x7fdb8009f1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:48.408 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:48.397+0000 7fdb7e59c700 1 -- 172.21.15.88:0/1565188173 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fdb80097cd0 msgr2=0x7fdb800980a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:33:48.408 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:48.397+0000 7fdb7e59c700 1 --2- 172.21.15.88:0/1565188173 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fdb80097cd0 0x7fdb800980a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:48.408 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:48.397+0000 7fdb7e59c700 10 monclient: _finish_hunting 0 2023-09-30T01:33:48.408 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:48.397+0000 7fdb7e59c700 1 monclient: found mon.noname-e 2023-09-30T01:33:48.408 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:48.397+0000 7fdb7e59c700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:33:48.408 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:48.397+0000 7fdb7e59c700 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.88:3304/0 2023-09-30T01:33:48.408 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:48.397+0000 7fdb7e59c700 1 -- 172.21.15.88:0/1565188173 --> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb8004bb20 con 0x7fdb800966a0 2023-09-30T01:33:48.408 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:48.397+0000 7fdb7e59c700 1 --2- 172.21.15.88:0/1565188173 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7fdb800966a0 0x7fdb80096a70 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fdb78009930 tx=0x7fdb78009120 comp rx=0 tx=0).ready entity=mon.8 client_cookie=c0d4033abaf334b1 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:33:48.408 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:48.397+0000 7fdb7d59a700 1 -- 172.21.15.88:0/1565188173 <== mon.8 v2:172.21.15.88:3304/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7fdb78005c10 con 0x7fdb800966a0 2023-09-30T01:33:48.408 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:48.397+0000 7fdb7d59a700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:33:48.408 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:48.397+0000 7fdb7d59a700 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2023-09-30T01:33:48.409 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:48.397+0000 7fdb7d59a700 10 monclient: dump: 2023-09-30T01:33:48.409 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:33:48.409 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:33:48.409 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:48.409 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:48.409 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:33:48.409 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:33:48.409 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:33:48.409 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:33:48.409 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:33:48.409 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:33:48.409 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:33:48.410 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:33:48.410 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:33:48.410 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:33:48.410 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:33:48.410 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:33:48.410 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:48.397+0000 7fdb7d59a700 10 monclient: _finish_auth 0 2023-09-30T01:33:48.410 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:48.397+0000 7fdb7d59a700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:33:48.410 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:48.397+0000 7fdb7d59a700 1 -- 172.21.15.88:0/1565188173 <== mon.8 v2:172.21.15.88:3304/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb78005d90 con 0x7fdb800966a0 2023-09-30T01:33:48.410 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:48.397+0000 7fdb7f59e700 5 monclient: authenticate success, global_id 4481 2023-09-30T01:33:48.410 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:48.397+0000 7fdb7f59e700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:33:48.410 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:48.397+0000 7fdb7d59a700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:33:48.410 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:48.397+0000 7fdb7f59e700 10 monclient: get_monmap_and_config success 2023-09-30T01:33:48.411 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:48.397+0000 7fdb7d59a700 1 -- 172.21.15.88:0/1565188173 <== mon.8 v2:172.21.15.88:3304/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7fdb78011420 con 0x7fdb800966a0 2023-09-30T01:33:48.411 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:48.397+0000 7fdb7f59e700 4 set_mon_vals no callback set 2023-09-30T01:33:48.411 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:48.397+0000 7fdb7d59a700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:33:48.411 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:48.397+0000 7fdb7d59a700 10 monclient: got monmap 1 from mon.e (according to old e1) 2023-09-30T01:33:48.411 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:48.397+0000 7fdb7d59a700 10 monclient: dump: 2023-09-30T01:33:48.411 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:33:48.411 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:33:48.411 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:48.411 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:48.411 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:33:48.411 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:33:48.411 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:33:48.411 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:33:48.412 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:33:48.412 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:33:48.412 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:33:48.412 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:33:48.412 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:33:48.412 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:33:48.412 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:33:48.412 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:33:48.412 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:48.397+0000 7fdb7f59e700 10 monclient: shutdown 2023-09-30T01:33:48.412 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:48.397+0000 7fdb7f59e700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:33:48.412 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:48.397+0000 7fdb7f59e700 1 -- 172.21.15.88:0/1565188173 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7fdb800966a0 msgr2=0x7fdb80096a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:33:48.412 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:48.397+0000 7fdb7f59e700 1 --2- 172.21.15.88:0/1565188173 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7fdb800966a0 0x7fdb80096a70 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fdb78009930 tx=0x7fdb78009120 comp rx=0 tx=0).stop 2023-09-30T01:33:48.413 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:48.397+0000 7fdb7f59e700 1 -- 172.21.15.88:0/1565188173 shutdown_connections 2023-09-30T01:33:48.413 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:48.397+0000 7fdb7f59e700 1 --2- 172.21.15.88:0/1565188173 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7fdb8009ed80 0x7fdb8009f1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:48.413 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:48.397+0000 7fdb7f59e700 1 --2- 172.21.15.88:0/1565188173 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fdb80097cd0 0x7fdb800980a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:48.413 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:48.397+0000 7fdb7f59e700 1 --2- 172.21.15.88:0/1565188173 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7fdb800966a0 0x7fdb80096a70 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:48.413 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:48.397+0000 7fdb7f59e700 1 -- 172.21.15.88:0/1565188173 >> 172.21.15.88:0/1565188173 conn(0x7fdb800a0910 msgr2=0x7fdb800a33c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:33:48.413 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:48.397+0000 7fdb7f59e700 1 -- 172.21.15.88:0/1565188173 shutdown_connections 2023-09-30T01:33:48.413 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:48.397+0000 7fdb7f59e700 1 -- 172.21.15.88:0/1565188173 wait complete. 2023-09-30T01:33:48.413 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:48.401+0000 7fdb7f59e700 10 monclient: build_initial_monmap 2023-09-30T01:33:48.413 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:48.401+0000 7fdb7f59e700 10 monclient: monmap: 2023-09-30T01:33:48.413 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2023-09-30T01:33:48.413 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:33:48.413 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 0.000000 2023-09-30T01:33:48.414 INFO:teuthology.orchestra.run.smithi088.stderr:created 0.000000 2023-09-30T01:33:48.414 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2023-09-30T01:33:48.414 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 1 2023-09-30T01:33:48.414 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.noname-a 2023-09-30T01:33:48.414 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.noname-f 2023-09-30T01:33:48.414 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.noname-b 2023-09-30T01:33:48.414 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.noname-g 2023-09-30T01:33:48.414 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.noname-c 2023-09-30T01:33:48.414 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.noname-h 2023-09-30T01:33:48.414 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.noname-d 2023-09-30T01:33:48.414 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.noname-i 2023-09-30T01:33:48.414 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.noname-e 2023-09-30T01:33:48.414 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:33:48.418 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:48.401+0000 7fdb7f59e700 1 Processor -- start 2023-09-30T01:33:48.418 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:48.401+0000 7fdb7f59e700 1 -- start start 2023-09-30T01:33:48.418 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:48.401+0000 7fdb7f59e700 10 monclient: init 2023-09-30T01:33:48.418 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:48.401+0000 7fdb7f59e700 10 monclient: _reopen_session rank -1 2023-09-30T01:33:48.419 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:48.401+0000 7fdb7f59e700 10 monclient: _add_conns ranks=[0,5,3,6,8,2,7,1,4] 2023-09-30T01:33:48.419 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:48.401+0000 7fdb7f59e700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdb800966a0 0x7fdb80142690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:33:48.419 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:48.401+0000 7fdb7f59e700 10 monclient(hunting): picked mon.noname-a con 0x7fdb800966a0 addr [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] 2023-09-30T01:33:48.419 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:48.401+0000 7fdb7f59e700 1 --2- >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7fdb80097cd0 0x7fdb80142bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:33:48.419 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:48.401+0000 7fdb7f59e700 10 monclient(hunting): picked mon.noname-h con 0x7fdb80097cd0 addr [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] 2023-09-30T01:33:48.419 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:48.401+0000 7fdb7f59e700 1 --2- >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7fdb8009ed80 0x7fdb80143110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:33:48.419 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:48.401+0000 7fdb7f59e700 10 monclient(hunting): picked mon.noname-g con 0x7fdb8009ed80 addr [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] 2023-09-30T01:33:48.419 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:48.401+0000 7fdb7f59e700 10 monclient(hunting): start opening mon connection 2023-09-30T01:33:48.419 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:48.401+0000 7fdb7f59e700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb80090660 con 0x7fdb800966a0 2023-09-30T01:33:48.419 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:48.401+0000 7fdb7f59e700 10 monclient(hunting): start opening mon connection 2023-09-30T01:33:48.419 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:48.401+0000 7fdb7f59e700 1 -- --> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb800996c0 con 0x7fdb8009ed80 2023-09-30T01:33:48.419 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:48.401+0000 7fdb7f59e700 10 monclient(hunting): start opening mon connection 2023-09-30T01:33:48.420 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:48.401+0000 7fdb7f59e700 1 -- --> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] -- mon_getmap magic: 0 v1 -- 0x7fdb80063630 con 0x7fdb80097cd0 2023-09-30T01:33:48.420 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:48.401+0000 7fdb7f59e700 10 monclient(hunting): _renew_subs 2023-09-30T01:33:48.420 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:48.401+0000 7fdb7f59e700 10 monclient(hunting): authenticate will time out at 900.442940s 2023-09-30T01:33:48.420 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:48.401+0000 7fdb7e59c700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdb800966a0 0x7fdb80142690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:33:48.420 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:48.401+0000 7fdb7e59c700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdb800966a0 0x7fdb80142690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:53412/0 (socket says 172.21.15.88:53412) 2023-09-30T01:33:48.420 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:48.401+0000 7fdb7e59c700 1 -- 172.21.15.88:0/2002853254 learned_addr learned my addr 172.21.15.88:0/2002853254 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-09-30T01:33:48.420 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:48.401+0000 7fdb7e59c700 10 monclient(hunting): get_auth_request con 0x7fdb800966a0 auth_method 0 2023-09-30T01:33:48.420 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:48.401+0000 7fdb7e59c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:33:48.420 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:48.401+0000 7fdb7e59c700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:33:48.420 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:48.401+0000 7fdb7e59c700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:33:48.420 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:48.401+0000 7fdb7e59c700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:33:48.420 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:48.401+0000 7fdb7e59c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:33:48.421 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:48.401+0000 7fdb7e59c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:33:48.421 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:48.401+0000 7fdb7dd9b700 1 --2- 172.21.15.88:0/2002853254 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7fdb80097cd0 0x7fdb80142bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:33:48.421 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:48.401+0000 7fdb7ed9d700 1 --2- 172.21.15.88:0/2002853254 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7fdb8009ed80 0x7fdb80143110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:33:48.421 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:48.401+0000 7fdb7ed9d700 10 monclient(hunting): get_auth_request con 0x7fdb8009ed80 auth_method 0 2023-09-30T01:33:48.421 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:48.401+0000 7fdb7ed9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:33:48.421 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:48.401+0000 7fdb7ed9d700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:33:48.421 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:48.401+0000 7fdb7ed9d700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:33:48.421 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:48.401+0000 7fdb7dd9b700 10 monclient(hunting): get_auth_request con 0x7fdb80097cd0 auth_method 0 2023-09-30T01:33:48.421 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:48.401+0000 7fdb7dd9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:33:48.421 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:48.401+0000 7fdb7dd9b700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:33:48.421 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:48.401+0000 7fdb7dd9b700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:33:48.421 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:48.401+0000 7fdb7e59c700 10 monclient(hunting): handle_auth_done global_id 4509 payload 824 2023-09-30T01:33:48.422 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:48.401+0000 7fdb7e59c700 1 -- 172.21.15.88:0/2002853254 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7fdb80097cd0 msgr2=0x7fdb80142bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:33:48.422 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:48.401+0000 7fdb7e59c700 1 --2- 172.21.15.88:0/2002853254 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7fdb80097cd0 0x7fdb80142bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:48.422 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:48.401+0000 7fdb7e59c700 1 -- 172.21.15.88:0/2002853254 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7fdb8009ed80 msgr2=0x7fdb80143110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:33:48.422 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:48.401+0000 7fdb7e59c700 1 --2- 172.21.15.88:0/2002853254 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7fdb8009ed80 0x7fdb80143110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:48.422 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:48.401+0000 7fdb7e59c700 10 monclient: _finish_hunting 0 2023-09-30T01:33:48.422 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:48.401+0000 7fdb7e59c700 1 monclient: found mon.noname-a 2023-09-30T01:33:48.422 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:48.401+0000 7fdb7e59c700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:33:48.422 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:48.401+0000 7fdb7e59c700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.88:3300/0 2023-09-30T01:33:48.422 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:48.401+0000 7fdb7e59c700 1 -- 172.21.15.88:0/2002853254 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb780093c0 con 0x7fdb800966a0 2023-09-30T01:33:48.422 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:48.401+0000 7fdb7e59c700 1 --2- 172.21.15.88:0/2002853254 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdb800966a0 0x7fdb80142690 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fdb78012000 tx=0x7fdb78013000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:33:48.422 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:48.401+0000 7fdb767fc700 1 -- 172.21.15.88:0/2002853254 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7fdb7801dbb0 con 0x7fdb800966a0 2023-09-30T01:33:48.422 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:48.401+0000 7fdb767fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:33:48.423 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:48.401+0000 7fdb767fc700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T01:33:48.423 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:48.401+0000 7fdb767fc700 10 monclient: dump: 2023-09-30T01:33:48.423 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:33:48.423 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:33:48.423 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:48.423 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:48.423 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:33:48.423 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:33:48.423 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:33:48.423 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:33:48.423 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:33:48.423 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:33:48.424 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:33:48.424 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:33:48.424 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:33:48.424 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:33:48.424 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:33:48.424 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:33:48.424 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:48.405+0000 7fdb767fc700 10 monclient: _finish_auth 0 2023-09-30T01:33:48.424 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:48.405+0000 7fdb767fc700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:33:48.424 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:48.405+0000 7fdb7f59e700 5 monclient: authenticate success, global_id 4509 2023-09-30T01:33:48.424 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:48.405+0000 7fdb767fc700 1 -- 172.21.15.88:0/2002853254 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb7801dd90 con 0x7fdb800966a0 2023-09-30T01:33:48.424 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:48.405+0000 7fdb7f59e700 10 monclient: _renew_subs 2023-09-30T01:33:48.424 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:48.405+0000 7fdb7f59e700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.88:3300/0 2023-09-30T01:33:48.425 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:48.405+0000 7fdb7f59e700 1 -- 172.21.15.88:0/2002853254 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdb80091c70 con 0x7fdb800966a0 2023-09-30T01:33:48.425 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:48.405+0000 7fdb767fc700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:33:48.425 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:48.405+0000 7fdb767fc700 1 -- 172.21.15.88:0/2002853254 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7fdb7801b900 con 0x7fdb800966a0 2023-09-30T01:33:48.425 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:48.405+0000 7fdb767fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:33:48.425 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:48.405+0000 7fdb7cd99700 4 set_mon_vals no callback set 2023-09-30T01:33:48.425 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:48.405+0000 7fdb767fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:33:48.425 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:48.405+0000 7fdb767fc700 10 monclient: dump: 2023-09-30T01:33:48.425 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:33:48.425 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:33:48.425 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:48.425 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:48.425 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:33:48.425 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:33:48.426 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:33:48.426 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:33:48.426 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:33:48.426 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:33:48.426 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:33:48.426 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:33:48.426 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:33:48.426 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:33:48.426 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:33:48.426 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:33:48.426 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:48.405+0000 7fdb7f59e700 10 monclient: _renew_subs 2023-09-30T01:33:48.427 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:48.405+0000 7fdb7f59e700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.88:3300/0 2023-09-30T01:33:48.427 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:48.405+0000 7fdb7f59e700 1 -- 172.21.15.88:0/2002853254 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdb8008f8a0 con 0x7fdb800966a0 2023-09-30T01:33:48.427 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:48.405+0000 7fdb767fc700 1 -- 172.21.15.88:0/2002853254 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 40144+0+0 (secure 0 0 0) 0x7fdb7801b480 con 0x7fdb800966a0 2023-09-30T01:33:48.427 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:48.405+0000 7fdb7f59e700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:33:48.427 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:48.405+0000 7fdb7f59e700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:33:48.427 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:48.405+0000 7fdb7f59e700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.88:3300/0 2023-09-30T01:33:48.427 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:48.405+0000 7fdb7f59e700 1 -- 172.21.15.88:0/2002853254 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdb80067520 con 0x7fdb800966a0 2023-09-30T01:33:48.427 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:48.405+0000 7fdb767fc700 1 --2- 172.21.15.88:0/2002853254 >> v2:172.21.15.107:6812/29926 conn(0x7fdb60033390 0x7fdb60035840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:33:48.427 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:48.405+0000 7fdb767fc700 1 -- 172.21.15.88:0/2002853254 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 3502+0+0 (secure 0 0 0) 0x7fdb78044eb0 con 0x7fdb800966a0 2023-09-30T01:33:48.427 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:48.405+0000 7fdb7dd9b700 1 --2- 172.21.15.88:0/2002853254 >> v2:172.21.15.107:6812/29926 conn(0x7fdb60033390 0x7fdb60035840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:33:48.427 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:48.405+0000 7fdb7dd9b700 10 monclient: get_auth_request con 0x7fdb60033390 auth_method 0 2023-09-30T01:33:48.427 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:48.405+0000 7fdb7dd9b700 1 --2- 172.21.15.88:0/2002853254 >> v2:172.21.15.107:6812/29926 conn(0x7fdb60033390 0x7fdb60035840 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fdb700072f0 tx=0x7fdb7000b040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:33:48.428 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:48.413+0000 7fdb767fc700 1 -- 172.21.15.88:0/2002853254 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fdb78010d10 con 0x7fdb800966a0 2023-09-30T01:33:48.428 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:48.413+0000 7fdb767fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:33:48.428 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:48.413+0000 7fdb767fc700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:33:48.617 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:48.613+0000 7fdb7f59e700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T01:33:48.618 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:48.613+0000 7fdb7f59e700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:33:48.618 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:48.613+0000 7fdb7f59e700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.88:3300/0 2023-09-30T01:33:48.618 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:48.613+0000 7fdb7f59e700 1 -- 172.21.15.88:0/2002853254 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fdb80071960 con 0x7fdb800966a0 2023-09-30T01:33:48.618 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:48.613+0000 7fdb767fc700 1 -- 172.21.15.88:0/2002853254 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7fdb7802a100 con 0x7fdb800966a0 2023-09-30T01:33:48.619 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:48.613+0000 7fdb767fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:33:48.619 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:48.613+0000 7fdb767fc700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:33:48.619 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-09-30T01:33:48.619 INFO:teuthology.orchestra.run.smithi088.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:33:48.622 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:48.617+0000 7fdb7f59e700 1 -- 172.21.15.88:0/2002853254 >> v2:172.21.15.107:6812/29926 conn(0x7fdb60033390 msgr2=0x7fdb60035840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:33:48.622 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:48.617+0000 7fdb7f59e700 1 --2- 172.21.15.88:0/2002853254 >> v2:172.21.15.107:6812/29926 conn(0x7fdb60033390 0x7fdb60035840 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fdb700072f0 tx=0x7fdb7000b040 comp rx=0 tx=0).stop 2023-09-30T01:33:48.622 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:48.617+0000 7fdb7f59e700 10 monclient: shutdown 2023-09-30T01:33:48.622 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:48.617+0000 7fdb7f59e700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:33:48.622 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:48.617+0000 7fdb7f59e700 1 -- 172.21.15.88:0/2002853254 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdb800966a0 msgr2=0x7fdb80142690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:33:48.622 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:48.617+0000 7fdb7f59e700 1 --2- 172.21.15.88:0/2002853254 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdb800966a0 0x7fdb80142690 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fdb78012000 tx=0x7fdb78013000 comp rx=0 tx=0).stop 2023-09-30T01:33:48.622 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:48.617+0000 7fdb7f59e700 1 -- 172.21.15.88:0/2002853254 shutdown_connections 2023-09-30T01:33:48.623 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:48.617+0000 7fdb7f59e700 1 --2- 172.21.15.88:0/2002853254 >> v2:172.21.15.107:6812/29926 conn(0x7fdb60033390 0x7fdb60035840 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:48.623 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:48.617+0000 7fdb7f59e700 1 --2- 172.21.15.88:0/2002853254 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7fdb8009ed80 0x7fdb80143110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:48.623 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:48.617+0000 7fdb7f59e700 1 --2- 172.21.15.88:0/2002853254 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7fdb80097cd0 0x7fdb80142bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:48.623 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:48.617+0000 7fdb7f59e700 1 --2- 172.21.15.88:0/2002853254 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdb800966a0 0x7fdb80142690 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:48.623 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:48.617+0000 7fdb7f59e700 1 -- 172.21.15.88:0/2002853254 >> 172.21.15.88:0/2002853254 conn(0x7fdb800a0910 msgr2=0x7fdb80098f00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:33:48.623 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:48.617+0000 7fdb7f59e700 1 -- 172.21.15.88:0/2002853254 shutdown_connections 2023-09-30T01:33:48.623 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:48.617+0000 7fdb7f59e700 1 -- 172.21.15.88:0/2002853254 wait complete. 2023-09-30T01:33:51.638 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:33:51.747 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:51.744+0000 7fdf0a339700 10 monclient: get_monmap_and_config 2023-09-30T01:33:51.748 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:51.744+0000 7fdf0a339700 10 monclient: build_initial_monmap 2023-09-30T01:33:51.748 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:51.744+0000 7fdf0a339700 10 monclient: monmap: 2023-09-30T01:33:51.748 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2023-09-30T01:33:51.748 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:33:51.748 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:33:51.748106+0000 2023-09-30T01:33:51.748 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:33:51.748106+0000 2023-09-30T01:33:51.748 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2023-09-30T01:33:51.748 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:33:51.748 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a-legacy 2023-09-30T01:33:51.748 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.107:6789/0 mon.noname-f-legacy 2023-09-30T01:33:51.749 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.noname-b 2023-09-30T01:33:51.749 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.noname-g 2023-09-30T01:33:51.749 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.noname-c 2023-09-30T01:33:51.749 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.noname-h 2023-09-30T01:33:51.749 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.noname-d 2023-09-30T01:33:51.749 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.noname-i 2023-09-30T01:33:51.749 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.noname-e 2023-09-30T01:33:51.749 INFO:teuthology.orchestra.run.smithi088.stderr:9: v2:172.21.15.88:3300/0 mon.noname-a 2023-09-30T01:33:51.749 INFO:teuthology.orchestra.run.smithi088.stderr:10: v2:172.21.15.107:3300/0 mon.noname-f 2023-09-30T01:33:51.749 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:33:51.753 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:51.744+0000 7fdf0a339700 1 Processor -- start 2023-09-30T01:33:51.753 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:51.744+0000 7fdf0a339700 1 -- start start 2023-09-30T01:33:51.753 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:51.744+0000 7fdf0a339700 10 monclient: init 2023-09-30T01:33:51.754 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:51.744+0000 7fdf0a339700 10 monclient: _reopen_session rank -1 2023-09-30T01:33:51.754 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:51.744+0000 7fdf0a339700 10 monclient: _add_conns ranks=[4,0,8,5,6,9,3,7,1,10,2] 2023-09-30T01:33:51.754 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:51.744+0000 7fdf0a339700 1 --2- >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7fdf0409d1b0 0x7fdf04099dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:33:51.754 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:51.744+0000 7fdf0a339700 10 monclient(hunting): picked mon.noname-c con 0x7fdf0409d1b0 addr [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] 2023-09-30T01:33:51.754 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:51.744+0000 7fdf0a339700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fdf0409dbf0 addr v1:172.21.15.88:6789/0 2023-09-30T01:33:51.754 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:51.744+0000 7fdf0a339700 1 --2- >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7fdf0409e6d0 0x7fdf04137c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:33:51.754 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:51.744+0000 7fdf0a339700 10 monclient(hunting): picked mon.noname-e con 0x7fdf0409e6d0 addr [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] 2023-09-30T01:33:51.754 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:51.744+0000 7fdf0a339700 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf0404bb20 con 0x7fdf0409dbf0 2023-09-30T01:33:51.754 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:51.744+0000 7fdf0a339700 10 monclient(hunting): start opening mon connection 2023-09-30T01:33:51.754 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:51.744+0000 7fdf0a339700 1 -- --> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] -- mon_getmap magic: 0 v1 -- 0x7fdf0409a780 con 0x7fdf0409d1b0 2023-09-30T01:33:51.754 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:51.744+0000 7fdf0a339700 10 monclient(hunting): start opening mon connection 2023-09-30T01:33:51.754 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:51.744+0000 7fdf0a339700 1 -- --> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] -- mon_getmap magic: 0 v1 -- 0x7fdf040993f0 con 0x7fdf0409e6d0 2023-09-30T01:33:51.755 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:51.744+0000 7fdf0a339700 10 monclient(hunting): _renew_subs 2023-09-30T01:33:51.755 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:51.744+0000 7fdf0a339700 10 monclient(hunting): authenticate will time out at 903.787247s 2023-09-30T01:33:51.755 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:51.744+0000 7fdf08b36700 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7fdf0409dbf0 0x7fdf0409dfc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:48174/0 (socket says 172.21.15.88:48174) 2023-09-30T01:33:51.755 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:51.744+0000 7fdf08b36700 1 -- 172.21.15.88:0/717101003 learned_addr learned my addr 172.21.15.88:0/717101003 (peer_addr_for_me v1:172.21.15.88:0/0) 2023-09-30T01:33:51.755 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:51.744+0000 7fdf09337700 1 --2- 172.21.15.88:0/717101003 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7fdf0409d1b0 0x7fdf04099dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:33:51.755 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:51.744+0000 7fdf09b38700 1 --2- 172.21.15.88:0/717101003 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7fdf0409e6d0 0x7fdf04137c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:33:51.755 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:51.744+0000 7fdf09337700 10 monclient(hunting): get_auth_request con 0x7fdf0409d1b0 auth_method 0 2023-09-30T01:33:51.755 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:51.744+0000 7fdf09337700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:33:51.755 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:51.744+0000 7fdf09337700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:33:51.755 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:51.744+0000 7fdf09337700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:33:51.755 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:51.744+0000 7fdf09b38700 10 monclient(hunting): get_auth_request con 0x7fdf0409e6d0 auth_method 0 2023-09-30T01:33:51.755 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:51.744+0000 7fdf09b38700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:33:51.756 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:51.744+0000 7fdf09b38700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:33:51.756 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:51.744+0000 7fdf09b38700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:33:51.756 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:51.744+0000 7fdf09337700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:33:51.756 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:51.744+0000 7fdf09337700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:33:51.756 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:51.744+0000 7fdf09337700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:33:51.756 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:51.744+0000 7fdf09b38700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:33:51.756 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:51.744+0000 7fdf09b38700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:33:51.756 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:51.744+0000 7fdefbfff700 1 -- 172.21.15.88:0/717101003 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 534859421 0 0) 0x7fdf0404bb20 con 0x7fdf0409dbf0 2023-09-30T01:33:51.756 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:51.744+0000 7fdf09b38700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:33:51.756 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:51.744+0000 7fdefbfff700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:33:51.756 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:51.744+0000 7fdefbfff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:33:51.756 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:51.744+0000 7fdefbfff700 10 monclient(hunting): my global_id is 4518 2023-09-30T01:33:51.757 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:51.744+0000 7fdefbfff700 1 -- 172.21.15.88:0/717101003 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdef0003570 con 0x7fdf0409dbf0 2023-09-30T01:33:51.757 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:51.748+0000 7fdf09337700 10 monclient(hunting): handle_auth_done global_id 4450 payload 274 2023-09-30T01:33:51.757 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:51.748+0000 7fdf09337700 1 -- 172.21.15.88:0/717101003 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7fdf0409e6d0 msgr2=0x7fdf04137c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:33:51.757 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:51.748+0000 7fdf09337700 1 --2- 172.21.15.88:0/717101003 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7fdf0409e6d0 0x7fdf04137c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:51.757 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:51.748+0000 7fdf09337700 1 -- 172.21.15.88:0/717101003 >> v1:172.21.15.88:6789/0 conn(0x7fdf0409dbf0 legacy=0x7fdf0409dfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:33:51.757 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:51.748+0000 7fdf09337700 10 monclient: _finish_hunting 0 2023-09-30T01:33:51.757 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:51.748+0000 7fdf09337700 1 monclient: found mon.noname-c 2023-09-30T01:33:51.757 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:51.748+0000 7fdf09337700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:33:51.757 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:51.748+0000 7fdf09337700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.88:3302/0 2023-09-30T01:33:51.757 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:51.748+0000 7fdf09337700 1 -- 172.21.15.88:0/717101003 --> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf0409f0d0 con 0x7fdf0409d1b0 2023-09-30T01:33:51.757 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:51.748+0000 7fdf09b38700 1 --2- 172.21.15.88:0/717101003 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7fdf0409e6d0 0x7fdf04137c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:33:51.757 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:51.748+0000 7fdf09337700 1 --2- 172.21.15.88:0/717101003 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7fdf0409d1b0 0x7fdf04099dd0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fdf00009910 tx=0x7fdf000091a0 comp rx=0 tx=0).ready entity=mon.4 client_cookie=3f897fdeba2ed7ef server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:33:51.758 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:51.748+0000 7fdefbfff700 1 -- 172.21.15.88:0/717101003 <== mon.4 v2:172.21.15.88:3302/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7fdf00011ab0 con 0x7fdf0409d1b0 2023-09-30T01:33:51.758 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:51.748+0000 7fdefbfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:33:51.758 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:51.748+0000 7fdefbfff700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T01:33:51.758 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:51.748+0000 7fdefbfff700 10 monclient: dump: 2023-09-30T01:33:51.758 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:33:51.758 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:33:51.758 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:51.758 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:51.758 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:33:51.758 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:33:51.758 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:33:51.758 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:33:51.758 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:33:51.759 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:33:51.759 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:33:51.759 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:33:51.759 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:33:51.759 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:33:51.759 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:33:51.759 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:33:51.759 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:51.748+0000 7fdefbfff700 10 monclient: _finish_auth 0 2023-09-30T01:33:51.759 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:51.748+0000 7fdefbfff700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:33:51.759 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:51.748+0000 7fdf0a339700 5 monclient: authenticate success, global_id 4450 2023-09-30T01:33:51.759 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:51.748+0000 7fdefbfff700 1 -- 172.21.15.88:0/717101003 <== mon.4 v2:172.21.15.88:3302/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf00011c30 con 0x7fdf0409d1b0 2023-09-30T01:33:51.759 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:51.748+0000 7fdf0a339700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:33:51.760 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:51.748+0000 7fdefbfff700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:33:51.760 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:51.748+0000 7fdf0a339700 10 monclient: get_monmap_and_config success 2023-09-30T01:33:51.760 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:51.748+0000 7fdefbfff700 1 -- 172.21.15.88:0/717101003 <== mon.4 v2:172.21.15.88:3302/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7fdf00011da0 con 0x7fdf0409d1b0 2023-09-30T01:33:51.760 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:51.748+0000 7fdf0a339700 4 set_mon_vals no callback set 2023-09-30T01:33:51.760 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:51.748+0000 7fdefbfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:33:51.760 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:51.748+0000 7fdefbfff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T01:33:51.760 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:51.748+0000 7fdefbfff700 10 monclient: dump: 2023-09-30T01:33:51.760 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:33:51.760 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:33:51.760 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:51.760 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:51.760 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:33:51.761 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:33:51.761 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:33:51.761 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:33:51.761 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:33:51.761 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:33:51.761 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:33:51.761 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:33:51.761 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:33:51.761 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:33:51.761 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:33:51.761 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:33:51.761 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:51.748+0000 7fdf0a339700 10 monclient: shutdown 2023-09-30T01:33:51.761 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:51.748+0000 7fdf0a339700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:33:51.762 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:51.748+0000 7fdf0a339700 1 -- 172.21.15.88:0/717101003 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7fdf0409d1b0 msgr2=0x7fdf04099dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:33:51.762 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:51.748+0000 7fdf0a339700 1 --2- 172.21.15.88:0/717101003 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7fdf0409d1b0 0x7fdf04099dd0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fdf00009910 tx=0x7fdf000091a0 comp rx=0 tx=0).stop 2023-09-30T01:33:51.762 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:51.748+0000 7fdf0a339700 1 -- 172.21.15.88:0/717101003 shutdown_connections 2023-09-30T01:33:51.762 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:51.748+0000 7fdf0a339700 1 --2- 172.21.15.88:0/717101003 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7fdf0409e6d0 0x7fdf04137c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:51.762 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:51.748+0000 7fdf0a339700 1 --2- 172.21.15.88:0/717101003 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7fdf0409d1b0 0x7fdf04099dd0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:51.762 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:51.748+0000 7fdf0a339700 1 -- 172.21.15.88:0/717101003 >> 172.21.15.88:0/717101003 conn(0x7fdf0409fe10 msgr2=0x7fdf040a0210 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:33:51.762 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:51.748+0000 7fdf0a339700 1 -- 172.21.15.88:0/717101003 shutdown_connections 2023-09-30T01:33:51.762 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:51.748+0000 7fdf0a339700 1 -- 172.21.15.88:0/717101003 wait complete. 2023-09-30T01:33:51.767 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:51.748+0000 7fdf0a339700 10 monclient: build_initial_monmap 2023-09-30T01:33:51.767 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:51.748+0000 7fdf0a339700 10 monclient: monmap: 2023-09-30T01:33:51.767 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2023-09-30T01:33:51.767 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:33:51.767 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 0.000000 2023-09-30T01:33:51.768 INFO:teuthology.orchestra.run.smithi088.stderr:created 0.000000 2023-09-30T01:33:51.768 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2023-09-30T01:33:51.768 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 1 2023-09-30T01:33:51.768 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.noname-a 2023-09-30T01:33:51.768 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.noname-f 2023-09-30T01:33:51.768 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.noname-b 2023-09-30T01:33:51.768 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.noname-g 2023-09-30T01:33:51.768 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.noname-c 2023-09-30T01:33:51.768 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.noname-h 2023-09-30T01:33:51.768 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.noname-d 2023-09-30T01:33:51.768 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.noname-i 2023-09-30T01:33:51.768 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.noname-e 2023-09-30T01:33:51.769 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:33:51.769 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:51.748+0000 7fdf0a339700 1 Processor -- start 2023-09-30T01:33:51.769 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:51.748+0000 7fdf0a339700 1 -- start start 2023-09-30T01:33:51.769 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:51.748+0000 7fdf0a339700 10 monclient: init 2023-09-30T01:33:51.769 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:51.752+0000 7fdf0a339700 10 monclient: _reopen_session rank -1 2023-09-30T01:33:51.769 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:51.752+0000 7fdf0a339700 10 monclient: _add_conns ranks=[7,2,6,4,3,5,8,1,0] 2023-09-30T01:33:51.769 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:51.752+0000 7fdf0a339700 1 --2- >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7fdf0409d1b0 0x7fdf04135f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:33:51.769 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:51.752+0000 7fdf0a339700 10 monclient(hunting): picked mon.noname-i con 0x7fdf0409d1b0 addr [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] 2023-09-30T01:33:51.769 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:51.752+0000 7fdf0a339700 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fdf0409dbf0 0x7fdf0411f9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:33:51.769 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:51.752+0000 7fdf0a339700 10 monclient(hunting): picked mon.noname-b con 0x7fdf0409dbf0 addr [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] 2023-09-30T01:33:51.769 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:51.752+0000 7fdf0a339700 1 --2- >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7fdf0409e6d0 0x7fdf0411ff00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:33:51.769 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:51.752+0000 7fdf0a339700 10 monclient(hunting): picked mon.noname-d con 0x7fdf0409e6d0 addr [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] 2023-09-30T01:33:51.769 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:51.752+0000 7fdf0a339700 10 monclient(hunting): start opening mon connection 2023-09-30T01:33:51.770 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:51.752+0000 7fdf0a339700 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf041236e0 con 0x7fdf0409dbf0 2023-09-30T01:33:51.770 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:51.752+0000 7fdf0a339700 10 monclient(hunting): start opening mon connection 2023-09-30T01:33:51.770 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:51.752+0000 7fdf0a339700 1 -- --> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] -- mon_getmap magic: 0 v1 -- 0x7fdf04098930 con 0x7fdf0409e6d0 2023-09-30T01:33:51.770 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:51.752+0000 7fdf0a339700 10 monclient(hunting): start opening mon connection 2023-09-30T01:33:51.770 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:51.752+0000 7fdf0a339700 1 -- --> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] -- mon_getmap magic: 0 v1 -- 0x7fdf04098460 con 0x7fdf0409d1b0 2023-09-30T01:33:51.770 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:51.752+0000 7fdf0a339700 10 monclient(hunting): _renew_subs 2023-09-30T01:33:51.770 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:51.752+0000 7fdf0a339700 10 monclient(hunting): authenticate will time out at 903.792150s 2023-09-30T01:33:51.770 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:51.752+0000 7fdf08b36700 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fdf0409dbf0 0x7fdf0411f9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:33:51.770 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:51.752+0000 7fdf08b36700 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fdf0409dbf0 0x7fdf0411f9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3301/0 says I am v2:172.21.15.88:36974/0 (socket says 172.21.15.88:36974) 2023-09-30T01:33:51.770 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:51.752+0000 7fdf09b38700 1 --2- >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7fdf0409e6d0 0x7fdf0411ff00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:33:51.770 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:51.752+0000 7fdf08b36700 1 -- 172.21.15.88:0/1851438573 learned_addr learned my addr 172.21.15.88:0/1851438573 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-09-30T01:33:51.770 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:51.752+0000 7fdf08b36700 10 monclient(hunting): get_auth_request con 0x7fdf0409dbf0 auth_method 0 2023-09-30T01:33:51.771 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:51.752+0000 7fdf08b36700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:33:51.771 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:51.752+0000 7fdf08b36700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:33:51.771 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:51.752+0000 7fdf08b36700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:33:51.771 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:51.752+0000 7fdf09b38700 10 monclient(hunting): get_auth_request con 0x7fdf0409e6d0 auth_method 0 2023-09-30T01:33:51.771 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:51.752+0000 7fdf09b38700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:33:51.771 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:51.752+0000 7fdf09b38700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:33:51.771 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:51.752+0000 7fdf09b38700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:33:51.771 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:51.752+0000 7fdf08b36700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:33:51.771 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:51.752+0000 7fdf08b36700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:33:51.771 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:51.752+0000 7fdf08b36700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:33:51.771 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:51.752+0000 7fdf09337700 1 --2- 172.21.15.88:0/1851438573 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7fdf0409d1b0 0x7fdf04135f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:33:51.771 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:51.752+0000 7fdf09337700 10 monclient(hunting): get_auth_request con 0x7fdf0409d1b0 auth_method 0 2023-09-30T01:33:51.772 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:51.752+0000 7fdf09337700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:33:51.772 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:51.752+0000 7fdf09337700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:33:51.772 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:51.752+0000 7fdf09337700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:33:51.772 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:51.752+0000 7fdf09b38700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:33:51.772 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:51.752+0000 7fdf09b38700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:33:51.772 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:51.752+0000 7fdf09b38700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:33:51.772 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:51.752+0000 7fdf09337700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:33:51.772 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:51.752+0000 7fdf09337700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:33:51.772 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:51.752+0000 7fdf09337700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:33:51.772 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:51.752+0000 7fdf08b36700 10 monclient(hunting): handle_auth_done global_id 4430 payload 824 2023-09-30T01:33:51.772 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:51.752+0000 7fdf08b36700 1 -- 172.21.15.88:0/1851438573 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7fdf0409d1b0 msgr2=0x7fdf04135f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:33:51.772 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:51.752+0000 7fdf08b36700 1 --2- 172.21.15.88:0/1851438573 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7fdf0409d1b0 0x7fdf04135f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:51.772 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:51.752+0000 7fdf08b36700 1 -- 172.21.15.88:0/1851438573 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7fdf0409e6d0 msgr2=0x7fdf0411ff00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:33:51.773 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:51.752+0000 7fdf08b36700 1 --2- 172.21.15.88:0/1851438573 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7fdf0409e6d0 0x7fdf0411ff00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:51.773 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:51.752+0000 7fdf08b36700 10 monclient: _finish_hunting 0 2023-09-30T01:33:51.773 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:51.752+0000 7fdf08b36700 1 monclient: found mon.noname-b 2023-09-30T01:33:51.773 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:51.752+0000 7fdf08b36700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:33:51.773 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:51.752+0000 7fdf08b36700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.88:3301/0 2023-09-30T01:33:51.773 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:51.752+0000 7fdf08b36700 1 -- 172.21.15.88:0/1851438573 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdefc009a20 con 0x7fdf0409dbf0 2023-09-30T01:33:51.773 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:51.752+0000 7fdf08b36700 1 --2- 172.21.15.88:0/1851438573 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fdf0409dbf0 0x7fdf0411f9c0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fdef400c7c0 tx=0x7fdef4007df0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:33:51.773 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:51.752+0000 7fdef97fa700 1 -- 172.21.15.88:0/1851438573 <== mon.2 v2:172.21.15.88:3301/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7fdef40070d0 con 0x7fdf0409dbf0 2023-09-30T01:33:51.773 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:51.752+0000 7fdef97fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:33:51.773 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:51.752+0000 7fdef97fa700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T01:33:51.773 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:51.752+0000 7fdef97fa700 10 monclient: dump: 2023-09-30T01:33:51.773 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:33:51.774 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:33:51.774 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:51.774 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:51.774 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:33:51.774 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:33:51.774 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:33:51.774 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:33:51.774 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:33:51.774 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:33:51.774 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:33:51.774 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:33:51.774 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:33:51.775 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:33:51.775 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:33:51.775 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:33:51.775 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:51.752+0000 7fdef97fa700 10 monclient: _finish_auth 0 2023-09-30T01:33:51.775 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:51.752+0000 7fdef97fa700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:33:51.775 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:51.752+0000 7fdef97fa700 1 -- 172.21.15.88:0/1851438573 <== mon.2 v2:172.21.15.88:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdef4007290 con 0x7fdf0409dbf0 2023-09-30T01:33:51.775 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:51.752+0000 7fdef97fa700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:33:51.775 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:51.752+0000 7fdef97fa700 1 -- 172.21.15.88:0/1851438573 <== mon.2 v2:172.21.15.88:3301/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7fdef4011360 con 0x7fdf0409dbf0 2023-09-30T01:33:51.775 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:51.752+0000 7fdf0a339700 5 monclient: authenticate success, global_id 4430 2023-09-30T01:33:51.775 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:51.752+0000 7fdefaffd700 4 set_mon_vals no callback set 2023-09-30T01:33:51.775 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:51.752+0000 7fdf0a339700 10 monclient: _renew_subs 2023-09-30T01:33:51.775 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:51.752+0000 7fdf0a339700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.88:3301/0 2023-09-30T01:33:51.776 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:51.752+0000 7fdf0a339700 1 -- 172.21.15.88:0/1851438573 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf00009420 con 0x7fdf0409dbf0 2023-09-30T01:33:51.776 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:51.752+0000 7fdef97fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:33:51.776 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:51.752+0000 7fdef97fa700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T01:33:51.776 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:51.752+0000 7fdef97fa700 10 monclient: dump: 2023-09-30T01:33:51.776 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:33:51.776 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:33:51.776 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:51.776 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:51.776 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:33:51.776 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:33:51.776 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:33:51.776 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:33:51.776 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:33:51.777 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:33:51.777 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:33:51.777 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:33:51.777 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:33:51.777 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:33:51.777 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:33:51.777 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:33:51.777 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:51.752+0000 7fdf0a339700 10 monclient: _renew_subs 2023-09-30T01:33:51.777 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:51.752+0000 7fdf0a339700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.88:3301/0 2023-09-30T01:33:51.777 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:51.752+0000 7fdf0a339700 1 -- 172.21.15.88:0/1851438573 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf041221a0 con 0x7fdf0409dbf0 2023-09-30T01:33:51.777 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:51.752+0000 7fdef97fa700 1 -- 172.21.15.88:0/1851438573 <== mon.2 v2:172.21.15.88:3301/0 4 ==== mgrmap(e 5) v1 ==== 40144+0+0 (secure 0 0 0) 0x7fdef4005b30 con 0x7fdf0409dbf0 2023-09-30T01:33:51.777 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:51.752+0000 7fdf0a339700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:33:51.777 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:51.752+0000 7fdf0a339700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:33:51.778 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:51.752+0000 7fdf0a339700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.88:3301/0 2023-09-30T01:33:51.778 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:51.752+0000 7fdef97fa700 1 --2- 172.21.15.88:0/1851438573 >> v2:172.21.15.107:6812/29926 conn(0x7fdee8033360 0x7fdee8035810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:33:51.778 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:51.752+0000 7fdf0a339700 1 -- 172.21.15.88:0/1851438573 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdf041231b0 con 0x7fdf0409dbf0 2023-09-30T01:33:51.778 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:51.752+0000 7fdef97fa700 1 -- 172.21.15.88:0/1851438573 <== mon.2 v2:172.21.15.88:3301/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 3502+0+0 (secure 0 0 0) 0x7fdef40423c0 con 0x7fdf0409dbf0 2023-09-30T01:33:51.778 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:51.752+0000 7fdf09337700 1 --2- 172.21.15.88:0/1851438573 >> v2:172.21.15.107:6812/29926 conn(0x7fdee8033360 0x7fdee8035810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:33:51.778 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:51.752+0000 7fdf09337700 10 monclient: get_auth_request con 0x7fdee8033360 auth_method 0 2023-09-30T01:33:51.778 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:51.756+0000 7fdf09337700 1 --2- 172.21.15.88:0/1851438573 >> v2:172.21.15.107:6812/29926 conn(0x7fdee8033360 0x7fdee8035810 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fdf00009910 tx=0x7fdf00017000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:33:51.778 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:51.760+0000 7fdef97fa700 1 -- 172.21.15.88:0/1851438573 <== mon.2 v2:172.21.15.88:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fdef4017030 con 0x7fdf0409dbf0 2023-09-30T01:33:51.778 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:51.760+0000 7fdef97fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:33:51.778 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:51.760+0000 7fdef97fa700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:33:51.958 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:51.952+0000 7fdf0a339700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T01:33:51.958 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:51.952+0000 7fdf0a339700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:33:51.958 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:51.952+0000 7fdf0a339700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.88:3301/0 2023-09-30T01:33:51.958 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:51.952+0000 7fdf0a339700 1 -- 172.21.15.88:0/1851438573 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fdf04071960 con 0x7fdf0409dbf0 2023-09-30T01:33:51.958 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:51.952+0000 7fdef97fa700 1 -- 172.21.15.88:0/1851438573 <== mon.2 v2:172.21.15.88:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7fdef4017030 con 0x7fdf0409dbf0 2023-09-30T01:33:51.959 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:51.952+0000 7fdef97fa700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:33:51.959 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:51.952+0000 7fdef97fa700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:33:51.959 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-09-30T01:33:51.959 INFO:teuthology.orchestra.run.smithi088.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:33:51.961 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:51.956+0000 7fdf0a339700 1 -- 172.21.15.88:0/1851438573 >> v2:172.21.15.107:6812/29926 conn(0x7fdee8033360 msgr2=0x7fdee8035810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:33:51.961 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:51.956+0000 7fdf0a339700 1 --2- 172.21.15.88:0/1851438573 >> v2:172.21.15.107:6812/29926 conn(0x7fdee8033360 0x7fdee8035810 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fdf00009910 tx=0x7fdf00017000 comp rx=0 tx=0).stop 2023-09-30T01:33:51.961 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:51.956+0000 7fdf0a339700 10 monclient: shutdown 2023-09-30T01:33:51.961 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:51.956+0000 7fdf0a339700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:33:51.961 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:51.956+0000 7fdf0a339700 1 -- 172.21.15.88:0/1851438573 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fdf0409dbf0 msgr2=0x7fdf0411f9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:33:51.961 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:51.956+0000 7fdf0a339700 1 --2- 172.21.15.88:0/1851438573 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fdf0409dbf0 0x7fdf0411f9c0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fdef400c7c0 tx=0x7fdef4007df0 comp rx=0 tx=0).stop 2023-09-30T01:33:51.961 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:51.956+0000 7fdf0a339700 1 -- 172.21.15.88:0/1851438573 shutdown_connections 2023-09-30T01:33:51.961 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:51.956+0000 7fdf0a339700 1 --2- 172.21.15.88:0/1851438573 >> v2:172.21.15.107:6812/29926 conn(0x7fdee8033360 0x7fdee8035810 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:51.961 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:51.956+0000 7fdf0a339700 1 --2- 172.21.15.88:0/1851438573 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7fdf0409e6d0 0x7fdf0411ff00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:51.961 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:51.956+0000 7fdf0a339700 1 --2- 172.21.15.88:0/1851438573 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fdf0409dbf0 0x7fdf0411f9c0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:51.962 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:51.956+0000 7fdf0a339700 1 --2- 172.21.15.88:0/1851438573 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7fdf0409d1b0 0x7fdf04135f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:51.962 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:51.956+0000 7fdf0a339700 1 -- 172.21.15.88:0/1851438573 >> 172.21.15.88:0/1851438573 conn(0x7fdf0409fe10 msgr2=0x7fdf04062c70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:33:51.962 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:51.956+0000 7fdf0a339700 1 -- 172.21.15.88:0/1851438573 shutdown_connections 2023-09-30T01:33:51.962 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:51.956+0000 7fdf0a339700 1 -- 172.21.15.88:0/1851438573 wait complete. 2023-09-30T01:33:54.973 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:33:55.083 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:55.076+0000 7f5ae6731700 10 monclient: get_monmap_and_config 2023-09-30T01:33:55.083 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:55.076+0000 7f5ae6731700 10 monclient: build_initial_monmap 2023-09-30T01:33:55.083 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:55.076+0000 7f5ae6731700 10 monclient: monmap: 2023-09-30T01:33:55.083 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2023-09-30T01:33:55.083 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:33:55.083 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:33:55.083544+0000 2023-09-30T01:33:55.083 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:33:55.083544+0000 2023-09-30T01:33:55.084 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2023-09-30T01:33:55.084 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:33:55.084 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a-legacy 2023-09-30T01:33:55.084 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.107:6789/0 mon.noname-f-legacy 2023-09-30T01:33:55.084 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.noname-b 2023-09-30T01:33:55.084 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.noname-g 2023-09-30T01:33:55.084 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.noname-c 2023-09-30T01:33:55.084 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.noname-h 2023-09-30T01:33:55.084 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.noname-d 2023-09-30T01:33:55.084 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.noname-i 2023-09-30T01:33:55.084 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.noname-e 2023-09-30T01:33:55.084 INFO:teuthology.orchestra.run.smithi088.stderr:9: v2:172.21.15.88:3300/0 mon.noname-a 2023-09-30T01:33:55.084 INFO:teuthology.orchestra.run.smithi088.stderr:10: v2:172.21.15.107:3300/0 mon.noname-f 2023-09-30T01:33:55.085 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:33:55.088 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:55.080+0000 7f5ae6731700 1 Processor -- start 2023-09-30T01:33:55.088 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:55.080+0000 7f5ae6731700 1 -- start start 2023-09-30T01:33:55.088 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:55.080+0000 7f5ae6731700 10 monclient: init 2023-09-30T01:33:55.088 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:55.080+0000 7f5ae6731700 10 monclient: _reopen_session rank -1 2023-09-30T01:33:55.088 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:55.080+0000 7f5ae6731700 10 monclient: _add_conns ranks=[5,4,0,6,3,2,7,1,10,8,9] 2023-09-30T01:33:55.088 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:55.080+0000 7f5ae6731700 1 --2- >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7f5ae009e8d0 0x7f5ae0099a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:33:55.089 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:55.080+0000 7f5ae6731700 10 monclient(hunting): picked mon.noname-h con 0x7f5ae009e8d0 addr [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] 2023-09-30T01:33:55.089 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:55.080+0000 7f5ae6731700 1 --2- >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f5ae0096b90 0x7f5ae0099f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:33:55.089 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:55.080+0000 7f5ae6731700 10 monclient(hunting): picked mon.noname-c con 0x7f5ae0096b90 addr [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] 2023-09-30T01:33:55.089 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:55.080+0000 7f5ae6731700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f5ae009a4a0 addr v1:172.21.15.88:6789/0 2023-09-30T01:33:55.089 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:55.080+0000 7f5ae6731700 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ae004bb20 con 0x7f5ae009a4a0 2023-09-30T01:33:55.089 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:55.080+0000 7f5ae6731700 10 monclient(hunting): start opening mon connection 2023-09-30T01:33:55.089 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:55.080+0000 7f5ae6731700 1 -- --> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] -- mon_getmap magic: 0 v1 -- 0x7f5ae009a8f0 con 0x7f5ae0096b90 2023-09-30T01:33:55.089 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:55.080+0000 7f5ae6731700 10 monclient(hunting): start opening mon connection 2023-09-30T01:33:55.089 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:55.080+0000 7f5ae6731700 1 -- --> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] -- mon_getmap magic: 0 v1 -- 0x7f5ae009c580 con 0x7f5ae009e8d0 2023-09-30T01:33:55.089 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:55.080+0000 7f5ae6731700 10 monclient(hunting): _renew_subs 2023-09-30T01:33:55.089 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:55.080+0000 7f5ae6731700 10 monclient(hunting): authenticate will time out at 907.120992s 2023-09-30T01:33:55.089 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:55.080+0000 7f5ae4f2e700 1 --2- >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f5ae0096b90 0x7f5ae0099f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:33:55.090 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:55.080+0000 7f5ae5f30700 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f5ae009a4a0 0x7f5ae01366d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:48180/0 (socket says 172.21.15.88:48180) 2023-09-30T01:33:55.090 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:55.080+0000 7f5ae5f30700 1 -- 172.21.15.88:0/4088577955 learned_addr learned my addr 172.21.15.88:0/4088577955 (peer_addr_for_me v1:172.21.15.88:0/0) 2023-09-30T01:33:55.090 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:55.080+0000 7f5ae4f2e700 10 monclient(hunting): get_auth_request con 0x7f5ae0096b90 auth_method 0 2023-09-30T01:33:55.090 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:55.080+0000 7f5ae4f2e700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:33:55.090 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:55.080+0000 7f5ae4f2e700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:33:55.090 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:55.080+0000 7f5ae4f2e700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:33:55.090 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:55.080+0000 7f5ae572f700 1 --2- 172.21.15.88:0/4088577955 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7f5ae009e8d0 0x7f5ae0099a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:33:55.090 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:55.080+0000 7f5ae572f700 10 monclient(hunting): get_auth_request con 0x7f5ae009e8d0 auth_method 0 2023-09-30T01:33:55.090 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:55.080+0000 7f5ae572f700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:33:55.090 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:55.080+0000 7f5ae572f700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:33:55.090 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:55.080+0000 7f5ae572f700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:33:55.090 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:55.080+0000 7f5ae572f700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:33:55.091 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:55.080+0000 7f5ae572f700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:33:55.091 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:55.080+0000 7f5ae572f700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:33:55.091 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:55.080+0000 7f5ae4f2e700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:33:55.091 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:55.080+0000 7f5ae4f2e700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:33:55.091 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:55.080+0000 7f5ae4f2e700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:33:55.091 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:55.080+0000 7f5ad7fff700 1 -- 172.21.15.88:0/4088577955 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 607243596 0 0) 0x7f5ae004bb20 con 0x7f5ae009a4a0 2023-09-30T01:33:55.091 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:55.080+0000 7f5ae4f2e700 10 monclient(hunting): handle_auth_done global_id 4459 payload 274 2023-09-30T01:33:55.091 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:55.080+0000 7f5ae4f2e700 1 -- 172.21.15.88:0/4088577955 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7f5ae009e8d0 msgr2=0x7f5ae0099a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:33:55.091 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:55.080+0000 7f5ae4f2e700 1 --2- 172.21.15.88:0/4088577955 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7f5ae009e8d0 0x7f5ae0099a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:55.091 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:55.080+0000 7f5ae4f2e700 1 -- 172.21.15.88:0/4088577955 >> v1:172.21.15.88:6789/0 conn(0x7f5ae009a4a0 legacy=0x7f5ae01366d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:33:55.091 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:55.080+0000 7f5ae4f2e700 10 monclient: _finish_hunting 0 2023-09-30T01:33:55.091 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:55.080+0000 7f5ae4f2e700 1 monclient: found mon.noname-c 2023-09-30T01:33:55.092 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:55.080+0000 7f5ae4f2e700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:33:55.092 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:55.080+0000 7f5ae4f2e700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.88:3302/0 2023-09-30T01:33:55.092 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:55.080+0000 7f5ae4f2e700 1 -- 172.21.15.88:0/4088577955 --> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ae009bf30 con 0x7f5ae0096b90 2023-09-30T01:33:55.092 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:55.080+0000 7f5ae572f700 1 --2- 172.21.15.88:0/4088577955 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7f5ae009e8d0 0x7f5ae0099a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:33:55.092 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:55.080+0000 7f5ad7fff700 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2023-09-30T01:33:55.092 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:55.080+0000 7f5ae4f2e700 1 --2- 172.21.15.88:0/4088577955 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f5ae0096b90 0x7f5ae0099f60 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f5ad8009ec0 tx=0x7f5ad8009780 comp rx=0 tx=0).ready entity=mon.4 client_cookie=e3d7fea263c03cac server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:33:55.092 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:55.080+0000 7f5ad7fff700 1 -- 172.21.15.88:0/4088577955 <== mon.4 v2:172.21.15.88:3302/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f5ad800e070 con 0x7f5ae0096b90 2023-09-30T01:33:55.092 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:55.080+0000 7f5ad7fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:33:55.092 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:55.080+0000 7f5ad7fff700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T01:33:55.092 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:55.080+0000 7f5ad7fff700 10 monclient: dump: 2023-09-30T01:33:55.092 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:33:55.092 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:33:55.093 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:55.093 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:55.093 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:33:55.093 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:33:55.093 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:33:55.093 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:33:55.093 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:33:55.093 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:33:55.093 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:33:55.093 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:33:55.093 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:33:55.093 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:33:55.093 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:33:55.094 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:33:55.094 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:55.080+0000 7f5ad7fff700 10 monclient: _finish_auth 0 2023-09-30T01:33:55.094 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:55.080+0000 7f5ad7fff700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:33:55.094 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:55.080+0000 7f5ae6731700 5 monclient: authenticate success, global_id 4459 2023-09-30T01:33:55.094 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:55.080+0000 7f5ae6731700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:33:55.094 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:55.080+0000 7f5ad7fff700 1 -- 172.21.15.88:0/4088577955 <== mon.4 v2:172.21.15.88:3302/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ad800bbf0 con 0x7f5ae0096b90 2023-09-30T01:33:55.094 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:55.080+0000 7f5ad7fff700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:33:55.094 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:55.080+0000 7f5ae6731700 10 monclient: get_monmap_and_config success 2023-09-30T01:33:55.094 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:55.080+0000 7f5ae6731700 4 set_mon_vals no callback set 2023-09-30T01:33:55.094 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:55.080+0000 7f5ad7fff700 1 -- 172.21.15.88:0/4088577955 <== mon.4 v2:172.21.15.88:3302/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f5ad800bd60 con 0x7f5ae0096b90 2023-09-30T01:33:55.094 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:55.080+0000 7f5ad7fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:33:55.094 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:55.080+0000 7f5ad7fff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T01:33:55.095 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:55.080+0000 7f5ad7fff700 10 monclient: dump: 2023-09-30T01:33:55.095 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:33:55.095 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:33:55.095 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:55.095 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:55.095 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:33:55.095 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:33:55.095 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:33:55.095 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:33:55.095 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:33:55.095 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:33:55.095 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:33:55.095 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:33:55.096 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:33:55.096 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:33:55.096 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:33:55.096 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:33:55.096 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:55.080+0000 7f5ae6731700 10 monclient: shutdown 2023-09-30T01:33:55.096 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:55.080+0000 7f5ae6731700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:33:55.096 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:55.080+0000 7f5ae6731700 1 -- 172.21.15.88:0/4088577955 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f5ae0096b90 msgr2=0x7f5ae0099f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:33:55.096 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:55.080+0000 7f5ae6731700 1 --2- 172.21.15.88:0/4088577955 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f5ae0096b90 0x7f5ae0099f60 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f5ad8009ec0 tx=0x7f5ad8009780 comp rx=0 tx=0).stop 2023-09-30T01:33:55.096 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:55.080+0000 7f5ae6731700 1 -- 172.21.15.88:0/4088577955 shutdown_connections 2023-09-30T01:33:55.096 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:55.080+0000 7f5ae6731700 1 --2- 172.21.15.88:0/4088577955 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f5ae0096b90 0x7f5ae0099f60 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:55.096 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:55.080+0000 7f5ae6731700 1 --2- 172.21.15.88:0/4088577955 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7f5ae009e8d0 0x7f5ae0099a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:55.096 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:55.080+0000 7f5ae6731700 1 -- 172.21.15.88:0/4088577955 >> 172.21.15.88:0/4088577955 conn(0x7f5ae00a2540 msgr2=0x7f5ae00a2910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:33:55.097 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:55.080+0000 7f5ae6731700 1 -- 172.21.15.88:0/4088577955 shutdown_connections 2023-09-30T01:33:55.097 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:55.080+0000 7f5ae6731700 1 -- 172.21.15.88:0/4088577955 wait complete. 2023-09-30T01:33:55.097 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:55.084+0000 7f5ae6731700 10 monclient: build_initial_monmap 2023-09-30T01:33:55.097 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:55.084+0000 7f5ae6731700 10 monclient: monmap: 2023-09-30T01:33:55.097 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2023-09-30T01:33:55.097 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:33:55.097 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 0.000000 2023-09-30T01:33:55.097 INFO:teuthology.orchestra.run.smithi088.stderr:created 0.000000 2023-09-30T01:33:55.097 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2023-09-30T01:33:55.097 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 1 2023-09-30T01:33:55.097 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.noname-a 2023-09-30T01:33:55.098 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.noname-f 2023-09-30T01:33:55.098 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.noname-b 2023-09-30T01:33:55.098 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.noname-g 2023-09-30T01:33:55.098 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.noname-c 2023-09-30T01:33:55.098 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.noname-h 2023-09-30T01:33:55.098 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.noname-d 2023-09-30T01:33:55.098 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.noname-i 2023-09-30T01:33:55.098 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.noname-e 2023-09-30T01:33:55.098 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:33:55.098 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:55.084+0000 7f5ae6731700 1 Processor -- start 2023-09-30T01:33:55.098 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:55.084+0000 7f5ae6731700 1 -- start start 2023-09-30T01:33:55.098 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:55.084+0000 7f5ae6731700 10 monclient: init 2023-09-30T01:33:55.102 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:55.084+0000 7f5ae6731700 10 monclient: _reopen_session rank -1 2023-09-30T01:33:55.102 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:55.084+0000 7f5ae6731700 10 monclient: _add_conns ranks=[5,7,8,0,4,3,1,6,2] 2023-09-30T01:33:55.102 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:55.084+0000 7f5ae6731700 1 --2- >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7f5ae0096b90 0x7f5ae0095260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:33:55.102 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:55.084+0000 7f5ae6731700 10 monclient(hunting): picked mon.noname-h con 0x7f5ae0096b90 addr [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] 2023-09-30T01:33:55.102 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:55.084+0000 7f5ae6731700 1 --2- >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7f5ae009a4a0 0x7f5ae00957a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:33:55.102 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:55.084+0000 7f5ae6731700 10 monclient(hunting): picked mon.noname-i con 0x7f5ae009a4a0 addr [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] 2023-09-30T01:33:55.102 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:55.084+0000 7f5ae6731700 1 --2- >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f5ae009e8d0 0x7f5ae006aea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:33:55.102 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:55.084+0000 7f5ae6731700 10 monclient(hunting): picked mon.noname-e con 0x7f5ae009e8d0 addr [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] 2023-09-30T01:33:55.102 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:55.084+0000 7f5ae6731700 10 monclient(hunting): start opening mon connection 2023-09-30T01:33:55.102 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:55.084+0000 7f5ae6731700 1 -- --> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] -- mon_getmap magic: 0 v1 -- 0x7f5ae00961f0 con 0x7f5ae0096b90 2023-09-30T01:33:55.103 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:55.084+0000 7f5ae6731700 10 monclient(hunting): start opening mon connection 2023-09-30T01:33:55.103 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:55.084+0000 7f5ae6731700 1 -- --> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] -- mon_getmap magic: 0 v1 -- 0x7f5ae006f250 con 0x7f5ae009a4a0 2023-09-30T01:33:55.103 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:55.084+0000 7f5ae6731700 10 monclient(hunting): start opening mon connection 2023-09-30T01:33:55.103 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:55.084+0000 7f5ae6731700 1 -- --> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] -- mon_getmap magic: 0 v1 -- 0x7f5ae006e5f0 con 0x7f5ae009e8d0 2023-09-30T01:33:55.103 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:55.084+0000 7f5ae6731700 10 monclient(hunting): _renew_subs 2023-09-30T01:33:55.103 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:55.084+0000 7f5ae6731700 10 monclient(hunting): authenticate will time out at 907.125883s 2023-09-30T01:33:55.103 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:55.084+0000 7f5ae5f30700 1 --2- >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f5ae009e8d0 0x7f5ae006aea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:33:55.103 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:55.084+0000 7f5ae5f30700 1 --2- >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f5ae009e8d0 0x7f5ae006aea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3304/0 says I am v2:172.21.15.88:44038/0 (socket says 172.21.15.88:44038) 2023-09-30T01:33:55.103 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:55.084+0000 7f5ae572f700 1 --2- >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7f5ae0096b90 0x7f5ae0095260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:33:55.103 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:55.084+0000 7f5ae4f2e700 1 --2- >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7f5ae009a4a0 0x7f5ae00957a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:33:55.103 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:55.084+0000 7f5ae5f30700 1 -- 172.21.15.88:0/1541995927 learned_addr learned my addr 172.21.15.88:0/1541995927 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-09-30T01:33:55.103 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:55.084+0000 7f5ae5f30700 10 monclient(hunting): get_auth_request con 0x7f5ae009e8d0 auth_method 0 2023-09-30T01:33:55.104 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:55.084+0000 7f5ae5f30700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:33:55.104 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:55.084+0000 7f5ae5f30700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:33:55.104 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:55.084+0000 7f5ae5f30700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:33:55.104 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:55.084+0000 7f5ae4f2e700 10 monclient(hunting): get_auth_request con 0x7f5ae009a4a0 auth_method 0 2023-09-30T01:33:55.104 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:55.084+0000 7f5ae4f2e700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:33:55.104 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:55.084+0000 7f5ae4f2e700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:33:55.104 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:55.084+0000 7f5ae4f2e700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:33:55.104 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:55.084+0000 7f5ae572f700 10 monclient(hunting): get_auth_request con 0x7f5ae0096b90 auth_method 0 2023-09-30T01:33:55.104 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:55.084+0000 7f5ae572f700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:33:55.104 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:55.084+0000 7f5ae572f700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:33:55.104 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:55.084+0000 7f5ae572f700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:33:55.104 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:55.084+0000 7f5ae5f30700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:33:55.105 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:55.084+0000 7f5ae5f30700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:33:55.105 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:55.084+0000 7f5ae5f30700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:33:55.105 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:55.084+0000 7f5ae572f700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:33:55.105 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:55.084+0000 7f5ae572f700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:33:55.105 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:55.084+0000 7f5ae572f700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:33:55.105 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:55.084+0000 7f5ae4f2e700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:33:55.105 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:55.084+0000 7f5ae4f2e700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:33:55.105 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:55.084+0000 7f5ae4f2e700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:33:55.105 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:55.084+0000 7f5ae5f30700 10 monclient(hunting): handle_auth_done global_id 4499 payload 824 2023-09-30T01:33:55.105 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:55.084+0000 7f5ae5f30700 1 -- 172.21.15.88:0/1541995927 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7f5ae009a4a0 msgr2=0x7f5ae00957a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:33:55.105 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:55.084+0000 7f5ae5f30700 1 --2- 172.21.15.88:0/1541995927 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7f5ae009a4a0 0x7f5ae00957a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:55.105 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:55.084+0000 7f5ae5f30700 1 -- 172.21.15.88:0/1541995927 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7f5ae0096b90 msgr2=0x7f5ae0095260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:33:55.106 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:55.084+0000 7f5ae5f30700 1 --2- 172.21.15.88:0/1541995927 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7f5ae0096b90 0x7f5ae0095260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:55.106 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:55.084+0000 7f5ae5f30700 10 monclient: _finish_hunting 0 2023-09-30T01:33:55.106 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:55.084+0000 7f5ae5f30700 1 monclient: found mon.noname-e 2023-09-30T01:33:55.106 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:55.084+0000 7f5ae5f30700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:33:55.106 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:55.084+0000 7f5ae5f30700 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.88:3304/0 2023-09-30T01:33:55.106 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:55.084+0000 7f5ae5f30700 1 -- 172.21.15.88:0/1541995927 --> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ad0009420 con 0x7f5ae009e8d0 2023-09-30T01:33:55.106 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:55.084+0000 7f5ae5f30700 1 --2- 172.21.15.88:0/1541995927 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f5ae009e8d0 0x7f5ae006aea0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f5adc00ea20 tx=0x7f5adc007dc0 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:33:55.106 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:55.084+0000 7f5ad57fa700 1 -- 172.21.15.88:0/1541995927 <== mon.8 v2:172.21.15.88:3304/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f5adc00ee00 con 0x7f5ae009e8d0 2023-09-30T01:33:55.106 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:55.084+0000 7f5ad57fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:33:55.106 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:55.084+0000 7f5ad57fa700 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2023-09-30T01:33:55.106 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:55.084+0000 7f5ad57fa700 10 monclient: dump: 2023-09-30T01:33:55.106 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:33:55.107 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:33:55.107 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:55.107 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:55.107 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:33:55.107 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:33:55.107 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:33:55.107 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:33:55.107 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:33:55.107 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:33:55.107 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:33:55.107 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:33:55.107 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:33:55.108 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:33:55.108 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:33:55.108 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:33:55.108 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:55.084+0000 7f5ad57fa700 10 monclient: _finish_auth 0 2023-09-30T01:33:55.108 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:55.084+0000 7f5ad57fa700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:33:55.108 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:55.084+0000 7f5ae6731700 5 monclient: authenticate success, global_id 4499 2023-09-30T01:33:55.108 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:55.084+0000 7f5ae6731700 10 monclient: _renew_subs 2023-09-30T01:33:55.108 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:55.084+0000 7f5ad57fa700 1 -- 172.21.15.88:0/1541995927 <== mon.8 v2:172.21.15.88:3304/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5adc0070f0 con 0x7f5ae009e8d0 2023-09-30T01:33:55.108 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:55.084+0000 7f5ae6731700 10 monclient: _send_mon_message to mon.e at v2:172.21.15.88:3304/0 2023-09-30T01:33:55.108 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:55.084+0000 7f5ae6731700 1 -- 172.21.15.88:0/1541995927 --> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ad8009a20 con 0x7f5ae009e8d0 2023-09-30T01:33:55.108 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:55.084+0000 7f5ad57fa700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:33:55.108 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:55.084+0000 7f5ad57fa700 1 -- 172.21.15.88:0/1541995927 <== mon.8 v2:172.21.15.88:3304/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f5adc0076e0 con 0x7f5ae009e8d0 2023-09-30T01:33:55.108 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:55.084+0000 7f5ad57fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:33:55.109 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:55.084+0000 7f5ad77fe700 4 set_mon_vals no callback set 2023-09-30T01:33:55.109 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:55.084+0000 7f5ad57fa700 10 monclient: got monmap 1 from mon.e (according to old e1) 2023-09-30T01:33:55.109 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:55.084+0000 7f5ad57fa700 10 monclient: dump: 2023-09-30T01:33:55.109 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:33:55.109 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:33:55.109 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:55.109 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:55.109 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:33:55.109 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:33:55.109 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:33:55.109 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:33:55.109 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:33:55.110 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:33:55.110 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:33:55.110 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:33:55.110 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:33:55.110 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:33:55.110 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:33:55.110 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:33:55.110 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:55.084+0000 7f5ae6731700 10 monclient: _renew_subs 2023-09-30T01:33:55.110 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:55.084+0000 7f5ae6731700 10 monclient: _send_mon_message to mon.e at v2:172.21.15.88:3304/0 2023-09-30T01:33:55.110 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:55.084+0000 7f5ae6731700 1 -- 172.21.15.88:0/1541995927 --> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ae006c850 con 0x7f5ae009e8d0 2023-09-30T01:33:55.110 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:55.088+0000 7f5ae6731700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:33:55.110 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:55.088+0000 7f5ae6731700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:33:55.110 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:55.088+0000 7f5ae6731700 10 monclient: _send_mon_message to mon.e at v2:172.21.15.88:3304/0 2023-09-30T01:33:55.111 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:55.088+0000 7f5ad57fa700 1 -- 172.21.15.88:0/1541995927 <== mon.8 v2:172.21.15.88:3304/0 4 ==== mgrmap(e 5) v1 ==== 40144+0+0 (secure 0 0 0) 0x7f5adc007260 con 0x7f5ae009e8d0 2023-09-30T01:33:55.111 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:55.088+0000 7f5ae6731700 1 -- 172.21.15.88:0/1541995927 --> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ae006ea80 con 0x7f5ae009e8d0 2023-09-30T01:33:55.111 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:55.088+0000 7f5ad57fa700 1 --2- 172.21.15.88:0/1541995927 >> v2:172.21.15.107:6812/29926 conn(0x7f5ac4033360 0x7f5ac4035810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:33:55.111 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:55.088+0000 7f5ad57fa700 1 -- 172.21.15.88:0/1541995927 <== mon.8 v2:172.21.15.88:3304/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 3502+0+0 (secure 0 0 0) 0x7f5adc045730 con 0x7f5ae009e8d0 2023-09-30T01:33:55.111 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:55.088+0000 7f5ae572f700 1 --2- 172.21.15.88:0/1541995927 >> v2:172.21.15.107:6812/29926 conn(0x7f5ac4033360 0x7f5ac4035810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:33:55.111 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:55.088+0000 7f5ae572f700 10 monclient: get_auth_request con 0x7f5ac4033360 auth_method 0 2023-09-30T01:33:55.111 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:55.088+0000 7f5ae572f700 1 --2- 172.21.15.88:0/1541995927 >> v2:172.21.15.107:6812/29926 conn(0x7f5ac4033360 0x7f5ac4035810 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f5ad0009ea0 tx=0x7f5ad00091a0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:33:55.111 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:55.092+0000 7f5ad57fa700 1 -- 172.21.15.88:0/1541995927 <== mon.8 v2:172.21.15.88:3304/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f5adc007510 con 0x7f5ae009e8d0 2023-09-30T01:33:55.111 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:55.092+0000 7f5ad57fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:33:55.111 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:55.092+0000 7f5ad57fa700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:33:55.290 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:55.284+0000 7f5ae6731700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T01:33:55.291 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:55.284+0000 7f5ae6731700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:33:55.291 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:55.284+0000 7f5ae6731700 10 monclient: _send_mon_message to mon.e at v2:172.21.15.88:3304/0 2023-09-30T01:33:55.291 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:55.284+0000 7f5ae6731700 1 -- 172.21.15.88:0/1541995927 --> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f5ae00710d0 con 0x7f5ae009e8d0 2023-09-30T01:33:55.291 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:55.284+0000 7f5ad57fa700 1 -- 172.21.15.88:0/1541995927 <== mon.8 v2:172.21.15.88:3304/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f5adc018b50 con 0x7f5ae009e8d0 2023-09-30T01:33:55.291 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:55.284+0000 7f5ad57fa700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:33:55.291 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:55.284+0000 7f5ad57fa700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:33:55.291 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-09-30T01:33:55.292 INFO:teuthology.orchestra.run.smithi088.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:33:55.293 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:55.288+0000 7f5ae6731700 1 -- 172.21.15.88:0/1541995927 >> v2:172.21.15.107:6812/29926 conn(0x7f5ac4033360 msgr2=0x7f5ac4035810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:33:55.293 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:55.288+0000 7f5ae6731700 1 --2- 172.21.15.88:0/1541995927 >> v2:172.21.15.107:6812/29926 conn(0x7f5ac4033360 0x7f5ac4035810 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f5ad0009ea0 tx=0x7f5ad00091a0 comp rx=0 tx=0).stop 2023-09-30T01:33:55.293 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:55.288+0000 7f5ae6731700 10 monclient: shutdown 2023-09-30T01:33:55.294 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:55.288+0000 7f5ae6731700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:33:55.294 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:55.288+0000 7f5ae6731700 1 -- 172.21.15.88:0/1541995927 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f5ae009e8d0 msgr2=0x7f5ae006aea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:33:55.294 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:55.288+0000 7f5ae6731700 1 --2- 172.21.15.88:0/1541995927 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f5ae009e8d0 0x7f5ae006aea0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f5adc00ea20 tx=0x7f5adc007dc0 comp rx=0 tx=0).stop 2023-09-30T01:33:55.294 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:55.288+0000 7f5ae6731700 1 -- 172.21.15.88:0/1541995927 shutdown_connections 2023-09-30T01:33:55.294 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:55.288+0000 7f5ae6731700 1 --2- 172.21.15.88:0/1541995927 >> v2:172.21.15.107:6812/29926 conn(0x7f5ac4033360 0x7f5ac4035810 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:55.294 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:55.288+0000 7f5ae6731700 1 --2- 172.21.15.88:0/1541995927 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f5ae009e8d0 0x7f5ae006aea0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:55.294 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:55.288+0000 7f5ae6731700 1 --2- 172.21.15.88:0/1541995927 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7f5ae009a4a0 0x7f5ae00957a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:55.294 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:55.288+0000 7f5ae6731700 1 --2- 172.21.15.88:0/1541995927 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7f5ae0096b90 0x7f5ae0095260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:55.294 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:55.288+0000 7f5ae6731700 1 -- 172.21.15.88:0/1541995927 >> 172.21.15.88:0/1541995927 conn(0x7f5ae00a2540 msgr2=0x7f5ae0132c30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:33:55.294 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:55.288+0000 7f5ae6731700 1 -- 172.21.15.88:0/1541995927 shutdown_connections 2023-09-30T01:33:55.294 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:55.288+0000 7f5ae6731700 1 -- 172.21.15.88:0/1541995927 wait complete. 2023-09-30T01:33:58.306 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:33:58.409 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:58.404+0000 7ffb60988700 10 monclient: get_monmap_and_config 2023-09-30T01:33:58.409 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:58.404+0000 7ffb60988700 10 monclient: build_initial_monmap 2023-09-30T01:33:58.409 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:58.404+0000 7ffb60988700 10 monclient: monmap: 2023-09-30T01:33:58.409 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2023-09-30T01:33:58.410 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:33:58.410 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:33:58.409767+0000 2023-09-30T01:33:58.410 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:33:58.409767+0000 2023-09-30T01:33:58.410 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2023-09-30T01:33:58.410 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:33:58.410 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a-legacy 2023-09-30T01:33:58.410 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.107:6789/0 mon.noname-f-legacy 2023-09-30T01:33:58.410 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.noname-b 2023-09-30T01:33:58.410 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.noname-g 2023-09-30T01:33:58.410 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.noname-c 2023-09-30T01:33:58.410 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.noname-h 2023-09-30T01:33:58.410 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.noname-d 2023-09-30T01:33:58.410 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.noname-i 2023-09-30T01:33:58.411 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.noname-e 2023-09-30T01:33:58.411 INFO:teuthology.orchestra.run.smithi088.stderr:9: v2:172.21.15.88:3300/0 mon.noname-a 2023-09-30T01:33:58.411 INFO:teuthology.orchestra.run.smithi088.stderr:10: v2:172.21.15.107:3300/0 mon.noname-f 2023-09-30T01:33:58.411 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:33:58.421 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:58.404+0000 7ffb60988700 1 Processor -- start 2023-09-30T01:33:58.422 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:58.404+0000 7ffb60988700 1 -- start start 2023-09-30T01:33:58.422 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:58.404+0000 7ffb60988700 10 monclient: init 2023-09-30T01:33:58.422 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:58.404+0000 7ffb60988700 10 monclient: _reopen_session rank -1 2023-09-30T01:33:58.422 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:58.404+0000 7ffb60988700 10 monclient: _add_conns ranks=[9,4,10,5,6,0,2,7,8,3,1] 2023-09-30T01:33:58.422 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:58.404+0000 7ffb60988700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7ffb5c0976f0 0x7ffb5c098de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:33:58.422 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:58.404+0000 7ffb60988700 10 monclient(hunting): picked mon.noname-a con 0x7ffb5c0976f0 addr v2:172.21.15.88:3300/0 2023-09-30T01:33:58.422 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:58.404+0000 7ffb60988700 1 --2- >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7ffb5c09ed80 0x7ffb5c09f150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:33:58.422 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:58.404+0000 7ffb60988700 10 monclient(hunting): picked mon.noname-c con 0x7ffb5c09ed80 addr [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] 2023-09-30T01:33:58.422 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:58.404+0000 7ffb60988700 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7ffb5c09f690 0x7ffb5c130570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:33:58.422 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:58.404+0000 7ffb60988700 10 monclient(hunting): picked mon.noname-f con 0x7ffb5c09f690 addr v2:172.21.15.107:3300/0 2023-09-30T01:33:58.422 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:58.404+0000 7ffb60988700 10 monclient(hunting): start opening mon connection 2023-09-30T01:33:58.423 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:58.404+0000 7ffb60988700 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffb5c099320 con 0x7ffb5c0976f0 2023-09-30T01:33:58.423 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:58.404+0000 7ffb60988700 10 monclient(hunting): start opening mon connection 2023-09-30T01:33:58.423 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:58.404+0000 7ffb60988700 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffb5c098400 con 0x7ffb5c09f690 2023-09-30T01:33:58.423 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:58.404+0000 7ffb60988700 10 monclient(hunting): start opening mon connection 2023-09-30T01:33:58.423 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:58.404+0000 7ffb60988700 1 -- --> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] -- mon_getmap magic: 0 v1 -- 0x7ffb5c098a30 con 0x7ffb5c09ed80 2023-09-30T01:33:58.423 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:58.404+0000 7ffb60988700 10 monclient(hunting): _renew_subs 2023-09-30T01:33:58.423 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:58.404+0000 7ffb60988700 10 monclient(hunting): authenticate will time out at 910.447866s 2023-09-30T01:33:58.423 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:58.408+0000 7ffb5ad9d700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7ffb5c0976f0 0x7ffb5c098de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:33:58.423 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:58.408+0000 7ffb5259c700 1 --2- >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7ffb5c09ed80 0x7ffb5c09f150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:33:58.423 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:58.408+0000 7ffb5ad9d700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7ffb5c0976f0 0x7ffb5c098de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:39578/0 (socket says 172.21.15.88:39578) 2023-09-30T01:33:58.423 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:58.408+0000 7ffb5ad9d700 1 -- 172.21.15.88:0/806295774 learned_addr learned my addr 172.21.15.88:0/806295774 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-09-30T01:33:58.423 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:58.408+0000 7ffb5ad9d700 10 monclient(hunting): get_auth_request con 0x7ffb5c0976f0 auth_method 0 2023-09-30T01:33:58.424 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:58.408+0000 7ffb5ad9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:33:58.424 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:58.408+0000 7ffb5ad9d700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:33:58.424 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:58.408+0000 7ffb5ad9d700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:33:58.424 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:58.408+0000 7ffb5259c700 10 monclient(hunting): get_auth_request con 0x7ffb5c09ed80 auth_method 0 2023-09-30T01:33:58.424 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:58.408+0000 7ffb5259c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:33:58.424 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:58.408+0000 7ffb5259c700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:33:58.424 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:58.408+0000 7ffb5259c700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:33:58.424 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:58.408+0000 7ffb5b59e700 1 --2- 172.21.15.88:0/806295774 >> v2:172.21.15.107:3300/0 conn(0x7ffb5c09f690 0x7ffb5c130570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:33:58.424 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:58.408+0000 7ffb5259c700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:33:58.425 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:58.408+0000 7ffb5259c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:33:58.425 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:58.408+0000 7ffb5259c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:33:58.425 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:58.408+0000 7ffb5b59e700 10 monclient(hunting): get_auth_request con 0x7ffb5c09f690 auth_method 0 2023-09-30T01:33:58.425 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:58.408+0000 7ffb5b59e700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:33:58.425 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:58.408+0000 7ffb5b59e700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:33:58.425 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:58.408+0000 7ffb5b59e700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:33:58.425 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:58.408+0000 7ffb5ad9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:33:58.425 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:58.408+0000 7ffb5ad9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:33:58.425 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:58.408+0000 7ffb5ad9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:33:58.425 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:58.408+0000 7ffb5b59e700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:33:58.425 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:58.408+0000 7ffb5b59e700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:33:58.426 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:58.408+0000 7ffb5b59e700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:33:58.426 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:58.408+0000 7ffb5259c700 10 monclient(hunting): handle_auth_done global_id 4468 payload 274 2023-09-30T01:33:58.426 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:58.408+0000 7ffb5259c700 1 -- 172.21.15.88:0/806295774 >> v2:172.21.15.107:3300/0 conn(0x7ffb5c09f690 msgr2=0x7ffb5c130570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:33:58.426 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:58.408+0000 7ffb5259c700 1 --2- 172.21.15.88:0/806295774 >> v2:172.21.15.107:3300/0 conn(0x7ffb5c09f690 0x7ffb5c130570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:58.426 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:58.408+0000 7ffb5259c700 1 -- 172.21.15.88:0/806295774 >> v2:172.21.15.88:3300/0 conn(0x7ffb5c0976f0 msgr2=0x7ffb5c098de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:33:58.426 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:58.408+0000 7ffb5259c700 1 --2- 172.21.15.88:0/806295774 >> v2:172.21.15.88:3300/0 conn(0x7ffb5c0976f0 0x7ffb5c098de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:58.426 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:58.408+0000 7ffb5259c700 10 monclient: _finish_hunting 0 2023-09-30T01:33:58.426 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:58.408+0000 7ffb5259c700 1 monclient: found mon.noname-c 2023-09-30T01:33:58.426 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:58.408+0000 7ffb5259c700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:33:58.426 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:58.408+0000 7ffb5259c700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.88:3302/0 2023-09-30T01:33:58.427 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:58.408+0000 7ffb5259c700 1 -- 172.21.15.88:0/806295774 --> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb5c04bb20 con 0x7ffb5c09ed80 2023-09-30T01:33:58.427 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:58.408+0000 7ffb5259c700 1 --2- 172.21.15.88:0/806295774 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7ffb5c09ed80 0x7ffb5c09f150 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7ffb440098e0 tx=0x7ffb440091a0 comp rx=0 tx=0).ready entity=mon.4 client_cookie=f2f6e97194d85aaa server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:33:58.427 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:58.408+0000 7ffb5a59c700 1 -- 172.21.15.88:0/806295774 <== mon.4 v2:172.21.15.88:3302/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7ffb44011ab0 con 0x7ffb5c09ed80 2023-09-30T01:33:58.427 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:58.408+0000 7ffb5a59c700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:33:58.427 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:58.408+0000 7ffb5a59c700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T01:33:58.427 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:58.408+0000 7ffb5a59c700 10 monclient: dump: 2023-09-30T01:33:58.427 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:33:58.427 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:33:58.427 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:58.427 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:58.427 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:33:58.428 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:33:58.428 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:33:58.428 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:33:58.428 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:33:58.428 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:33:58.428 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:33:58.428 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:33:58.428 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:33:58.428 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:33:58.428 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:33:58.428 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:33:58.429 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:58.408+0000 7ffb5a59c700 10 monclient: _finish_auth 0 2023-09-30T01:33:58.429 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:58.408+0000 7ffb5a59c700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:33:58.429 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:58.408+0000 7ffb60988700 5 monclient: authenticate success, global_id 4468 2023-09-30T01:33:58.429 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:58.408+0000 7ffb60988700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:33:58.429 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:58.408+0000 7ffb5a59c700 1 -- 172.21.15.88:0/806295774 <== mon.4 v2:172.21.15.88:3302/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb44011c30 con 0x7ffb5c09ed80 2023-09-30T01:33:58.429 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:58.408+0000 7ffb5a59c700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:33:58.429 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:58.408+0000 7ffb60988700 10 monclient: get_monmap_and_config success 2023-09-30T01:33:58.429 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:58.408+0000 7ffb60988700 4 set_mon_vals no callback set 2023-09-30T01:33:58.429 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:58.408+0000 7ffb5a59c700 1 -- 172.21.15.88:0/806295774 <== mon.4 v2:172.21.15.88:3302/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7ffb44011da0 con 0x7ffb5c09ed80 2023-09-30T01:33:58.429 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:58.408+0000 7ffb5a59c700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:33:58.429 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:58.408+0000 7ffb5a59c700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T01:33:58.430 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:58.408+0000 7ffb5a59c700 10 monclient: dump: 2023-09-30T01:33:58.430 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:33:58.430 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:33:58.430 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:58.430 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:58.430 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:33:58.430 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:33:58.430 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:33:58.431 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:33:58.431 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:33:58.431 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:33:58.431 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:33:58.431 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:33:58.431 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:33:58.431 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:33:58.431 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:33:58.431 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:33:58.431 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:58.408+0000 7ffb60988700 10 monclient: shutdown 2023-09-30T01:33:58.432 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:58.408+0000 7ffb60988700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:33:58.432 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:58.408+0000 7ffb60988700 1 -- 172.21.15.88:0/806295774 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7ffb5c09ed80 msgr2=0x7ffb5c09f150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:33:58.432 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:58.408+0000 7ffb60988700 1 --2- 172.21.15.88:0/806295774 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7ffb5c09ed80 0x7ffb5c09f150 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7ffb440098e0 tx=0x7ffb440091a0 comp rx=0 tx=0).stop 2023-09-30T01:33:58.432 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:58.408+0000 7ffb60988700 1 -- 172.21.15.88:0/806295774 shutdown_connections 2023-09-30T01:33:58.432 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:58.408+0000 7ffb60988700 1 --2- 172.21.15.88:0/806295774 >> v2:172.21.15.107:3300/0 conn(0x7ffb5c09f690 0x7ffb5c130570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:58.432 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:58.408+0000 7ffb60988700 1 --2- 172.21.15.88:0/806295774 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7ffb5c09ed80 0x7ffb5c09f150 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:58.432 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:58.408+0000 7ffb60988700 1 --2- 172.21.15.88:0/806295774 >> v2:172.21.15.88:3300/0 conn(0x7ffb5c0976f0 0x7ffb5c098de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:58.432 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:58.408+0000 7ffb60988700 1 -- 172.21.15.88:0/806295774 >> 172.21.15.88:0/806295774 conn(0x7ffb5c0a0910 msgr2=0x7ffb5c0a33c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:33:58.432 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:58.408+0000 7ffb60988700 1 -- 172.21.15.88:0/806295774 shutdown_connections 2023-09-30T01:33:58.432 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:58.408+0000 7ffb60988700 1 -- 172.21.15.88:0/806295774 wait complete. 2023-09-30T01:33:58.433 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:58.408+0000 7ffb60988700 10 monclient: build_initial_monmap 2023-09-30T01:33:58.433 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:58.408+0000 7ffb60988700 10 monclient: monmap: 2023-09-30T01:33:58.433 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2023-09-30T01:33:58.433 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:33:58.433 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 0.000000 2023-09-30T01:33:58.433 INFO:teuthology.orchestra.run.smithi088.stderr:created 0.000000 2023-09-30T01:33:58.433 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2023-09-30T01:33:58.433 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 1 2023-09-30T01:33:58.433 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.noname-a 2023-09-30T01:33:58.433 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.noname-f 2023-09-30T01:33:58.433 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.noname-b 2023-09-30T01:33:58.434 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.noname-g 2023-09-30T01:33:58.434 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.noname-c 2023-09-30T01:33:58.434 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.noname-h 2023-09-30T01:33:58.434 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.noname-d 2023-09-30T01:33:58.434 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.noname-i 2023-09-30T01:33:58.434 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.noname-e 2023-09-30T01:33:58.434 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:33:58.434 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:58.408+0000 7ffb60988700 1 Processor -- start 2023-09-30T01:33:58.434 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:58.408+0000 7ffb60988700 1 -- start start 2023-09-30T01:33:58.435 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:58.408+0000 7ffb60988700 10 monclient: init 2023-09-30T01:33:58.435 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:58.412+0000 7ffb60988700 10 monclient: _reopen_session rank -1 2023-09-30T01:33:58.435 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:58.412+0000 7ffb60988700 10 monclient: _add_conns ranks=[7,0,3,6,5,8,4,2,1] 2023-09-30T01:33:58.435 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:58.412+0000 7ffb60988700 1 --2- >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7ffb5c0976f0 0x7ffb5c128560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:33:58.435 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:58.412+0000 7ffb60988700 10 monclient(hunting): picked mon.noname-i con 0x7ffb5c0976f0 addr [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] 2023-09-30T01:33:58.435 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:58.412+0000 7ffb60988700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ffb5c09ed80 0x7ffb5c06deb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:33:58.435 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:58.412+0000 7ffb60988700 10 monclient(hunting): picked mon.noname-a con 0x7ffb5c09ed80 addr [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] 2023-09-30T01:33:58.435 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:58.412+0000 7ffb60988700 1 --2- >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7ffb5c09f690 0x7ffb5c12e8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:33:58.435 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:58.412+0000 7ffb60988700 10 monclient(hunting): picked mon.noname-g con 0x7ffb5c09f690 addr [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] 2023-09-30T01:33:58.435 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:58.412+0000 7ffb60988700 10 monclient(hunting): start opening mon connection 2023-09-30T01:33:58.435 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:58.412+0000 7ffb60988700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb5c06e570 con 0x7ffb5c09ed80 2023-09-30T01:33:58.436 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:58.412+0000 7ffb60988700 10 monclient(hunting): start opening mon connection 2023-09-30T01:33:58.436 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:58.412+0000 7ffb60988700 1 -- --> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb5c097d80 con 0x7ffb5c09f690 2023-09-30T01:33:58.436 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:58.412+0000 7ffb60988700 10 monclient(hunting): start opening mon connection 2023-09-30T01:33:58.436 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:58.412+0000 7ffb60988700 1 -- --> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] -- mon_getmap magic: 0 v1 -- 0x7ffb5c06da10 con 0x7ffb5c0976f0 2023-09-30T01:33:58.436 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:58.412+0000 7ffb60988700 10 monclient(hunting): _renew_subs 2023-09-30T01:33:58.436 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:58.412+0000 7ffb60988700 10 monclient(hunting): authenticate will time out at 910.452690s 2023-09-30T01:33:58.436 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:58.412+0000 7ffb5259c700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ffb5c09ed80 0x7ffb5c06deb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:33:58.436 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:58.412+0000 7ffb5259c700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ffb5c09ed80 0x7ffb5c06deb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:39584/0 (socket says 172.21.15.88:39584) 2023-09-30T01:33:58.436 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:58.412+0000 7ffb5259c700 1 -- 172.21.15.88:0/1259613086 learned_addr learned my addr 172.21.15.88:0/1259613086 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-09-30T01:33:58.436 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:58.412+0000 7ffb5259c700 10 monclient(hunting): get_auth_request con 0x7ffb5c09ed80 auth_method 0 2023-09-30T01:33:58.437 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:58.412+0000 7ffb5259c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:33:58.437 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:58.412+0000 7ffb5259c700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:33:58.437 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:58.412+0000 7ffb5259c700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:33:58.437 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:58.412+0000 7ffb5ad9d700 1 --2- 172.21.15.88:0/1259613086 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7ffb5c0976f0 0x7ffb5c128560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:33:58.437 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:58.412+0000 7ffb5b59e700 1 --2- 172.21.15.88:0/1259613086 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7ffb5c09f690 0x7ffb5c12e8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:33:58.437 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:58.412+0000 7ffb5ad9d700 10 monclient(hunting): get_auth_request con 0x7ffb5c0976f0 auth_method 0 2023-09-30T01:33:58.437 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:58.412+0000 7ffb5ad9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:33:58.437 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:58.412+0000 7ffb5ad9d700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:33:58.437 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:58.412+0000 7ffb5ad9d700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:33:58.437 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:58.412+0000 7ffb5259c700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:33:58.437 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:58.412+0000 7ffb5259c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:33:58.438 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:58.412+0000 7ffb5259c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:33:58.438 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:58.412+0000 7ffb5b59e700 10 monclient(hunting): get_auth_request con 0x7ffb5c09f690 auth_method 0 2023-09-30T01:33:58.438 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:58.412+0000 7ffb5b59e700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:33:58.438 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:58.412+0000 7ffb5b59e700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:33:58.438 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:58.412+0000 7ffb5b59e700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:33:58.438 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:58.412+0000 7ffb5ad9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:33:58.438 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:58.412+0000 7ffb5ad9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:33:58.438 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:58.412+0000 7ffb5ad9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:33:58.438 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:58.412+0000 7ffb5b59e700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:33:58.438 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:58.412+0000 7ffb5b59e700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:33:58.438 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:58.412+0000 7ffb5b59e700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:33:58.439 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:58.412+0000 7ffb5259c700 10 monclient(hunting): handle_auth_done global_id 4545 payload 824 2023-09-30T01:33:58.439 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:58.412+0000 7ffb5259c700 1 -- 172.21.15.88:0/1259613086 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7ffb5c0976f0 msgr2=0x7ffb5c128560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:33:58.439 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:58.412+0000 7ffb5259c700 1 --2- 172.21.15.88:0/1259613086 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7ffb5c0976f0 0x7ffb5c128560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:58.439 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:58.412+0000 7ffb5259c700 1 -- 172.21.15.88:0/1259613086 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7ffb5c09f690 msgr2=0x7ffb5c12e8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:33:58.439 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:58.412+0000 7ffb5259c700 1 --2- 172.21.15.88:0/1259613086 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7ffb5c09f690 0x7ffb5c12e8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:58.439 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:58.412+0000 7ffb5259c700 10 monclient: _finish_hunting 0 2023-09-30T01:33:58.439 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:58.412+0000 7ffb5259c700 1 monclient: found mon.noname-a 2023-09-30T01:33:58.439 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:58.412+0000 7ffb5259c700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:33:58.439 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:58.412+0000 7ffb5259c700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.88:3300/0 2023-09-30T01:33:58.439 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:58.412+0000 7ffb5259c700 1 -- 172.21.15.88:0/1259613086 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb44009420 con 0x7ffb5c09ed80 2023-09-30T01:33:58.440 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:58.412+0000 7ffb5259c700 1 --2- 172.21.15.88:0/1259613086 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ffb5c09ed80 0x7ffb5c06deb0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7ffb5c098b70 tx=0x7ffb44017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:33:58.440 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:58.412+0000 7ffb537fe700 1 -- 172.21.15.88:0/1259613086 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7ffb4401f070 con 0x7ffb5c09ed80 2023-09-30T01:33:58.440 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:58.412+0000 7ffb537fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:33:58.440 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:58.412+0000 7ffb537fe700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T01:33:58.440 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:58.412+0000 7ffb537fe700 10 monclient: dump: 2023-09-30T01:33:58.440 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:33:58.440 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:33:58.440 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:58.440 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:58.440 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:33:58.440 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:33:58.440 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:33:58.441 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:33:58.441 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:33:58.441 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:33:58.441 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:33:58.441 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:33:58.441 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:33:58.441 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:33:58.441 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:33:58.441 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:33:58.441 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:58.412+0000 7ffb537fe700 10 monclient: _finish_auth 0 2023-09-30T01:33:58.441 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:58.412+0000 7ffb537fe700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:33:58.441 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:58.412+0000 7ffb537fe700 1 -- 172.21.15.88:0/1259613086 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb440193b0 con 0x7ffb5c09ed80 2023-09-30T01:33:58.442 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:58.412+0000 7ffb60988700 5 monclient: authenticate success, global_id 4545 2023-09-30T01:33:58.442 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:58.412+0000 7ffb60988700 10 monclient: _renew_subs 2023-09-30T01:33:58.442 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:58.412+0000 7ffb60988700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.88:3300/0 2023-09-30T01:33:58.442 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:58.412+0000 7ffb60988700 1 -- 172.21.15.88:0/1259613086 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb5c06f160 con 0x7ffb5c09ed80 2023-09-30T01:33:58.442 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:58.412+0000 7ffb537fe700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:33:58.442 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:58.412+0000 7ffb537fe700 1 -- 172.21.15.88:0/1259613086 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7ffb440199a0 con 0x7ffb5c09ed80 2023-09-30T01:33:58.442 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:58.412+0000 7ffb537fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:33:58.442 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:58.412+0000 7ffb5959a700 4 set_mon_vals no callback set 2023-09-30T01:33:58.442 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:58.412+0000 7ffb537fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:33:58.442 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:58.412+0000 7ffb537fe700 10 monclient: dump: 2023-09-30T01:33:58.442 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:33:58.442 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:33:58.443 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:58.443 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:33:58.443 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:33:58.443 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:33:58.443 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:33:58.443 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:33:58.443 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:33:58.443 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:33:58.443 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:33:58.443 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:33:58.443 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:33:58.444 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:33:58.444 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:33:58.444 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:33:58.444 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:58.412+0000 7ffb60988700 10 monclient: _renew_subs 2023-09-30T01:33:58.444 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:58.412+0000 7ffb60988700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.88:3300/0 2023-09-30T01:33:58.444 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:58.412+0000 7ffb60988700 1 -- 172.21.15.88:0/1259613086 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb5c12ede0 con 0x7ffb5c09ed80 2023-09-30T01:33:58.444 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:58.412+0000 7ffb537fe700 1 -- 172.21.15.88:0/1259613086 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 40144+0+0 (secure 0 0 0) 0x7ffb44019520 con 0x7ffb5c09ed80 2023-09-30T01:33:58.444 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:58.412+0000 7ffb60988700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:33:58.444 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:58.412+0000 7ffb60988700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:33:58.444 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:58.412+0000 7ffb60988700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.88:3300/0 2023-09-30T01:33:58.444 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:58.412+0000 7ffb60988700 1 -- 172.21.15.88:0/1259613086 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffb5c071960 con 0x7ffb5c09ed80 2023-09-30T01:33:58.444 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:58.412+0000 7ffb537fe700 1 --2- 172.21.15.88:0/1259613086 >> v2:172.21.15.107:6812/29926 conn(0x7ffb3c033360 0x7ffb3c035810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:33:58.445 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:58.412+0000 7ffb537fe700 1 -- 172.21.15.88:0/1259613086 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 3502+0+0 (secure 0 0 0) 0x7ffb440443e0 con 0x7ffb5c09ed80 2023-09-30T01:33:58.445 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:58.416+0000 7ffb5ad9d700 1 --2- 172.21.15.88:0/1259613086 >> v2:172.21.15.107:6812/29926 conn(0x7ffb3c033360 0x7ffb3c035810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:33:58.445 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:58.416+0000 7ffb5ad9d700 10 monclient: get_auth_request con 0x7ffb3c033360 auth_method 0 2023-09-30T01:33:58.445 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:58.416+0000 7ffb5ad9d700 1 --2- 172.21.15.88:0/1259613086 >> v2:172.21.15.107:6812/29926 conn(0x7ffb3c033360 0x7ffb3c035810 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7ffb4c006d40 tx=0x7ffb4c009250 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:33:58.445 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:58.424+0000 7ffb537fe700 1 -- 172.21.15.88:0/1259613086 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7ffb44021310 con 0x7ffb5c09ed80 2023-09-30T01:33:58.446 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:58.424+0000 7ffb537fe700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:33:58.446 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:58.424+0000 7ffb537fe700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:33:58.627 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:58.624+0000 7ffb60988700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T01:33:58.628 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:58.624+0000 7ffb60988700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:33:58.628 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:58.624+0000 7ffb60988700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.88:3300/0 2023-09-30T01:33:58.628 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:58.624+0000 7ffb60988700 1 -- 172.21.15.88:0/1259613086 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7ffb5c0a34d0 con 0x7ffb5c09ed80 2023-09-30T01:33:58.628 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:58.624+0000 7ffb537fe700 1 -- 172.21.15.88:0/1259613086 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7ffb44015070 con 0x7ffb5c09ed80 2023-09-30T01:33:58.628 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:58.624+0000 7ffb537fe700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:33:58.628 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:58.624+0000 7ffb537fe700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:33:58.629 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-09-30T01:33:58.629 INFO:teuthology.orchestra.run.smithi088.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:33:58.631 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:58.624+0000 7ffb60988700 1 -- 172.21.15.88:0/1259613086 >> v2:172.21.15.107:6812/29926 conn(0x7ffb3c033360 msgr2=0x7ffb3c035810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:33:58.631 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:58.624+0000 7ffb60988700 1 --2- 172.21.15.88:0/1259613086 >> v2:172.21.15.107:6812/29926 conn(0x7ffb3c033360 0x7ffb3c035810 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7ffb4c006d40 tx=0x7ffb4c009250 comp rx=0 tx=0).stop 2023-09-30T01:33:58.631 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:58.624+0000 7ffb60988700 10 monclient: shutdown 2023-09-30T01:33:58.631 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:58.624+0000 7ffb60988700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:33:58.631 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:58.624+0000 7ffb60988700 1 -- 172.21.15.88:0/1259613086 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ffb5c09ed80 msgr2=0x7ffb5c06deb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:33:58.631 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:58.624+0000 7ffb60988700 1 --2- 172.21.15.88:0/1259613086 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ffb5c09ed80 0x7ffb5c06deb0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7ffb5c098b70 tx=0x7ffb44017000 comp rx=0 tx=0).stop 2023-09-30T01:33:58.631 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:58.624+0000 7ffb60988700 1 -- 172.21.15.88:0/1259613086 shutdown_connections 2023-09-30T01:33:58.631 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:58.624+0000 7ffb60988700 1 --2- 172.21.15.88:0/1259613086 >> v2:172.21.15.107:6812/29926 conn(0x7ffb3c033360 0x7ffb3c035810 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:58.631 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:58.624+0000 7ffb60988700 1 --2- 172.21.15.88:0/1259613086 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7ffb5c09f690 0x7ffb5c12e8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:58.632 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:58.624+0000 7ffb60988700 1 --2- 172.21.15.88:0/1259613086 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ffb5c09ed80 0x7ffb5c06deb0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:58.632 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:58.624+0000 7ffb60988700 1 --2- 172.21.15.88:0/1259613086 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7ffb5c0976f0 0x7ffb5c128560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:33:58.632 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:58.624+0000 7ffb60988700 1 -- 172.21.15.88:0/1259613086 >> 172.21.15.88:0/1259613086 conn(0x7ffb5c0a0910 msgr2=0x7ffb5c062c70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:33:58.632 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:58.624+0000 7ffb60988700 1 -- 172.21.15.88:0/1259613086 shutdown_connections 2023-09-30T01:33:58.632 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:33:58.624+0000 7ffb60988700 1 -- 172.21.15.88:0/1259613086 wait complete. 2023-09-30T01:34:01.642 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:34:01.753 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:01.748+0000 7fd0cae48700 10 monclient: get_monmap_and_config 2023-09-30T01:34:01.754 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:01.748+0000 7fd0cae48700 10 monclient: build_initial_monmap 2023-09-30T01:34:01.754 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:01.748+0000 7fd0cae48700 10 monclient: monmap: 2023-09-30T01:34:01.754 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2023-09-30T01:34:01.754 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:34:01.754 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:34:01.754240+0000 2023-09-30T01:34:01.754 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:34:01.754240+0000 2023-09-30T01:34:01.754 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2023-09-30T01:34:01.754 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:34:01.754 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a-legacy 2023-09-30T01:34:01.754 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.107:6789/0 mon.noname-f-legacy 2023-09-30T01:34:01.755 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.noname-b 2023-09-30T01:34:01.755 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.noname-g 2023-09-30T01:34:01.755 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.noname-c 2023-09-30T01:34:01.755 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.noname-h 2023-09-30T01:34:01.755 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.noname-d 2023-09-30T01:34:01.755 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.noname-i 2023-09-30T01:34:01.755 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.noname-e 2023-09-30T01:34:01.755 INFO:teuthology.orchestra.run.smithi088.stderr:9: v2:172.21.15.88:3300/0 mon.noname-a 2023-09-30T01:34:01.755 INFO:teuthology.orchestra.run.smithi088.stderr:10: v2:172.21.15.107:3300/0 mon.noname-f 2023-09-30T01:34:01.755 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:34:01.760 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:01.748+0000 7fd0cae48700 1 Processor -- start 2023-09-30T01:34:01.760 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:01.748+0000 7fd0cae48700 1 -- start start 2023-09-30T01:34:01.760 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:01.748+0000 7fd0cae48700 10 monclient: init 2023-09-30T01:34:01.760 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:01.748+0000 7fd0cae48700 10 monclient: _reopen_session rank -1 2023-09-30T01:34:01.760 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:01.748+0000 7fd0cae48700 10 monclient: _add_conns ranks=[7,9,10,0,5,3,6,4,8,1,2] 2023-09-30T01:34:01.760 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:01.748+0000 7fd0cae48700 1 --2- >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7fd0c4097a40 0x7fd0c4132700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:34:01.760 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:01.748+0000 7fd0cae48700 10 monclient(hunting): picked mon.noname-i con 0x7fd0c4097a40 addr [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] 2023-09-30T01:34:01.760 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:01.748+0000 7fd0cae48700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fd0c41311c0 0x7fd0c4132c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:34:01.761 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:01.748+0000 7fd0cae48700 10 monclient(hunting): picked mon.noname-a con 0x7fd0c41311c0 addr v2:172.21.15.88:3300/0 2023-09-30T01:34:01.761 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:01.748+0000 7fd0cae48700 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fd0c41332e0 0x7fd0c4133740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:34:01.761 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:01.748+0000 7fd0cae48700 10 monclient(hunting): picked mon.noname-f con 0x7fd0c41332e0 addr v2:172.21.15.107:3300/0 2023-09-30T01:34:01.761 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:01.748+0000 7fd0cae48700 10 monclient(hunting): start opening mon connection 2023-09-30T01:34:01.761 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:01.748+0000 7fd0cae48700 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd0c4131590 con 0x7fd0c41311c0 2023-09-30T01:34:01.761 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:01.748+0000 7fd0cae48700 10 monclient(hunting): start opening mon connection 2023-09-30T01:34:01.761 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:01.748+0000 7fd0cae48700 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd0c409a5d0 con 0x7fd0c41332e0 2023-09-30T01:34:01.761 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:01.748+0000 7fd0cae48700 10 monclient(hunting): start opening mon connection 2023-09-30T01:34:01.761 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:01.748+0000 7fd0cae48700 1 -- --> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] -- mon_getmap magic: 0 v1 -- 0x7fd0c4097e10 con 0x7fd0c4097a40 2023-09-30T01:34:01.761 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:01.748+0000 7fd0cae48700 10 monclient(hunting): _renew_subs 2023-09-30T01:34:01.761 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:01.748+0000 7fd0cae48700 10 monclient(hunting): authenticate will time out at 913.791697s 2023-09-30T01:34:01.762 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:01.748+0000 7fd0c9645700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fd0c41311c0 0x7fd0c4132c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:34:01.762 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:01.748+0000 7fd0c9645700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fd0c41311c0 0x7fd0c4132c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:54508/0 (socket says 172.21.15.88:54508) 2023-09-30T01:34:01.762 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:01.748+0000 7fd0c9645700 1 -- 172.21.15.88:0/3153929573 learned_addr learned my addr 172.21.15.88:0/3153929573 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-09-30T01:34:01.762 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:01.748+0000 7fd0c9645700 10 monclient(hunting): get_auth_request con 0x7fd0c41311c0 auth_method 0 2023-09-30T01:34:01.762 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:01.748+0000 7fd0c9645700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:34:01.762 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:01.748+0000 7fd0c9645700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:34:01.762 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:01.748+0000 7fd0c9645700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:34:01.762 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:01.752+0000 7fd0c9645700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:34:01.762 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:01.752+0000 7fd0c9645700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:34:01.762 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:01.752+0000 7fd0c9645700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:34:01.762 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:01.752+0000 7fd0c9e46700 1 --2- 172.21.15.88:0/3153929573 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7fd0c4097a40 0x7fd0c4132700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:34:01.762 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:01.752+0000 7fd0ca647700 1 --2- 172.21.15.88:0/3153929573 >> v2:172.21.15.107:3300/0 conn(0x7fd0c41332e0 0x7fd0c4133740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:34:01.763 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:01.752+0000 7fd0c9645700 10 monclient(hunting): handle_auth_done global_id 4554 payload 274 2023-09-30T01:34:01.763 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:01.752+0000 7fd0c9645700 1 -- 172.21.15.88:0/3153929573 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7fd0c4097a40 msgr2=0x7fd0c4132700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:34:01.763 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:01.752+0000 7fd0c9645700 1 --2- 172.21.15.88:0/3153929573 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7fd0c4097a40 0x7fd0c4132700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:34:01.763 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:01.752+0000 7fd0c9645700 1 -- 172.21.15.88:0/3153929573 >> v2:172.21.15.107:3300/0 conn(0x7fd0c41332e0 msgr2=0x7fd0c4133740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:34:01.763 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:01.752+0000 7fd0c9645700 1 --2- 172.21.15.88:0/3153929573 >> v2:172.21.15.107:3300/0 conn(0x7fd0c41332e0 0x7fd0c4133740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:34:01.763 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:01.752+0000 7fd0c9645700 10 monclient: _finish_hunting 0 2023-09-30T01:34:01.763 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:01.752+0000 7fd0c9645700 1 monclient: found mon.noname-a 2023-09-30T01:34:01.763 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:01.752+0000 7fd0c9645700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:34:01.763 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:01.752+0000 7fd0c9645700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.88:3300/0 2023-09-30T01:34:01.763 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:01.752+0000 7fd0c9645700 1 -- 172.21.15.88:0/3153929573 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0c404bb20 con 0x7fd0c41311c0 2023-09-30T01:34:01.763 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:01.752+0000 7fd0c9e46700 10 monclient: get_auth_request con 0x7fd0c4097a40 auth_method 0 2023-09-30T01:34:01.763 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:01.752+0000 7fd0c9e46700 1 --2- 172.21.15.88:0/3153929573 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7fd0c4097a40 0x7fd0c4132700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-09-30T01:34:01.764 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:01.752+0000 7fd0ca647700 10 monclient: get_auth_request con 0x7fd0c41332e0 auth_method 0 2023-09-30T01:34:01.764 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:01.752+0000 7fd0ca647700 1 --2- 172.21.15.88:0/3153929573 >> v2:172.21.15.107:3300/0 conn(0x7fd0c41332e0 0x7fd0c4133740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-09-30T01:34:01.764 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:01.752+0000 7fd0c9645700 1 --2- 172.21.15.88:0/3153929573 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd0c41311c0 0x7fd0c4132c40 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fd0c40a3a30 tx=0x7fd0c0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=456ec3b1c029b3a0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:34:01.764 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:01.752+0000 7fd0c8e44700 1 -- 172.21.15.88:0/3153929573 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7fd0c000bce0 con 0x7fd0c41311c0 2023-09-30T01:34:01.764 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:01.752+0000 7fd0c8e44700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:34:01.764 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:01.752+0000 7fd0c8e44700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T01:34:01.764 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:01.752+0000 7fd0c8e44700 10 monclient: dump: 2023-09-30T01:34:01.764 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:34:01.764 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:34:01.764 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:34:01.764 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:34:01.764 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:34:01.765 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:34:01.765 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:34:01.765 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:34:01.765 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:34:01.765 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:34:01.765 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:34:01.765 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:34:01.765 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:34:01.765 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:34:01.765 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:34:01.765 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:34:01.765 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:01.752+0000 7fd0c8e44700 10 monclient: _finish_auth 0 2023-09-30T01:34:01.766 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:01.752+0000 7fd0c8e44700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:34:01.766 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:01.752+0000 7fd0c8e44700 1 -- 172.21.15.88:0/3153929573 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd0c000be60 con 0x7fd0c41311c0 2023-09-30T01:34:01.766 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:01.752+0000 7fd0c8e44700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:34:01.766 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:01.752+0000 7fd0c8e44700 1 -- 172.21.15.88:0/3153929573 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7fd0c00118a0 con 0x7fd0c41311c0 2023-09-30T01:34:01.766 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:01.752+0000 7fd0cae48700 5 monclient: authenticate success, global_id 4554 2023-09-30T01:34:01.766 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:01.752+0000 7fd0cae48700 10 monclient: get_monmap_and_config success 2023-09-30T01:34:01.766 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:01.752+0000 7fd0c8e44700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:34:01.766 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:01.752+0000 7fd0cae48700 4 set_mon_vals no callback set 2023-09-30T01:34:01.766 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:01.752+0000 7fd0c8e44700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:34:01.766 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:01.752+0000 7fd0c8e44700 10 monclient: dump: 2023-09-30T01:34:01.766 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:34:01.766 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:34:01.767 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:34:01.767 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:34:01.767 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:34:01.767 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:34:01.767 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:34:01.768 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:34:01.768 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:34:01.768 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:34:01.768 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:34:01.768 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:34:01.768 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:34:01.768 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:34:01.768 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:34:01.768 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:34:01.768 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:01.752+0000 7fd0cae48700 10 monclient: shutdown 2023-09-30T01:34:01.768 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:01.752+0000 7fd0cae48700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:34:01.768 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:01.752+0000 7fd0cae48700 1 -- 172.21.15.88:0/3153929573 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd0c41311c0 msgr2=0x7fd0c4132c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:34:01.769 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:01.752+0000 7fd0cae48700 1 --2- 172.21.15.88:0/3153929573 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd0c41311c0 0x7fd0c4132c40 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fd0c40a3a30 tx=0x7fd0c0009250 comp rx=0 tx=0).stop 2023-09-30T01:34:01.769 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:01.752+0000 7fd0cae48700 1 -- 172.21.15.88:0/3153929573 shutdown_connections 2023-09-30T01:34:01.769 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:01.752+0000 7fd0cae48700 1 --2- 172.21.15.88:0/3153929573 >> v2:172.21.15.107:3300/0 conn(0x7fd0c41332e0 0x7fd0c4133740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:34:01.769 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:01.752+0000 7fd0cae48700 1 --2- 172.21.15.88:0/3153929573 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd0c41311c0 0x7fd0c4132c40 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:34:01.769 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:01.752+0000 7fd0cae48700 1 --2- 172.21.15.88:0/3153929573 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7fd0c4097a40 0x7fd0c4132700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:34:01.769 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:01.752+0000 7fd0cae48700 1 -- 172.21.15.88:0/3153929573 >> 172.21.15.88:0/3153929573 conn(0x7fd0c406e5e0 msgr2=0x7fd0c406e9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:34:01.769 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:01.752+0000 7fd0cae48700 1 -- 172.21.15.88:0/3153929573 shutdown_connections 2023-09-30T01:34:01.769 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:01.752+0000 7fd0cae48700 1 -- 172.21.15.88:0/3153929573 wait complete. 2023-09-30T01:34:01.769 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:01.752+0000 7fd0cae48700 10 monclient: build_initial_monmap 2023-09-30T01:34:01.769 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:01.752+0000 7fd0cae48700 10 monclient: monmap: 2023-09-30T01:34:01.769 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2023-09-30T01:34:01.769 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:34:01.770 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 0.000000 2023-09-30T01:34:01.770 INFO:teuthology.orchestra.run.smithi088.stderr:created 0.000000 2023-09-30T01:34:01.770 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2023-09-30T01:34:01.770 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 1 2023-09-30T01:34:01.770 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.noname-a 2023-09-30T01:34:01.770 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.noname-f 2023-09-30T01:34:01.770 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.noname-b 2023-09-30T01:34:01.770 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.noname-g 2023-09-30T01:34:01.770 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.noname-c 2023-09-30T01:34:01.770 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.noname-h 2023-09-30T01:34:01.770 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.noname-d 2023-09-30T01:34:01.770 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.noname-i 2023-09-30T01:34:01.770 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.noname-e 2023-09-30T01:34:01.771 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:34:01.771 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:01.756+0000 7fd0cae48700 1 Processor -- start 2023-09-30T01:34:01.771 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:01.756+0000 7fd0cae48700 1 -- start start 2023-09-30T01:34:01.771 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:01.756+0000 7fd0cae48700 10 monclient: init 2023-09-30T01:34:01.776 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:01.756+0000 7fd0cae48700 10 monclient: _reopen_session rank -1 2023-09-30T01:34:01.776 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:01.756+0000 7fd0cae48700 10 monclient: _add_conns ranks=[6,1,3,8,7,2,4,5,0] 2023-09-30T01:34:01.776 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:01.756+0000 7fd0cae48700 1 --2- >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7fd0c4097a40 0x7fd0c412fef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:34:01.776 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:01.756+0000 7fd0cae48700 10 monclient(hunting): picked mon.noname-d con 0x7fd0c4097a40 addr [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] 2023-09-30T01:34:01.776 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:01.756+0000 7fd0cae48700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd0c41311c0 0x7fd0c4130430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:34:01.776 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:01.756+0000 7fd0cae48700 10 monclient(hunting): picked mon.noname-f con 0x7fd0c41311c0 addr [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] 2023-09-30T01:34:01.776 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:01.756+0000 7fd0cae48700 1 --2- >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7fd0c41332e0 0x7fd0c411cd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:34:01.776 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:01.756+0000 7fd0cae48700 10 monclient(hunting): picked mon.noname-g con 0x7fd0c41332e0 addr [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] 2023-09-30T01:34:01.777 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:01.756+0000 7fd0cae48700 10 monclient(hunting): start opening mon connection 2023-09-30T01:34:01.777 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:01.756+0000 7fd0cae48700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0c40996b0 con 0x7fd0c41311c0 2023-09-30T01:34:01.777 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:01.756+0000 7fd0cae48700 10 monclient(hunting): start opening mon connection 2023-09-30T01:34:01.777 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:01.756+0000 7fd0cae48700 1 -- --> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0c4120690 con 0x7fd0c41332e0 2023-09-30T01:34:01.777 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:01.756+0000 7fd0cae48700 10 monclient(hunting): start opening mon connection 2023-09-30T01:34:01.777 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:01.756+0000 7fd0cae48700 1 -- --> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] -- mon_getmap magic: 0 v1 -- 0x7fd0c4120040 con 0x7fd0c4097a40 2023-09-30T01:34:01.777 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:01.756+0000 7fd0cae48700 10 monclient(hunting): _renew_subs 2023-09-30T01:34:01.777 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:01.756+0000 7fd0cae48700 10 monclient(hunting): authenticate will time out at 913.797945s 2023-09-30T01:34:01.777 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:01.756+0000 7fd0c9e46700 1 --2- >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7fd0c4097a40 0x7fd0c412fef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:34:01.777 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:01.756+0000 7fd0c9e46700 1 --2- >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7fd0c4097a40 0x7fd0c412fef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3303/0 says I am v2:172.21.15.88:33212/0 (socket says 172.21.15.88:33212) 2023-09-30T01:34:01.777 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:01.756+0000 7fd0c9e46700 1 -- 172.21.15.88:0/2551574056 learned_addr learned my addr 172.21.15.88:0/2551574056 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-09-30T01:34:01.778 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:01.756+0000 7fd0c9e46700 10 monclient(hunting): get_auth_request con 0x7fd0c4097a40 auth_method 0 2023-09-30T01:34:01.778 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:01.756+0000 7fd0c9e46700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:34:01.778 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:01.756+0000 7fd0c9e46700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:34:01.778 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:01.756+0000 7fd0c9e46700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:34:01.778 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:01.756+0000 7fd0c9645700 1 --2- 172.21.15.88:0/2551574056 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd0c41311c0 0x7fd0c4130430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:34:01.778 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:01.756+0000 7fd0c9645700 10 monclient(hunting): get_auth_request con 0x7fd0c41311c0 auth_method 0 2023-09-30T01:34:01.778 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:01.756+0000 7fd0c9645700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:34:01.778 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:01.756+0000 7fd0c9645700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:34:01.778 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:01.756+0000 7fd0c9645700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:34:01.778 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:01.756+0000 7fd0c9e46700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:34:01.778 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:01.756+0000 7fd0c9e46700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:34:01.778 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:01.756+0000 7fd0ca647700 1 --2- 172.21.15.88:0/2551574056 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7fd0c41332e0 0x7fd0c411cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:34:01.779 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:01.756+0000 7fd0c9e46700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:34:01.779 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:01.756+0000 7fd0ca647700 10 monclient(hunting): get_auth_request con 0x7fd0c41332e0 auth_method 0 2023-09-30T01:34:01.779 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:01.756+0000 7fd0ca647700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:34:01.779 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:01.756+0000 7fd0ca647700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:34:01.779 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:01.756+0000 7fd0ca647700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:34:01.779 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:01.756+0000 7fd0c9645700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:34:01.779 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:01.756+0000 7fd0c9645700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:34:01.779 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:01.756+0000 7fd0c9645700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:34:01.779 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:01.756+0000 7fd0ca647700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:34:01.779 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:01.756+0000 7fd0ca647700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:34:01.779 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:01.756+0000 7fd0ca647700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:34:01.779 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:01.756+0000 7fd0c9e46700 10 monclient(hunting): handle_auth_done global_id 4380 payload 824 2023-09-30T01:34:01.780 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:01.756+0000 7fd0c9e46700 1 -- 172.21.15.88:0/2551574056 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7fd0c41332e0 msgr2=0x7fd0c411cd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:34:01.780 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:01.756+0000 7fd0c9e46700 1 --2- 172.21.15.88:0/2551574056 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7fd0c41332e0 0x7fd0c411cd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:34:01.780 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:01.756+0000 7fd0c9e46700 1 -- 172.21.15.88:0/2551574056 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd0c41311c0 msgr2=0x7fd0c4130430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:34:01.780 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:01.756+0000 7fd0c9e46700 1 --2- 172.21.15.88:0/2551574056 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd0c41311c0 0x7fd0c4130430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:34:01.780 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:01.756+0000 7fd0c9e46700 10 monclient: _finish_hunting 0 2023-09-30T01:34:01.780 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:01.756+0000 7fd0c9e46700 1 monclient: found mon.noname-d 2023-09-30T01:34:01.780 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:01.756+0000 7fd0c9e46700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:34:01.780 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:01.756+0000 7fd0c9e46700 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.88:3303/0 2023-09-30T01:34:01.780 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:01.756+0000 7fd0c9e46700 1 -- 172.21.15.88:0/2551574056 --> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0c00094f0 con 0x7fd0c4097a40 2023-09-30T01:34:01.780 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:01.756+0000 7fd0c9e46700 1 --2- 172.21.15.88:0/2551574056 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7fd0c4097a40 0x7fd0c412fef0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fd0b400c880 tx=0x7fd0b4008040 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:34:01.780 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:01.756+0000 7fd0b9ffb700 1 -- 172.21.15.88:0/2551574056 <== mon.6 v2:172.21.15.88:3303/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7fd0b4011a80 con 0x7fd0c4097a40 2023-09-30T01:34:01.780 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:01.756+0000 7fd0b9ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:34:01.781 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:01.756+0000 7fd0b9ffb700 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2023-09-30T01:34:01.781 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:01.756+0000 7fd0b9ffb700 10 monclient: dump: 2023-09-30T01:34:01.781 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:34:01.781 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:34:01.781 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:34:01.781 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:34:01.781 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:34:01.781 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:34:01.781 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:34:01.781 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:34:01.781 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:34:01.781 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:34:01.782 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:34:01.782 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:34:01.782 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:34:01.782 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:34:01.782 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:34:01.782 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:34:01.782 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:01.756+0000 7fd0b9ffb700 10 monclient: _finish_auth 0 2023-09-30T01:34:01.782 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:01.756+0000 7fd0b9ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:34:01.782 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:01.756+0000 7fd0b9ffb700 1 -- 172.21.15.88:0/2551574056 <== mon.6 v2:172.21.15.88:3303/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd0b4011be0 con 0x7fd0c4097a40 2023-09-30T01:34:01.782 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:01.756+0000 7fd0b9ffb700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:34:01.782 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:01.756+0000 7fd0b9ffb700 1 -- 172.21.15.88:0/2551574056 <== mon.6 v2:172.21.15.88:3303/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7fd0b4011d50 con 0x7fd0c4097a40 2023-09-30T01:34:01.782 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:01.756+0000 7fd0cae48700 5 monclient: authenticate success, global_id 4380 2023-09-30T01:34:01.783 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:01.756+0000 7fd0cae48700 10 monclient: _renew_subs 2023-09-30T01:34:01.783 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:01.756+0000 7fd0bbfff700 4 set_mon_vals no callback set 2023-09-30T01:34:01.783 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:01.756+0000 7fd0cae48700 10 monclient: _send_mon_message to mon.d at v2:172.21.15.88:3303/0 2023-09-30T01:34:01.783 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:01.756+0000 7fd0cae48700 1 -- 172.21.15.88:0/2551574056 --> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd0c4120320 con 0x7fd0c4097a40 2023-09-30T01:34:01.783 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:01.756+0000 7fd0b9ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:34:01.783 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:01.756+0000 7fd0b9ffb700 10 monclient: got monmap 1 from mon.d (according to old e1) 2023-09-30T01:34:01.783 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:01.756+0000 7fd0b9ffb700 10 monclient: dump: 2023-09-30T01:34:01.783 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:34:01.783 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:34:01.783 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:34:01.783 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:34:01.783 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:34:01.784 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:34:01.784 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:34:01.784 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:34:01.784 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:34:01.784 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:34:01.784 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:34:01.784 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:34:01.784 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:34:01.784 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:34:01.784 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:34:01.784 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:34:01.784 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:01.756+0000 7fd0cae48700 10 monclient: _renew_subs 2023-09-30T01:34:01.784 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:01.756+0000 7fd0cae48700 10 monclient: _send_mon_message to mon.d at v2:172.21.15.88:3303/0 2023-09-30T01:34:01.785 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:01.756+0000 7fd0cae48700 1 -- 172.21.15.88:0/2551574056 --> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd0c411fce0 con 0x7fd0c4097a40 2023-09-30T01:34:01.785 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:01.760+0000 7fd0cae48700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:34:01.785 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:01.760+0000 7fd0cae48700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:34:01.785 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:01.760+0000 7fd0cae48700 10 monclient: _send_mon_message to mon.d at v2:172.21.15.88:3303/0 2023-09-30T01:34:01.785 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:01.760+0000 7fd0cae48700 1 -- 172.21.15.88:0/2551574056 --> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd0c4071960 con 0x7fd0c4097a40 2023-09-30T01:34:01.785 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:01.760+0000 7fd0b9ffb700 1 -- 172.21.15.88:0/2551574056 <== mon.6 v2:172.21.15.88:3303/0 4 ==== mgrmap(e 5) v1 ==== 40144+0+0 (secure 0 0 0) 0x7fd0b4011600 con 0x7fd0c4097a40 2023-09-30T01:34:01.785 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:01.760+0000 7fd0b9ffb700 1 --2- 172.21.15.88:0/2551574056 >> v2:172.21.15.107:6812/29926 conn(0x7fd0a8033360 0x7fd0a8035810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:34:01.785 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:01.760+0000 7fd0b9ffb700 1 -- 172.21.15.88:0/2551574056 <== mon.6 v2:172.21.15.88:3303/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 3502+0+0 (secure 0 0 0) 0x7fd0b40431e0 con 0x7fd0c4097a40 2023-09-30T01:34:01.785 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:01.760+0000 7fd0c9645700 1 --2- 172.21.15.88:0/2551574056 >> v2:172.21.15.107:6812/29926 conn(0x7fd0a8033360 0x7fd0a8035810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:34:01.785 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:01.760+0000 7fd0c9645700 10 monclient: get_auth_request con 0x7fd0a8033360 auth_method 0 2023-09-30T01:34:01.785 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:01.760+0000 7fd0c9645700 1 --2- 172.21.15.88:0/2551574056 >> v2:172.21.15.107:6812/29926 conn(0x7fd0a8033360 0x7fd0a8035810 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fd0c000bfd0 tx=0x7fd0c0017000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:34:01.785 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:01.768+0000 7fd0b9ffb700 1 -- 172.21.15.88:0/2551574056 <== mon.6 v2:172.21.15.88:3303/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fd0b400ed60 con 0x7fd0c4097a40 2023-09-30T01:34:01.786 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:01.768+0000 7fd0b9ffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:34:01.786 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:01.768+0000 7fd0b9ffb700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:34:01.965 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:01.960+0000 7fd0cae48700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T01:34:01.965 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:01.960+0000 7fd0cae48700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:34:01.965 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:01.960+0000 7fd0cae48700 10 monclient: _send_mon_message to mon.d at v2:172.21.15.88:3303/0 2023-09-30T01:34:01.965 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:01.960+0000 7fd0cae48700 1 -- 172.21.15.88:0/2551574056 --> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fd0c411f100 con 0x7fd0c4097a40 2023-09-30T01:34:01.966 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:01.960+0000 7fd0b9ffb700 1 -- 172.21.15.88:0/2551574056 <== mon.6 v2:172.21.15.88:3303/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7fd0b400e300 con 0x7fd0c4097a40 2023-09-30T01:34:01.966 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:01.960+0000 7fd0b9ffb700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:34:01.966 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:01.960+0000 7fd0b9ffb700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:34:01.966 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-09-30T01:34:01.966 INFO:teuthology.orchestra.run.smithi088.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:34:01.968 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:01.964+0000 7fd0cae48700 1 -- 172.21.15.88:0/2551574056 >> v2:172.21.15.107:6812/29926 conn(0x7fd0a8033360 msgr2=0x7fd0a8035810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:34:01.968 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:01.964+0000 7fd0cae48700 1 --2- 172.21.15.88:0/2551574056 >> v2:172.21.15.107:6812/29926 conn(0x7fd0a8033360 0x7fd0a8035810 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fd0c000bfd0 tx=0x7fd0c0017000 comp rx=0 tx=0).stop 2023-09-30T01:34:01.968 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:01.964+0000 7fd0cae48700 10 monclient: shutdown 2023-09-30T01:34:01.968 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:01.964+0000 7fd0cae48700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:34:01.969 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:01.964+0000 7fd0cae48700 1 -- 172.21.15.88:0/2551574056 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7fd0c4097a40 msgr2=0x7fd0c412fef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:34:01.969 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:01.964+0000 7fd0cae48700 1 --2- 172.21.15.88:0/2551574056 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7fd0c4097a40 0x7fd0c412fef0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fd0b400c880 tx=0x7fd0b4008040 comp rx=0 tx=0).stop 2023-09-30T01:34:01.969 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:01.964+0000 7fd0cae48700 1 -- 172.21.15.88:0/2551574056 shutdown_connections 2023-09-30T01:34:01.969 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:01.964+0000 7fd0cae48700 1 --2- 172.21.15.88:0/2551574056 >> v2:172.21.15.107:6812/29926 conn(0x7fd0a8033360 0x7fd0a8035810 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:34:01.969 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:01.964+0000 7fd0cae48700 1 --2- 172.21.15.88:0/2551574056 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7fd0c41332e0 0x7fd0c411cd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:34:01.969 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:01.964+0000 7fd0cae48700 1 --2- 172.21.15.88:0/2551574056 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd0c41311c0 0x7fd0c4130430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:34:01.969 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:01.964+0000 7fd0cae48700 1 --2- 172.21.15.88:0/2551574056 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7fd0c4097a40 0x7fd0c412fef0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:34:01.969 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:01.964+0000 7fd0cae48700 1 -- 172.21.15.88:0/2551574056 >> 172.21.15.88:0/2551574056 conn(0x7fd0c406e5e0 msgr2=0x7fd0c4131f60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:34:01.969 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:01.964+0000 7fd0cae48700 1 -- 172.21.15.88:0/2551574056 shutdown_connections 2023-09-30T01:34:01.969 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:01.964+0000 7fd0cae48700 1 -- 172.21.15.88:0/2551574056 wait complete. 2023-09-30T01:34:04.980 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:34:05.092 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:05.088+0000 7f0fee8fe700 10 monclient: get_monmap_and_config 2023-09-30T01:34:05.092 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:05.088+0000 7f0fee8fe700 10 monclient: build_initial_monmap 2023-09-30T01:34:05.092 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:05.088+0000 7f0fee8fe700 10 monclient: monmap: 2023-09-30T01:34:05.092 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2023-09-30T01:34:05.092 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:34:05.092 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:34:05.092604+0000 2023-09-30T01:34:05.093 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:34:05.092604+0000 2023-09-30T01:34:05.093 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2023-09-30T01:34:05.093 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:34:05.093 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a-legacy 2023-09-30T01:34:05.093 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.107:6789/0 mon.noname-f-legacy 2023-09-30T01:34:05.093 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.noname-b 2023-09-30T01:34:05.093 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.noname-g 2023-09-30T01:34:05.093 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.noname-c 2023-09-30T01:34:05.093 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.noname-h 2023-09-30T01:34:05.093 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.noname-d 2023-09-30T01:34:05.093 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.noname-i 2023-09-30T01:34:05.093 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.noname-e 2023-09-30T01:34:05.094 INFO:teuthology.orchestra.run.smithi088.stderr:9: v2:172.21.15.88:3300/0 mon.noname-a 2023-09-30T01:34:05.094 INFO:teuthology.orchestra.run.smithi088.stderr:10: v2:172.21.15.107:3300/0 mon.noname-f 2023-09-30T01:34:05.094 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:34:05.097 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:05.088+0000 7f0fee8fe700 1 Processor -- start 2023-09-30T01:34:05.098 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:05.088+0000 7f0fee8fe700 1 -- start start 2023-09-30T01:34:05.098 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:05.088+0000 7f0fee8fe700 10 monclient: init 2023-09-30T01:34:05.098 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:05.088+0000 7f0fee8fe700 10 monclient: _reopen_session rank -1 2023-09-30T01:34:05.098 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:05.088+0000 7f0fee8fe700 10 monclient: _add_conns ranks=[0,8,3,10,9,4,6,5,7,2,1] 2023-09-30T01:34:05.098 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:05.088+0000 7f0fee8fe700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f0fe8097a40 addr v1:172.21.15.88:6789/0 2023-09-30T01:34:05.098 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:05.088+0000 7f0fee8fe700 1 --2- >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f0fe8132e10 0x7f0fe8138650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:34:05.098 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:05.088+0000 7f0fee8fe700 10 monclient(hunting): picked mon.noname-e con 0x7f0fe8132e10 addr [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] 2023-09-30T01:34:05.098 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:05.088+0000 7f0fee8fe700 1 --2- >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7f0fe81311c0 0x7f0fe813aba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:34:05.098 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:05.088+0000 7f0fee8fe700 10 monclient(hunting): picked mon.noname-g con 0x7f0fe81311c0 addr [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] 2023-09-30T01:34:05.098 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:05.088+0000 7f0fee8fe700 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0fe804bb20 con 0x7f0fe8097a40 2023-09-30T01:34:05.098 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:05.088+0000 7f0fee8fe700 10 monclient(hunting): start opening mon connection 2023-09-30T01:34:05.099 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:05.088+0000 7f0fee8fe700 1 -- --> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0fe809a5d0 con 0x7f0fe81311c0 2023-09-30T01:34:05.099 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:05.088+0000 7f0fee8fe700 10 monclient(hunting): start opening mon connection 2023-09-30T01:34:05.099 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:05.088+0000 7f0fee8fe700 1 -- --> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] -- mon_getmap magic: 0 v1 -- 0x7f0fe8097e10 con 0x7f0fe8132e10 2023-09-30T01:34:05.099 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:05.088+0000 7f0fee8fe700 10 monclient(hunting): _renew_subs 2023-09-30T01:34:05.099 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:05.088+0000 7f0fee8fe700 10 monclient(hunting): authenticate will time out at 917.130854s 2023-09-30T01:34:05.099 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:05.088+0000 7f0fed0fb700 1 --2- >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f0fe8132e10 0x7f0fe8138650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:34:05.099 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:05.088+0000 7f0fed8fc700 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f0fe8097a40 0x7f0fe8132700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:35736/0 (socket says 172.21.15.88:35736) 2023-09-30T01:34:05.099 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:05.088+0000 7f0fed8fc700 1 -- 172.21.15.88:0/605795811 learned_addr learned my addr 172.21.15.88:0/605795811 (peer_addr_for_me v1:172.21.15.88:0/0) 2023-09-30T01:34:05.099 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:05.088+0000 7f0fed0fb700 10 monclient(hunting): get_auth_request con 0x7f0fe8132e10 auth_method 0 2023-09-30T01:34:05.099 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:05.088+0000 7f0fed0fb700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:34:05.099 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:05.088+0000 7f0fed0fb700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:34:05.099 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:05.088+0000 7f0fed0fb700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:34:05.100 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:05.088+0000 7f0fee0fd700 1 --2- 172.21.15.88:0/605795811 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7f0fe81311c0 0x7f0fe813aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:34:05.100 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:05.088+0000 7f0fed0fb700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:34:05.100 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:05.088+0000 7f0fed0fb700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:34:05.100 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:05.088+0000 7f0fed0fb700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:34:05.100 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:05.088+0000 7f0fee0fd700 10 monclient(hunting): get_auth_request con 0x7f0fe81311c0 auth_method 0 2023-09-30T01:34:05.100 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:05.088+0000 7f0fee0fd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:34:05.100 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:05.088+0000 7f0fee0fd700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:34:05.100 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:05.088+0000 7f0fee0fd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:34:05.100 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:05.088+0000 7f0fec8fa700 1 -- 172.21.15.88:0/605795811 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1911238603 0 0) 0x7f0fe804bb20 con 0x7f0fe8097a40 2023-09-30T01:34:05.100 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:05.088+0000 7f0fec8fa700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:34:05.100 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:05.088+0000 7f0fec8fa700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:34:05.100 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:05.088+0000 7f0fec8fa700 10 monclient(hunting): my global_id is 4563 2023-09-30T01:34:05.100 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:05.088+0000 7f0fec8fa700 1 -- 172.21.15.88:0/605795811 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0fd4003570 con 0x7f0fe8097a40 2023-09-30T01:34:05.101 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:05.088+0000 7f0fed0fb700 10 monclient(hunting): handle_auth_done global_id 4508 payload 274 2023-09-30T01:34:05.101 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:05.088+0000 7f0fed0fb700 1 -- 172.21.15.88:0/605795811 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7f0fe81311c0 msgr2=0x7f0fe813aba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:34:05.101 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:05.088+0000 7f0fed0fb700 1 --2- 172.21.15.88:0/605795811 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7f0fe81311c0 0x7f0fe813aba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:34:05.101 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:05.088+0000 7f0fed0fb700 1 -- 172.21.15.88:0/605795811 >> v1:172.21.15.88:6789/0 conn(0x7f0fe8097a40 legacy=0x7f0fe8132700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:34:05.101 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:05.088+0000 7f0fed0fb700 10 monclient: _finish_hunting 0 2023-09-30T01:34:05.101 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:05.088+0000 7f0fed0fb700 1 monclient: found mon.noname-e 2023-09-30T01:34:05.101 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:05.088+0000 7f0fed0fb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:34:05.101 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:05.088+0000 7f0fed0fb700 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.88:3304/0 2023-09-30T01:34:05.101 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:05.088+0000 7f0fed0fb700 1 -- 172.21.15.88:0/605795811 --> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0fe8098d40 con 0x7f0fe8132e10 2023-09-30T01:34:05.101 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:05.088+0000 7f0fee0fd700 1 --2- 172.21.15.88:0/605795811 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7f0fe81311c0 0x7f0fe813aba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-30T01:34:05.101 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:05.092+0000 7f0fed0fb700 1 --2- 172.21.15.88:0/605795811 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f0fe8132e10 0x7f0fe8138650 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f0fe4009910 tx=0x7f0fe40091a0 comp rx=0 tx=0).ready entity=mon.8 client_cookie=d6eba632362d4f0b server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:34:05.102 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:05.092+0000 7f0fec8fa700 1 -- 172.21.15.88:0/605795811 <== mon.8 v2:172.21.15.88:3304/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f0fe400bb00 con 0x7f0fe8132e10 2023-09-30T01:34:05.102 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:05.092+0000 7f0fec8fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:34:05.102 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:05.092+0000 7f0fec8fa700 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2023-09-30T01:34:05.102 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:05.092+0000 7f0fec8fa700 10 monclient: dump: 2023-09-30T01:34:05.102 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:34:05.102 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:34:05.102 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:34:05.102 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:34:05.102 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:34:05.102 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:34:05.102 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:34:05.102 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:34:05.103 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:34:05.103 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:34:05.103 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:34:05.103 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:34:05.103 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:34:05.103 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:34:05.103 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:34:05.103 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:34:05.103 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:05.092+0000 7f0fec8fa700 10 monclient: _finish_auth 0 2023-09-30T01:34:05.103 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:05.092+0000 7f0fec8fa700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:34:05.103 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:05.092+0000 7f0fee8fe700 5 monclient: authenticate success, global_id 4508 2023-09-30T01:34:05.103 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:05.092+0000 7f0fee8fe700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:34:05.104 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:05.092+0000 7f0fec8fa700 1 -- 172.21.15.88:0/605795811 <== mon.8 v2:172.21.15.88:3304/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0fe400bc60 con 0x7f0fe8132e10 2023-09-30T01:34:05.104 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:05.092+0000 7f0fec8fa700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:34:05.104 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:05.092+0000 7f0fee8fe700 10 monclient: get_monmap_and_config success 2023-09-30T01:34:05.104 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:05.092+0000 7f0fee8fe700 4 set_mon_vals no callback set 2023-09-30T01:34:05.104 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:05.092+0000 7f0fec8fa700 1 -- 172.21.15.88:0/605795811 <== mon.8 v2:172.21.15.88:3304/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f0fe400bdd0 con 0x7f0fe8132e10 2023-09-30T01:34:05.104 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:05.092+0000 7f0fec8fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:34:05.104 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:05.092+0000 7f0fec8fa700 10 monclient: got monmap 1 from mon.e (according to old e1) 2023-09-30T01:34:05.104 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:05.092+0000 7f0fec8fa700 10 monclient: dump: 2023-09-30T01:34:05.104 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:34:05.104 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:34:05.105 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:34:05.105 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:34:05.105 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:34:05.105 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:34:05.105 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:34:05.105 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:34:05.105 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:34:05.105 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:34:05.105 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:34:05.105 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:34:05.105 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:34:05.105 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:34:05.106 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:34:05.106 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:34:05.106 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:05.092+0000 7f0fee8fe700 10 monclient: shutdown 2023-09-30T01:34:05.106 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:05.092+0000 7f0fee8fe700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:34:05.106 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:05.092+0000 7f0fee8fe700 1 -- 172.21.15.88:0/605795811 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f0fe8132e10 msgr2=0x7f0fe8138650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:34:05.106 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:05.092+0000 7f0fee8fe700 1 --2- 172.21.15.88:0/605795811 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f0fe8132e10 0x7f0fe8138650 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f0fe4009910 tx=0x7f0fe40091a0 comp rx=0 tx=0).stop 2023-09-30T01:34:05.106 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:05.092+0000 7f0fee8fe700 1 -- 172.21.15.88:0/605795811 shutdown_connections 2023-09-30T01:34:05.106 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:05.092+0000 7f0fee8fe700 1 --2- 172.21.15.88:0/605795811 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7f0fe81311c0 0x7f0fe813aba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:34:05.106 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:05.092+0000 7f0fee8fe700 1 --2- 172.21.15.88:0/605795811 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f0fe8132e10 0x7f0fe8138650 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:34:05.106 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:05.092+0000 7f0fee8fe700 1 -- 172.21.15.88:0/605795811 >> 172.21.15.88:0/605795811 conn(0x7f0fe806e5e0 msgr2=0x7f0fe806e9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:34:05.106 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:05.092+0000 7f0fee8fe700 1 -- 172.21.15.88:0/605795811 shutdown_connections 2023-09-30T01:34:05.106 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:05.092+0000 7f0fee8fe700 1 -- 172.21.15.88:0/605795811 wait complete. 2023-09-30T01:34:05.107 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:05.092+0000 7f0fee8fe700 10 monclient: build_initial_monmap 2023-09-30T01:34:05.107 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:05.092+0000 7f0fee8fe700 10 monclient: monmap: 2023-09-30T01:34:05.107 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2023-09-30T01:34:05.107 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:34:05.107 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 0.000000 2023-09-30T01:34:05.107 INFO:teuthology.orchestra.run.smithi088.stderr:created 0.000000 2023-09-30T01:34:05.107 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2023-09-30T01:34:05.107 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 1 2023-09-30T01:34:05.107 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.noname-a 2023-09-30T01:34:05.107 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.noname-f 2023-09-30T01:34:05.107 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.noname-b 2023-09-30T01:34:05.107 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.noname-g 2023-09-30T01:34:05.107 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.noname-c 2023-09-30T01:34:05.108 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.noname-h 2023-09-30T01:34:05.108 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.noname-d 2023-09-30T01:34:05.108 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.noname-i 2023-09-30T01:34:05.108 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.noname-e 2023-09-30T01:34:05.108 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:34:05.108 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:05.092+0000 7f0fee8fe700 1 Processor -- start 2023-09-30T01:34:05.108 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:05.092+0000 7f0fee8fe700 1 -- start start 2023-09-30T01:34:05.108 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:05.092+0000 7f0fee8fe700 10 monclient: init 2023-09-30T01:34:05.112 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:05.092+0000 7f0fee8fe700 10 monclient: _reopen_session rank -1 2023-09-30T01:34:05.112 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:05.092+0000 7f0fee8fe700 10 monclient: _add_conns ranks=[7,5,6,8,2,4,3,0,1] 2023-09-30T01:34:05.112 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:05.092+0000 7f0fee8fe700 1 --2- >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7f0fe8097a40 0x7f0fe8142630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:34:05.112 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:05.092+0000 7f0fee8fe700 10 monclient(hunting): picked mon.noname-i con 0x7f0fe8097a40 addr [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] 2023-09-30T01:34:05.112 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:05.092+0000 7f0fee8fe700 1 --2- >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7f0fe81311c0 0x7f0fe8142b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:34:05.112 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:05.092+0000 7f0fee8fe700 10 monclient(hunting): picked mon.noname-h con 0x7f0fe81311c0 addr [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] 2023-09-30T01:34:05.112 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:05.092+0000 7f0fee8fe700 1 --2- >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7f0fe8132e10 0x7f0fe811cd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:34:05.112 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:05.092+0000 7f0fee8fe700 10 monclient(hunting): picked mon.noname-d con 0x7f0fe8132e10 addr [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] 2023-09-30T01:34:05.112 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:05.092+0000 7f0fee8fe700 10 monclient(hunting): start opening mon connection 2023-09-30T01:34:05.112 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:05.092+0000 7f0fee8fe700 1 -- --> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] -- mon_getmap magic: 0 v1 -- 0x7f0fe8099c30 con 0x7f0fe81311c0 2023-09-30T01:34:05.113 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:05.092+0000 7f0fee8fe700 10 monclient(hunting): start opening mon connection 2023-09-30T01:34:05.113 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:05.092+0000 7f0fee8fe700 1 -- --> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] -- mon_getmap magic: 0 v1 -- 0x7f0fe80996b0 con 0x7f0fe8132e10 2023-09-30T01:34:05.113 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:05.092+0000 7f0fee8fe700 10 monclient(hunting): start opening mon connection 2023-09-30T01:34:05.113 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:05.092+0000 7f0fee8fe700 1 -- --> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] -- mon_getmap magic: 0 v1 -- 0x7f0fe8120280 con 0x7f0fe8097a40 2023-09-30T01:34:05.113 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:05.092+0000 7f0fee8fe700 10 monclient(hunting): _renew_subs 2023-09-30T01:34:05.113 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:05.092+0000 7f0fee8fe700 10 monclient(hunting): authenticate will time out at 917.135278s 2023-09-30T01:34:05.113 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:05.092+0000 7f0fee0fd700 1 --2- >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7f0fe8132e10 0x7f0fe811cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:34:05.113 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:05.092+0000 7f0fee0fd700 1 --2- >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7f0fe8132e10 0x7f0fe811cd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3303/0 says I am v2:172.21.15.88:33222/0 (socket says 172.21.15.88:33222) 2023-09-30T01:34:05.113 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:05.092+0000 7f0fee0fd700 1 -- 172.21.15.88:0/2956187227 learned_addr learned my addr 172.21.15.88:0/2956187227 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-09-30T01:34:05.113 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:05.092+0000 7f0fee0fd700 10 monclient(hunting): get_auth_request con 0x7f0fe8132e10 auth_method 0 2023-09-30T01:34:05.113 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:05.092+0000 7f0fee0fd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:34:05.113 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:05.092+0000 7f0fee0fd700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:34:05.114 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:05.092+0000 7f0fee0fd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:34:05.114 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:05.092+0000 7f0fee0fd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:34:05.114 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:05.092+0000 7f0fee0fd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:34:05.114 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:05.092+0000 7f0fed8fc700 1 --2- 172.21.15.88:0/2956187227 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7f0fe8097a40 0x7f0fe8142630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:34:05.114 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:05.092+0000 7f0fee0fd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:34:05.114 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:05.092+0000 7f0fed0fb700 1 --2- 172.21.15.88:0/2956187227 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7f0fe81311c0 0x7f0fe8142b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:34:05.114 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:05.092+0000 7f0fed8fc700 10 monclient(hunting): get_auth_request con 0x7f0fe8097a40 auth_method 0 2023-09-30T01:34:05.114 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:05.092+0000 7f0fed8fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:34:05.114 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:05.092+0000 7f0fed8fc700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:34:05.114 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:05.092+0000 7f0fed8fc700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:34:05.114 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:05.092+0000 7f0fed0fb700 10 monclient(hunting): get_auth_request con 0x7f0fe81311c0 auth_method 0 2023-09-30T01:34:05.114 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:05.092+0000 7f0fed0fb700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:34:05.115 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:05.092+0000 7f0fed0fb700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:34:05.115 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:05.092+0000 7f0fed0fb700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:34:05.115 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:05.092+0000 7f0fee0fd700 10 monclient(hunting): handle_auth_done global_id 4389 payload 824 2023-09-30T01:34:05.115 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:05.092+0000 7f0fee0fd700 1 -- 172.21.15.88:0/2956187227 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7f0fe8097a40 msgr2=0x7f0fe8142630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:34:05.115 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:05.092+0000 7f0fee0fd700 1 --2- 172.21.15.88:0/2956187227 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7f0fe8097a40 0x7f0fe8142630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:34:05.115 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:05.092+0000 7f0fee0fd700 1 -- 172.21.15.88:0/2956187227 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7f0fe81311c0 msgr2=0x7f0fe8142b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:34:05.115 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:05.092+0000 7f0fee0fd700 1 --2- 172.21.15.88:0/2956187227 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7f0fe81311c0 0x7f0fe8142b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:34:05.115 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:05.092+0000 7f0fee0fd700 10 monclient: _finish_hunting 0 2023-09-30T01:34:05.115 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:05.092+0000 7f0fee0fd700 1 monclient: found mon.noname-d 2023-09-30T01:34:05.115 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:05.092+0000 7f0fee0fd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:34:05.115 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:05.092+0000 7f0fee0fd700 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.88:3303/0 2023-09-30T01:34:05.115 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:05.092+0000 7f0fee0fd700 1 -- 172.21.15.88:0/2956187227 --> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0fe4009420 con 0x7f0fe8132e10 2023-09-30T01:34:05.116 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:05.092+0000 7f0fed8fc700 1 --2- 172.21.15.88:0/2956187227 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7f0fe8097a40 0x7f0fe8142630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-30T01:34:05.116 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:05.092+0000 7f0fed0fb700 1 --2- 172.21.15.88:0/2956187227 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7f0fe81311c0 0x7f0fe8142b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-30T01:34:05.116 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:05.096+0000 7f0fee0fd700 1 --2- 172.21.15.88:0/2956187227 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7f0fe8132e10 0x7f0fe811cd50 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f0fe000d1b0 tx=0x7f0fe0009a50 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:34:05.116 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:05.096+0000 7f0fddffb700 1 -- 172.21.15.88:0/2956187227 <== mon.6 v2:172.21.15.88:3303/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f0fe000de50 con 0x7f0fe8132e10 2023-09-30T01:34:05.116 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:05.096+0000 7f0fddffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:34:05.116 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:05.096+0000 7f0fddffb700 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2023-09-30T01:34:05.116 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:05.096+0000 7f0fddffb700 10 monclient: dump: 2023-09-30T01:34:05.116 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:34:05.116 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:34:05.116 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:34:05.116 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:34:05.116 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:34:05.117 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:34:05.117 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:34:05.117 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:34:05.117 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:34:05.117 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:34:05.117 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:34:05.117 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:34:05.117 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:34:05.117 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:34:05.117 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:34:05.117 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:34:05.117 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:05.096+0000 7f0fddffb700 10 monclient: _finish_auth 0 2023-09-30T01:34:05.117 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:05.096+0000 7f0fddffb700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:34:05.118 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:05.096+0000 7f0fddffb700 1 -- 172.21.15.88:0/2956187227 <== mon.6 v2:172.21.15.88:3303/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0fe000f040 con 0x7f0fe8132e10 2023-09-30T01:34:05.118 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:05.096+0000 7f0fee8fe700 5 monclient: authenticate success, global_id 4389 2023-09-30T01:34:05.118 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:05.096+0000 7f0fee8fe700 10 monclient: _renew_subs 2023-09-30T01:34:05.118 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:05.096+0000 7f0fee8fe700 10 monclient: _send_mon_message to mon.d at v2:172.21.15.88:3303/0 2023-09-30T01:34:05.118 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:05.096+0000 7f0fee8fe700 1 -- 172.21.15.88:0/2956187227 --> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0fe811f0b0 con 0x7f0fe8132e10 2023-09-30T01:34:05.118 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:05.096+0000 7f0fddffb700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:34:05.118 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:05.096+0000 7f0fddffb700 1 -- 172.21.15.88:0/2956187227 <== mon.6 v2:172.21.15.88:3303/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f0fe0014ca0 con 0x7f0fe8132e10 2023-09-30T01:34:05.118 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:05.096+0000 7f0fddffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:34:05.118 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:05.096+0000 7f0fddffb700 10 monclient: got monmap 1 from mon.d (according to old e1) 2023-09-30T01:34:05.118 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:05.096+0000 7f0fddffb700 10 monclient: dump: 2023-09-30T01:34:05.118 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:34:05.119 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:34:05.119 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:34:05.119 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:34:05.119 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:34:05.119 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:34:05.119 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:34:05.119 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:34:05.119 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:34:05.119 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:34:05.119 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:34:05.119 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:34:05.119 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:34:05.119 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:34:05.120 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:34:05.120 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:34:05.120 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:05.096+0000 7f0fee8fe700 10 monclient: _renew_subs 2023-09-30T01:34:05.120 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:05.096+0000 7f0fdffff700 4 set_mon_vals no callback set 2023-09-30T01:34:05.120 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:05.096+0000 7f0fee8fe700 10 monclient: _send_mon_message to mon.d at v2:172.21.15.88:3303/0 2023-09-30T01:34:05.120 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:05.096+0000 7f0fee8fe700 1 -- 172.21.15.88:0/2956187227 --> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0fe811e470 con 0x7f0fe8132e10 2023-09-30T01:34:05.120 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:05.096+0000 7f0fddffb700 1 -- 172.21.15.88:0/2956187227 <== mon.6 v2:172.21.15.88:3303/0 4 ==== mgrmap(e 5) v1 ==== 40144+0+0 (secure 0 0 0) 0x7f0fe00143a0 con 0x7f0fe8132e10 2023-09-30T01:34:05.120 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:05.096+0000 7f0fddffb700 1 --2- 172.21.15.88:0/2956187227 >> v2:172.21.15.107:6812/29926 conn(0x7f0fcc033360 0x7f0fcc035810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:34:05.120 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:05.096+0000 7f0fddffb700 1 -- 172.21.15.88:0/2956187227 <== mon.6 v2:172.21.15.88:3303/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 3502+0+0 (secure 0 0 0) 0x7f0fe00435f0 con 0x7f0fe8132e10 2023-09-30T01:34:05.120 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:05.096+0000 7f0fee8fe700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:34:05.120 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:05.096+0000 7f0fee8fe700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:34:05.120 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:05.096+0000 7f0fee8fe700 10 monclient: _send_mon_message to mon.d at v2:172.21.15.88:3303/0 2023-09-30T01:34:05.121 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:05.096+0000 7f0fee8fe700 1 -- 172.21.15.88:0/2956187227 --> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0fe81203f0 con 0x7f0fe8132e10 2023-09-30T01:34:05.121 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:05.096+0000 7f0fed8fc700 1 --2- 172.21.15.88:0/2956187227 >> v2:172.21.15.107:6812/29926 conn(0x7f0fcc033360 0x7f0fcc035810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:34:05.121 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:05.096+0000 7f0fed8fc700 10 monclient: get_auth_request con 0x7f0fcc033360 auth_method 0 2023-09-30T01:34:05.121 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:05.096+0000 7f0fed8fc700 1 --2- 172.21.15.88:0/2956187227 >> v2:172.21.15.107:6812/29926 conn(0x7f0fcc033360 0x7f0fcc035810 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f0fd80058d0 tx=0x7f0fd8009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:34:05.121 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:05.104+0000 7f0fddffb700 1 -- 172.21.15.88:0/2956187227 <== mon.6 v2:172.21.15.88:3303/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f0fe001a070 con 0x7f0fe8132e10 2023-09-30T01:34:05.121 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:05.104+0000 7f0fddffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:34:05.121 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:05.104+0000 7f0fddffb700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:34:05.303 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:05.300+0000 7f0fee8fe700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T01:34:05.303 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:05.300+0000 7f0fee8fe700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:34:05.304 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:05.300+0000 7f0fee8fe700 10 monclient: _send_mon_message to mon.d at v2:172.21.15.88:3303/0 2023-09-30T01:34:05.304 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:05.300+0000 7f0fee8fe700 1 -- 172.21.15.88:0/2956187227 --> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f0fe8071960 con 0x7f0fe8132e10 2023-09-30T01:34:05.304 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:05.300+0000 7f0fddffb700 1 -- 172.21.15.88:0/2956187227 <== mon.6 v2:172.21.15.88:3303/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f0fe00118a0 con 0x7f0fe8132e10 2023-09-30T01:34:05.304 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:05.300+0000 7f0fddffb700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:34:05.304 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:05.300+0000 7f0fddffb700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:34:05.305 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-09-30T01:34:05.305 INFO:teuthology.orchestra.run.smithi088.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:34:05.307 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:05.300+0000 7f0fee8fe700 1 -- 172.21.15.88:0/2956187227 >> v2:172.21.15.107:6812/29926 conn(0x7f0fcc033360 msgr2=0x7f0fcc035810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:34:05.307 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:05.300+0000 7f0fee8fe700 1 --2- 172.21.15.88:0/2956187227 >> v2:172.21.15.107:6812/29926 conn(0x7f0fcc033360 0x7f0fcc035810 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f0fd80058d0 tx=0x7f0fd8009040 comp rx=0 tx=0).stop 2023-09-30T01:34:05.307 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:05.300+0000 7f0fee8fe700 10 monclient: shutdown 2023-09-30T01:34:05.307 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:05.300+0000 7f0fee8fe700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:34:05.307 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:05.300+0000 7f0fee8fe700 1 -- 172.21.15.88:0/2956187227 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7f0fe8132e10 msgr2=0x7f0fe811cd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:34:05.307 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:05.300+0000 7f0fee8fe700 1 --2- 172.21.15.88:0/2956187227 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7f0fe8132e10 0x7f0fe811cd50 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f0fe000d1b0 tx=0x7f0fe0009a50 comp rx=0 tx=0).stop 2023-09-30T01:34:05.308 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:05.300+0000 7f0fee8fe700 1 -- 172.21.15.88:0/2956187227 shutdown_connections 2023-09-30T01:34:05.308 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:05.300+0000 7f0fee8fe700 1 --2- 172.21.15.88:0/2956187227 >> v2:172.21.15.107:6812/29926 conn(0x7f0fcc033360 0x7f0fcc035810 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:34:05.308 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:05.300+0000 7f0fee8fe700 1 --2- 172.21.15.88:0/2956187227 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7f0fe8132e10 0x7f0fe811cd50 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:34:05.308 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:05.300+0000 7f0fee8fe700 1 --2- 172.21.15.88:0/2956187227 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7f0fe81311c0 0x7f0fe8142b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:34:05.308 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:05.300+0000 7f0fee8fe700 1 --2- 172.21.15.88:0/2956187227 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7f0fe8097a40 0x7f0fe8142630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:34:05.308 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:05.300+0000 7f0fee8fe700 1 -- 172.21.15.88:0/2956187227 >> 172.21.15.88:0/2956187227 conn(0x7f0fe806e5e0 msgr2=0x7f0fe8131f10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:34:05.309 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:05.300+0000 7f0fee8fe700 1 -- 172.21.15.88:0/2956187227 shutdown_connections 2023-09-30T01:34:05.309 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:05.300+0000 7f0fee8fe700 1 -- 172.21.15.88:0/2956187227 wait complete. 2023-09-30T01:34:08.319 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:34:08.431 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:08.424+0000 7fce1b59e700 10 monclient: get_monmap_and_config 2023-09-30T01:34:08.431 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:08.424+0000 7fce1b59e700 10 monclient: build_initial_monmap 2023-09-30T01:34:08.431 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:08.424+0000 7fce1b59e700 10 monclient: monmap: 2023-09-30T01:34:08.431 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2023-09-30T01:34:08.431 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:34:08.431 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:34:08.431607+0000 2023-09-30T01:34:08.431 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:34:08.431607+0000 2023-09-30T01:34:08.431 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2023-09-30T01:34:08.432 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:34:08.432 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a-legacy 2023-09-30T01:34:08.432 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.107:6789/0 mon.noname-f-legacy 2023-09-30T01:34:08.432 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.noname-b 2023-09-30T01:34:08.432 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.noname-g 2023-09-30T01:34:08.432 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.noname-c 2023-09-30T01:34:08.432 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.noname-h 2023-09-30T01:34:08.432 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.noname-d 2023-09-30T01:34:08.432 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.noname-i 2023-09-30T01:34:08.432 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.noname-e 2023-09-30T01:34:08.432 INFO:teuthology.orchestra.run.smithi088.stderr:9: v2:172.21.15.88:3300/0 mon.noname-a 2023-09-30T01:34:08.433 INFO:teuthology.orchestra.run.smithi088.stderr:10: v2:172.21.15.107:3300/0 mon.noname-f 2023-09-30T01:34:08.433 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:34:08.437 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:08.428+0000 7fce1b59e700 1 Processor -- start 2023-09-30T01:34:08.437 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:08.428+0000 7fce1b59e700 1 -- start start 2023-09-30T01:34:08.437 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:08.428+0000 7fce1b59e700 10 monclient: init 2023-09-30T01:34:08.437 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:08.428+0000 7fce1b59e700 10 monclient: _reopen_session rank -1 2023-09-30T01:34:08.437 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:08.428+0000 7fce1b59e700 10 monclient: _add_conns ranks=[1,0,9,7,3,5,10,8,6,2,4] 2023-09-30T01:34:08.437 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:08.428+0000 7fce1b59e700 10 monclient(hunting): picked mon.noname-f-legacy con 0x7fce1c066f90 addr v1:172.21.15.107:6789/0 2023-09-30T01:34:08.438 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:08.428+0000 7fce1b59e700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fce1c06e3a0 addr v1:172.21.15.88:6789/0 2023-09-30T01:34:08.438 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:08.428+0000 7fce1b59e700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fce1c1365b0 0x7fce1c138990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:34:08.438 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:08.428+0000 7fce1b59e700 10 monclient(hunting): picked mon.noname-a con 0x7fce1c1365b0 addr v2:172.21.15.88:3300/0 2023-09-30T01:34:08.438 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:08.428+0000 7fce1b59e700 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce1c04bb20 con 0x7fce1c06e3a0 2023-09-30T01:34:08.438 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:08.428+0000 7fce1b59e700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce1c09f150 con 0x7fce1c066f90 2023-09-30T01:34:08.438 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:08.428+0000 7fce1b59e700 10 monclient(hunting): start opening mon connection 2023-09-30T01:34:08.438 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:08.428+0000 7fce1b59e700 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fce1c06bec0 con 0x7fce1c1365b0 2023-09-30T01:34:08.438 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:08.428+0000 7fce1b59e700 10 monclient(hunting): _renew_subs 2023-09-30T01:34:08.438 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:08.428+0000 7fce1b59e700 10 monclient(hunting): authenticate will time out at 920.470734s 2023-09-30T01:34:08.438 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:08.428+0000 7fce19d9b700 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7fce1c06e3a0 0x7fce1c06ab90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:35750/0 (socket says 172.21.15.88:35750) 2023-09-30T01:34:08.438 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:08.428+0000 7fce19d9b700 1 -- 172.21.15.88:0/2141180867 learned_addr learned my addr 172.21.15.88:0/2141180867 (peer_addr_for_me v1:172.21.15.88:0/0) 2023-09-30T01:34:08.439 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:08.428+0000 7fce1ad9d700 1 --2- 172.21.15.88:0/2141180867 >> v2:172.21.15.88:3300/0 conn(0x7fce1c1365b0 0x7fce1c138990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:34:08.439 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:08.428+0000 7fce1ad9d700 10 monclient(hunting): get_auth_request con 0x7fce1c1365b0 auth_method 0 2023-09-30T01:34:08.439 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:08.428+0000 7fce1ad9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:34:08.439 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:08.428+0000 7fce1ad9d700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:34:08.439 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:08.428+0000 7fce1ad9d700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:34:08.439 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:08.428+0000 7fce1ad9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:34:08.439 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:08.428+0000 7fce1ad9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:34:08.439 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:08.428+0000 7fce1ad9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:34:08.439 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:08.428+0000 7fce1959a700 1 -- 172.21.15.88:0/2141180867 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2508721941 0 0) 0x7fce1c04bb20 con 0x7fce1c06e3a0 2023-09-30T01:34:08.439 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:08.428+0000 7fce1959a700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:34:08.439 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:08.428+0000 7fce1959a700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:34:08.440 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:08.428+0000 7fce1959a700 10 monclient(hunting): my global_id is 4581 2023-09-30T01:34:08.440 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:08.428+0000 7fce1959a700 1 -- 172.21.15.88:0/2141180867 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fce04003570 con 0x7fce1c06e3a0 2023-09-30T01:34:08.440 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:08.428+0000 7fce1ad9d700 10 monclient(hunting): handle_auth_done global_id 4572 payload 274 2023-09-30T01:34:08.440 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:08.428+0000 7fce1ad9d700 1 -- 172.21.15.88:0/2141180867 >> v1:172.21.15.107:6789/0 conn(0x7fce1c066f90 legacy=0x7fce1c06a480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:34:08.440 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:08.428+0000 7fce1ad9d700 1 -- 172.21.15.88:0/2141180867 >> v1:172.21.15.88:6789/0 conn(0x7fce1c06e3a0 legacy=0x7fce1c06ab90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:34:08.440 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:08.428+0000 7fce1ad9d700 10 monclient: _finish_hunting 0 2023-09-30T01:34:08.440 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:08.428+0000 7fce1ad9d700 1 monclient: found mon.noname-a 2023-09-30T01:34:08.440 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:08.428+0000 7fce1ad9d700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:34:08.440 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:08.428+0000 7fce1ad9d700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.88:3300/0 2023-09-30T01:34:08.441 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:08.428+0000 7fce1ad9d700 1 -- 172.21.15.88:0/2141180867 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce1c067360 con 0x7fce1c1365b0 2023-09-30T01:34:08.441 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:08.428+0000 7fce1ad9d700 1 --2- 172.21.15.88:0/2141180867 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fce1c1365b0 0x7fce1c138990 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fce14009e60 tx=0x7fce14009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6bbf8f19ab9b20a3 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:34:08.441 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:08.432+0000 7fce1959a700 1 -- 172.21.15.88:0/2141180867 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7fce1400e070 con 0x7fce1c1365b0 2023-09-30T01:34:08.441 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:08.432+0000 7fce1959a700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:34:08.441 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:08.432+0000 7fce1959a700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T01:34:08.441 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:08.432+0000 7fce1959a700 10 monclient: dump: 2023-09-30T01:34:08.441 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:34:08.441 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:34:08.441 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:34:08.441 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:34:08.441 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:34:08.441 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:34:08.442 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:34:08.442 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:34:08.442 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:34:08.442 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:34:08.442 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:34:08.442 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:34:08.442 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:34:08.442 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:34:08.442 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:34:08.442 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:34:08.442 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:08.432+0000 7fce1959a700 10 monclient: _finish_auth 0 2023-09-30T01:34:08.443 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:08.432+0000 7fce1959a700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:34:08.443 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:08.432+0000 7fce1b59e700 5 monclient: authenticate success, global_id 4572 2023-09-30T01:34:08.443 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:08.432+0000 7fce1b59e700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:34:08.443 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:08.432+0000 7fce1959a700 1 -- 172.21.15.88:0/2141180867 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce14012430 con 0x7fce1c1365b0 2023-09-30T01:34:08.443 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:08.432+0000 7fce1959a700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:34:08.443 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:08.432+0000 7fce1b59e700 10 monclient: get_monmap_and_config success 2023-09-30T01:34:08.443 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:08.432+0000 7fce1b59e700 4 set_mon_vals no callback set 2023-09-30T01:34:08.443 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:08.432+0000 7fce1959a700 1 -- 172.21.15.88:0/2141180867 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7fce14012a20 con 0x7fce1c1365b0 2023-09-30T01:34:08.443 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:08.432+0000 7fce1959a700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:34:08.443 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:08.432+0000 7fce1959a700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:34:08.443 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:08.432+0000 7fce1959a700 10 monclient: dump: 2023-09-30T01:34:08.443 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:34:08.443 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:34:08.444 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:34:08.444 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:34:08.444 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:34:08.444 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:34:08.444 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:34:08.444 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:34:08.444 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:34:08.444 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:34:08.444 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:34:08.444 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:34:08.444 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:34:08.444 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:34:08.445 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:34:08.445 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:34:08.445 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:08.432+0000 7fce1b59e700 10 monclient: shutdown 2023-09-30T01:34:08.445 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:08.432+0000 7fce1b59e700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:34:08.445 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:08.432+0000 7fce1b59e700 1 -- 172.21.15.88:0/2141180867 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fce1c1365b0 msgr2=0x7fce1c138990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:34:08.445 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:08.432+0000 7fce1b59e700 1 --2- 172.21.15.88:0/2141180867 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fce1c1365b0 0x7fce1c138990 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fce14009e60 tx=0x7fce14009780 comp rx=0 tx=0).stop 2023-09-30T01:34:08.445 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:08.432+0000 7fce1b59e700 1 -- 172.21.15.88:0/2141180867 shutdown_connections 2023-09-30T01:34:08.445 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:08.432+0000 7fce1b59e700 1 --2- 172.21.15.88:0/2141180867 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fce1c1365b0 0x7fce1c138990 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:34:08.445 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:08.432+0000 7fce1b59e700 1 -- 172.21.15.88:0/2141180867 >> 172.21.15.88:0/2141180867 conn(0x7fce1c063610 msgr2=0x7fce1c0a2b90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:34:08.445 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:08.432+0000 7fce1b59e700 1 -- 172.21.15.88:0/2141180867 shutdown_connections 2023-09-30T01:34:08.445 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:08.432+0000 7fce1b59e700 1 -- 172.21.15.88:0/2141180867 wait complete. 2023-09-30T01:34:08.450 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:08.432+0000 7fce1b59e700 10 monclient: build_initial_monmap 2023-09-30T01:34:08.450 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:08.432+0000 7fce1b59e700 10 monclient: monmap: 2023-09-30T01:34:08.450 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2023-09-30T01:34:08.450 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:34:08.450 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 0.000000 2023-09-30T01:34:08.450 INFO:teuthology.orchestra.run.smithi088.stderr:created 0.000000 2023-09-30T01:34:08.450 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2023-09-30T01:34:08.450 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 1 2023-09-30T01:34:08.450 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.noname-a 2023-09-30T01:34:08.450 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.noname-f 2023-09-30T01:34:08.450 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.noname-b 2023-09-30T01:34:08.451 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.noname-g 2023-09-30T01:34:08.451 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.noname-c 2023-09-30T01:34:08.451 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.noname-h 2023-09-30T01:34:08.451 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.noname-d 2023-09-30T01:34:08.451 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.noname-i 2023-09-30T01:34:08.451 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.noname-e 2023-09-30T01:34:08.451 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:34:08.451 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:08.432+0000 7fce1b59e700 1 Processor -- start 2023-09-30T01:34:08.451 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:08.432+0000 7fce1b59e700 1 -- start start 2023-09-30T01:34:08.451 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:08.432+0000 7fce1b59e700 10 monclient: init 2023-09-30T01:34:08.451 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:08.436+0000 7fce1b59e700 10 monclient: _reopen_session rank -1 2023-09-30T01:34:08.451 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:08.436+0000 7fce1b59e700 10 monclient: _add_conns ranks=[7,1,4,0,5,6,8,3,2] 2023-09-30T01:34:08.452 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:08.436+0000 7fce1b59e700 1 --2- >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7fce1c066f90 0x7fce1c09c370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:34:08.452 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:08.436+0000 7fce1b59e700 10 monclient(hunting): picked mon.noname-i con 0x7fce1c066f90 addr [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] 2023-09-30T01:34:08.452 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:08.436+0000 7fce1b59e700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fce1c06e3a0 0x7fce1c13de60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:34:08.452 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:08.436+0000 7fce1b59e700 10 monclient(hunting): picked mon.noname-f con 0x7fce1c06e3a0 addr [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] 2023-09-30T01:34:08.452 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:08.436+0000 7fce1b59e700 1 --2- >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7fce1c1365b0 0x7fce1c13e3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:34:08.452 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:08.436+0000 7fce1b59e700 10 monclient(hunting): picked mon.noname-c con 0x7fce1c1365b0 addr [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] 2023-09-30T01:34:08.452 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:08.436+0000 7fce1b59e700 10 monclient(hunting): start opening mon connection 2023-09-30T01:34:08.452 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:08.436+0000 7fce1b59e700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce1c0991d0 con 0x7fce1c06e3a0 2023-09-30T01:34:08.452 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:08.436+0000 7fce1b59e700 10 monclient(hunting): start opening mon connection 2023-09-30T01:34:08.452 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:08.436+0000 7fce1b59e700 1 -- --> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] -- mon_getmap magic: 0 v1 -- 0x7fce1c099980 con 0x7fce1c1365b0 2023-09-30T01:34:08.452 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:08.436+0000 7fce1b59e700 10 monclient(hunting): start opening mon connection 2023-09-30T01:34:08.452 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:08.436+0000 7fce1b59e700 1 -- --> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] -- mon_getmap magic: 0 v1 -- 0x7fce1c09c8b0 con 0x7fce1c066f90 2023-09-30T01:34:08.453 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:08.436+0000 7fce1b59e700 10 monclient(hunting): _renew_subs 2023-09-30T01:34:08.453 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:08.436+0000 7fce1b59e700 10 monclient(hunting): authenticate will time out at 920.477206s 2023-09-30T01:34:08.453 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:08.436+0000 7fce1ad9d700 1 --2- >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7fce1c1365b0 0x7fce1c13e3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:34:08.453 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:08.436+0000 7fce1ad9d700 1 --2- >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7fce1c1365b0 0x7fce1c13e3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3302/0 says I am v2:172.21.15.88:47202/0 (socket says 172.21.15.88:47202) 2023-09-30T01:34:08.453 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:08.436+0000 7fce1ad9d700 1 -- 172.21.15.88:0/2955291725 learned_addr learned my addr 172.21.15.88:0/2955291725 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-09-30T01:34:08.453 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:08.436+0000 7fce1ad9d700 10 monclient(hunting): get_auth_request con 0x7fce1c1365b0 auth_method 0 2023-09-30T01:34:08.453 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:08.436+0000 7fce19d9b700 1 --2- 172.21.15.88:0/2955291725 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fce1c06e3a0 0x7fce1c13de60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:34:08.453 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:08.436+0000 7fce1ad9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:34:08.453 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:08.436+0000 7fce1ad9d700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:34:08.453 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:08.436+0000 7fce1ad9d700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:34:08.453 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:08.436+0000 7fce1a59c700 1 --2- 172.21.15.88:0/2955291725 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7fce1c066f90 0x7fce1c09c370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:34:08.453 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:08.436+0000 7fce19d9b700 10 monclient(hunting): get_auth_request con 0x7fce1c06e3a0 auth_method 0 2023-09-30T01:34:08.453 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:08.436+0000 7fce19d9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:34:08.454 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:08.436+0000 7fce19d9b700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:34:08.454 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:08.436+0000 7fce19d9b700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:34:08.454 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:08.436+0000 7fce1a59c700 10 monclient(hunting): get_auth_request con 0x7fce1c066f90 auth_method 0 2023-09-30T01:34:08.454 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:08.436+0000 7fce1a59c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:34:08.454 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:08.436+0000 7fce1a59c700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:34:08.454 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:08.436+0000 7fce1a59c700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:34:08.454 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:08.436+0000 7fce1ad9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:34:08.454 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:08.436+0000 7fce1ad9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:34:08.454 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:08.436+0000 7fce1ad9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:34:08.454 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:08.436+0000 7fce19d9b700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:34:08.454 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:08.436+0000 7fce19d9b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:34:08.454 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:08.436+0000 7fce19d9b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:34:08.455 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:08.436+0000 7fce1a59c700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:34:08.455 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:08.436+0000 7fce1a59c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:34:08.455 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:08.436+0000 7fce1a59c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:34:08.455 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:08.436+0000 7fce1ad9d700 10 monclient(hunting): handle_auth_done global_id 4477 payload 824 2023-09-30T01:34:08.455 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:08.436+0000 7fce1ad9d700 1 -- 172.21.15.88:0/2955291725 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7fce1c066f90 msgr2=0x7fce1c09c370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:34:08.455 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:08.436+0000 7fce1ad9d700 1 --2- 172.21.15.88:0/2955291725 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7fce1c066f90 0x7fce1c09c370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:34:08.455 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:08.436+0000 7fce1ad9d700 1 -- 172.21.15.88:0/2955291725 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fce1c06e3a0 msgr2=0x7fce1c13de60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:34:08.455 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:08.436+0000 7fce1ad9d700 1 --2- 172.21.15.88:0/2955291725 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fce1c06e3a0 0x7fce1c13de60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:34:08.455 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:08.436+0000 7fce1ad9d700 10 monclient: _finish_hunting 0 2023-09-30T01:34:08.455 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:08.436+0000 7fce1ad9d700 1 monclient: found mon.noname-c 2023-09-30T01:34:08.455 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:08.436+0000 7fce1ad9d700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:34:08.455 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:08.436+0000 7fce1ad9d700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.88:3302/0 2023-09-30T01:34:08.455 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:08.436+0000 7fce1ad9d700 1 -- 172.21.15.88:0/2955291725 --> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce14009a20 con 0x7fce1c1365b0 2023-09-30T01:34:08.456 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:08.436+0000 7fce1ad9d700 1 --2- 172.21.15.88:0/2955291725 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7fce1c1365b0 0x7fce1c13e3a0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fce1c06c000 tx=0x7fce14019000 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:34:08.456 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:08.436+0000 7fce0a7fc700 1 -- 172.21.15.88:0/2955291725 <== mon.4 v2:172.21.15.88:3302/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7fce140032b0 con 0x7fce1c1365b0 2023-09-30T01:34:08.456 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:08.436+0000 7fce0a7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:34:08.456 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:08.436+0000 7fce0a7fc700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T01:34:08.456 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:08.436+0000 7fce0a7fc700 10 monclient: dump: 2023-09-30T01:34:08.456 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:34:08.456 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:34:08.456 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:34:08.456 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:34:08.456 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:34:08.456 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:34:08.456 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:34:08.456 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:34:08.457 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:34:08.457 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:34:08.457 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:34:08.457 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:34:08.457 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:34:08.457 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:34:08.457 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:34:08.457 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:34:08.457 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:08.436+0000 7fce0a7fc700 10 monclient: _finish_auth 0 2023-09-30T01:34:08.457 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:08.436+0000 7fce0a7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:34:08.457 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:08.436+0000 7fce0a7fc700 1 -- 172.21.15.88:0/2955291725 <== mon.4 v2:172.21.15.88:3302/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce14012ca0 con 0x7fce1c1365b0 2023-09-30T01:34:08.457 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:08.436+0000 7fce1b59e700 5 monclient: authenticate success, global_id 4477 2023-09-30T01:34:08.457 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:08.436+0000 7fce1b59e700 10 monclient: _renew_subs 2023-09-30T01:34:08.458 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:08.436+0000 7fce1b59e700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.88:3302/0 2023-09-30T01:34:08.458 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:08.436+0000 7fce1b59e700 1 -- 172.21.15.88:0/2955291725 --> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce1c09b410 con 0x7fce1c1365b0 2023-09-30T01:34:08.458 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:08.436+0000 7fce0a7fc700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:34:08.458 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:08.436+0000 7fce0a7fc700 1 -- 172.21.15.88:0/2955291725 <== mon.4 v2:172.21.15.88:3302/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7fce14012e10 con 0x7fce1c1365b0 2023-09-30T01:34:08.458 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:08.436+0000 7fce0a7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:34:08.458 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:08.436+0000 7fce0a7fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T01:34:08.458 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:08.436+0000 7fce0bfff700 4 set_mon_vals no callback set 2023-09-30T01:34:08.458 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:08.436+0000 7fce0a7fc700 10 monclient: dump: 2023-09-30T01:34:08.458 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:34:08.458 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:34:08.458 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:34:08.458 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:34:08.459 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:34:08.459 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:34:08.459 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:34:08.459 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:34:08.459 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:34:08.459 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:34:08.459 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:34:08.459 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:34:08.459 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:34:08.459 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:34:08.459 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:34:08.459 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:34:08.459 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:08.436+0000 7fce1b59e700 10 monclient: _renew_subs 2023-09-30T01:34:08.460 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:08.436+0000 7fce1b59e700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.88:3302/0 2023-09-30T01:34:08.460 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:08.436+0000 7fce1b59e700 1 -- 172.21.15.88:0/2955291725 --> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fce1c09b970 con 0x7fce1c1365b0 2023-09-30T01:34:08.460 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:08.440+0000 7fce1b59e700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:34:08.460 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:08.440+0000 7fce1b59e700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:34:08.460 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:08.440+0000 7fce1b59e700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.88:3302/0 2023-09-30T01:34:08.460 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:08.440+0000 7fce1b59e700 1 -- 172.21.15.88:0/2955291725 --> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fce1c071960 con 0x7fce1c1365b0 2023-09-30T01:34:08.460 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:08.440+0000 7fce0a7fc700 1 -- 172.21.15.88:0/2955291725 <== mon.4 v2:172.21.15.88:3302/0 4 ==== mgrmap(e 5) v1 ==== 40144+0+0 (secure 0 0 0) 0x7fce140263f0 con 0x7fce1c1365b0 2023-09-30T01:34:08.460 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:08.440+0000 7fce0a7fc700 1 --2- 172.21.15.88:0/2955291725 >> v2:172.21.15.107:6812/29926 conn(0x7fcdfc0332c0 0x7fcdfc035770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:34:08.460 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:08.440+0000 7fce0a7fc700 1 -- 172.21.15.88:0/2955291725 <== mon.4 v2:172.21.15.88:3302/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 3502+0+0 (secure 0 0 0) 0x7fce14044f30 con 0x7fce1c1365b0 2023-09-30T01:34:08.460 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:08.440+0000 7fce1a59c700 1 --2- 172.21.15.88:0/2955291725 >> v2:172.21.15.107:6812/29926 conn(0x7fcdfc0332c0 0x7fcdfc035770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:34:08.460 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:08.440+0000 7fce1a59c700 10 monclient: get_auth_request con 0x7fcdfc0332c0 auth_method 0 2023-09-30T01:34:08.460 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:08.440+0000 7fce1a59c700 1 --2- 172.21.15.88:0/2955291725 >> v2:172.21.15.107:6812/29926 conn(0x7fcdfc0332c0 0x7fcdfc035770 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fce0c003a80 tx=0x7fce0c007ec0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:34:08.461 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:08.444+0000 7fce0a7fc700 1 -- 172.21.15.88:0/2955291725 <== mon.4 v2:172.21.15.88:3302/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fce14074be0 con 0x7fce1c1365b0 2023-09-30T01:34:08.461 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:08.444+0000 7fce0a7fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:34:08.461 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:08.444+0000 7fce0a7fc700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:34:08.646 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:08.640+0000 7fce1b59e700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T01:34:08.646 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:08.640+0000 7fce1b59e700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:34:08.646 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:08.640+0000 7fce1b59e700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.88:3302/0 2023-09-30T01:34:08.646 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:08.640+0000 7fce1b59e700 1 -- 172.21.15.88:0/2955291725 --> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fce1c068420 con 0x7fce1c1365b0 2023-09-30T01:34:08.647 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:08.640+0000 7fce0a7fc700 1 -- 172.21.15.88:0/2955291725 <== mon.4 v2:172.21.15.88:3302/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7fce1402e720 con 0x7fce1c1365b0 2023-09-30T01:34:08.647 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:08.640+0000 7fce0a7fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:34:08.647 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:08.640+0000 7fce0a7fc700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:34:08.647 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-09-30T01:34:08.647 INFO:teuthology.orchestra.run.smithi088.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:34:08.649 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:08.644+0000 7fce1b59e700 1 -- 172.21.15.88:0/2955291725 >> v2:172.21.15.107:6812/29926 conn(0x7fcdfc0332c0 msgr2=0x7fcdfc035770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:34:08.649 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:08.644+0000 7fce1b59e700 1 --2- 172.21.15.88:0/2955291725 >> v2:172.21.15.107:6812/29926 conn(0x7fcdfc0332c0 0x7fcdfc035770 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fce0c003a80 tx=0x7fce0c007ec0 comp rx=0 tx=0).stop 2023-09-30T01:34:08.649 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:08.644+0000 7fce1b59e700 10 monclient: shutdown 2023-09-30T01:34:08.650 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:08.644+0000 7fce1b59e700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:34:08.650 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:08.644+0000 7fce1b59e700 1 -- 172.21.15.88:0/2955291725 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7fce1c1365b0 msgr2=0x7fce1c13e3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:34:08.650 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:08.644+0000 7fce1b59e700 1 --2- 172.21.15.88:0/2955291725 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7fce1c1365b0 0x7fce1c13e3a0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fce1c06c000 tx=0x7fce14019000 comp rx=0 tx=0).stop 2023-09-30T01:34:08.650 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:08.644+0000 7fce1b59e700 1 -- 172.21.15.88:0/2955291725 shutdown_connections 2023-09-30T01:34:08.650 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:08.644+0000 7fce1b59e700 1 --2- 172.21.15.88:0/2955291725 >> v2:172.21.15.107:6812/29926 conn(0x7fcdfc0332c0 0x7fcdfc035770 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:34:08.650 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:08.644+0000 7fce1b59e700 1 --2- 172.21.15.88:0/2955291725 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7fce1c1365b0 0x7fce1c13e3a0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:34:08.650 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:08.644+0000 7fce1b59e700 1 --2- 172.21.15.88:0/2955291725 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fce1c06e3a0 0x7fce1c13de60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:34:08.650 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:08.644+0000 7fce1b59e700 1 --2- 172.21.15.88:0/2955291725 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7fce1c066f90 0x7fce1c09c370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:34:08.650 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:08.644+0000 7fce1b59e700 1 -- 172.21.15.88:0/2955291725 >> 172.21.15.88:0/2955291725 conn(0x7fce1c063610 msgr2=0x7fce1c0a2af0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:34:08.650 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:08.644+0000 7fce1b59e700 1 -- 172.21.15.88:0/2955291725 shutdown_connections 2023-09-30T01:34:08.650 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:08.644+0000 7fce1b59e700 1 -- 172.21.15.88:0/2955291725 wait complete. 2023-09-30T01:34:11.661 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:34:11.771 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:11.764+0000 7f3dfdcdf700 10 monclient: get_monmap_and_config 2023-09-30T01:34:11.771 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:11.764+0000 7f3dfdcdf700 10 monclient: build_initial_monmap 2023-09-30T01:34:11.771 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:11.764+0000 7f3dfdcdf700 10 monclient: monmap: 2023-09-30T01:34:11.771 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2023-09-30T01:34:11.771 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:34:11.771 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:34:11.771657+0000 2023-09-30T01:34:11.772 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:34:11.771657+0000 2023-09-30T01:34:11.772 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2023-09-30T01:34:11.772 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:34:11.772 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a-legacy 2023-09-30T01:34:11.772 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.107:6789/0 mon.noname-f-legacy 2023-09-30T01:34:11.772 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.noname-b 2023-09-30T01:34:11.772 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.noname-g 2023-09-30T01:34:11.772 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.noname-c 2023-09-30T01:34:11.772 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.noname-h 2023-09-30T01:34:11.772 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.noname-d 2023-09-30T01:34:11.772 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.noname-i 2023-09-30T01:34:11.772 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.noname-e 2023-09-30T01:34:11.773 INFO:teuthology.orchestra.run.smithi088.stderr:9: v2:172.21.15.88:3300/0 mon.noname-a 2023-09-30T01:34:11.773 INFO:teuthology.orchestra.run.smithi088.stderr:10: v2:172.21.15.107:3300/0 mon.noname-f 2023-09-30T01:34:11.773 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:34:11.776 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:11.768+0000 7f3dfdcdf700 1 Processor -- start 2023-09-30T01:34:11.777 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:11.768+0000 7f3dfdcdf700 1 -- start start 2023-09-30T01:34:11.777 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:11.768+0000 7f3dfdcdf700 10 monclient: init 2023-09-30T01:34:11.777 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:11.768+0000 7f3dfdcdf700 10 monclient: _reopen_session rank -1 2023-09-30T01:34:11.777 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:11.768+0000 7f3dfdcdf700 10 monclient: _add_conns ranks=[8,6,0,10,1,4,7,2,5,9,3] 2023-09-30T01:34:11.777 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:11.768+0000 7f3dfdcdf700 1 --2- >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f3df8097a40 0x7f3df8132700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:34:11.777 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:11.768+0000 7f3dfdcdf700 10 monclient(hunting): picked mon.noname-e con 0x7f3df8097a40 addr [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] 2023-09-30T01:34:11.777 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:11.768+0000 7f3dfdcdf700 1 --2- >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7f3df81311c0 0x7f3df8132c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:34:11.777 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:11.768+0000 7f3dfdcdf700 10 monclient(hunting): picked mon.noname-d con 0x7f3df81311c0 addr [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] 2023-09-30T01:34:11.777 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:11.768+0000 7f3dfdcdf700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f3df81332e0 addr v1:172.21.15.88:6789/0 2023-09-30T01:34:11.777 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:11.768+0000 7f3dfdcdf700 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3df804bb20 con 0x7f3df81332e0 2023-09-30T01:34:11.777 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:11.768+0000 7f3dfdcdf700 10 monclient(hunting): start opening mon connection 2023-09-30T01:34:11.778 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:11.768+0000 7f3dfdcdf700 1 -- --> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] -- mon_getmap magic: 0 v1 -- 0x7f3df8131590 con 0x7f3df81311c0 2023-09-30T01:34:11.778 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:11.768+0000 7f3dfdcdf700 10 monclient(hunting): start opening mon connection 2023-09-30T01:34:11.778 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:11.768+0000 7f3dfdcdf700 1 -- --> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] -- mon_getmap magic: 0 v1 -- 0x7f3df809a5d0 con 0x7f3df8097a40 2023-09-30T01:34:11.778 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:11.768+0000 7f3dfdcdf700 10 monclient(hunting): _renew_subs 2023-09-30T01:34:11.778 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:11.768+0000 7f3dfdcdf700 10 monclient(hunting): authenticate will time out at 923.809568s 2023-09-30T01:34:11.778 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:11.768+0000 7f3dfccdd700 1 --2- >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f3df8097a40 0x7f3df8132700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:34:11.778 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:11.768+0000 7f3dfccdd700 1 --2- >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f3df8097a40 0x7f3df8132700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3304/0 says I am v2:172.21.15.88:50308/0 (socket says 172.21.15.88:50308) 2023-09-30T01:34:11.778 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:11.768+0000 7f3dfccdd700 1 -- 172.21.15.88:0/3045760454 learned_addr learned my addr 172.21.15.88:0/3045760454 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-09-30T01:34:11.778 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:11.768+0000 7f3dfccdd700 10 monclient(hunting): get_auth_request con 0x7f3df8097a40 auth_method 0 2023-09-30T01:34:11.778 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:11.768+0000 7f3dfccdd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:34:11.778 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:11.768+0000 7f3dfccdd700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:34:11.779 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:11.768+0000 7f3dfccdd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:34:11.779 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:11.768+0000 7f3df7fff700 1 --2- 172.21.15.88:0/3045760454 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7f3df81311c0 0x7f3df8132c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:34:11.779 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:11.768+0000 7f3df7fff700 10 monclient(hunting): get_auth_request con 0x7f3df81311c0 auth_method 0 2023-09-30T01:34:11.779 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:11.768+0000 7f3df7fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:34:11.779 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:11.768+0000 7f3df7fff700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:34:11.779 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:11.768+0000 7f3df7fff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:34:11.779 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:11.768+0000 7f3dfccdd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:34:11.779 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:11.768+0000 7f3dfccdd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:34:11.779 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:11.768+0000 7f3dfccdd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:34:11.779 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:11.768+0000 7f3df7fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:34:11.779 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:11.768+0000 7f3df7fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:34:11.779 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:11.768+0000 7f3df7fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:34:11.779 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:11.768+0000 7f3dfccdd700 10 monclient(hunting): handle_auth_done global_id 4517 payload 274 2023-09-30T01:34:11.780 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:11.768+0000 7f3dfccdd700 1 -- 172.21.15.88:0/3045760454 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7f3df81311c0 msgr2=0x7f3df8132c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:34:11.780 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:11.768+0000 7f3dfccdd700 1 --2- 172.21.15.88:0/3045760454 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7f3df81311c0 0x7f3df8132c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:34:11.780 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:11.768+0000 7f3dfccdd700 1 -- 172.21.15.88:0/3045760454 >> v1:172.21.15.88:6789/0 conn(0x7f3df81332e0 legacy=0x7f3df8133710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:34:11.780 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:11.768+0000 7f3dfccdd700 10 monclient: _finish_hunting 0 2023-09-30T01:34:11.780 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:11.768+0000 7f3dfccdd700 1 monclient: found mon.noname-e 2023-09-30T01:34:11.780 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:11.768+0000 7f3dfccdd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:34:11.780 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:11.768+0000 7f3dfccdd700 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.88:3304/0 2023-09-30T01:34:11.780 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:11.768+0000 7f3dfccdd700 1 -- 172.21.15.88:0/3045760454 --> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3df8097e10 con 0x7f3df8097a40 2023-09-30T01:34:11.780 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:11.768+0000 7f3dfccdd700 1 --2- 172.21.15.88:0/3045760454 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f3df8097a40 0x7f3df8132700 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f3de80099e0 tx=0x7f3de8009250 comp rx=0 tx=0).ready entity=mon.8 client_cookie=56d87917608b66f9 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:34:11.780 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:11.768+0000 7f3df77fe700 1 -- 172.21.15.88:0/3045760454 <== mon.8 v2:172.21.15.88:3304/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f3de800bce0 con 0x7f3df8097a40 2023-09-30T01:34:11.780 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:11.768+0000 7f3df77fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:34:11.780 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:11.768+0000 7f3df77fe700 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2023-09-30T01:34:11.781 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:11.768+0000 7f3df77fe700 10 monclient: dump: 2023-09-30T01:34:11.781 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:34:11.781 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:34:11.781 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:34:11.781 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:34:11.781 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:34:11.781 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:34:11.781 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:34:11.781 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:34:11.781 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:34:11.781 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:34:11.781 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:34:11.781 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:34:11.782 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:34:11.782 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:34:11.782 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:34:11.782 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:34:11.782 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:11.768+0000 7f3df77fe700 10 monclient: _finish_auth 0 2023-09-30T01:34:11.782 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:11.768+0000 7f3df77fe700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:34:11.782 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:11.768+0000 7f3df77fe700 1 -- 172.21.15.88:0/3045760454 <== mon.8 v2:172.21.15.88:3304/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3de800be60 con 0x7f3df8097a40 2023-09-30T01:34:11.782 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:11.768+0000 7f3dfdcdf700 5 monclient: authenticate success, global_id 4517 2023-09-30T01:34:11.782 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:11.768+0000 7f3dfdcdf700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:34:11.782 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:11.768+0000 7f3df77fe700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:34:11.782 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:11.768+0000 7f3dfdcdf700 10 monclient: get_monmap_and_config success 2023-09-30T01:34:11.782 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:11.768+0000 7f3df77fe700 1 -- 172.21.15.88:0/3045760454 <== mon.8 v2:172.21.15.88:3304/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f3de80118a0 con 0x7f3df8097a40 2023-09-30T01:34:11.783 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:11.768+0000 7f3df77fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:34:11.783 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:11.768+0000 7f3dfdcdf700 4 set_mon_vals no callback set 2023-09-30T01:34:11.783 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:11.768+0000 7f3df77fe700 10 monclient: got monmap 1 from mon.e (according to old e1) 2023-09-30T01:34:11.783 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:11.768+0000 7f3df77fe700 10 monclient: dump: 2023-09-30T01:34:11.783 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:34:11.783 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:34:11.783 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:34:11.783 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:34:11.783 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:34:11.783 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:34:11.783 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:34:11.783 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:34:11.783 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:34:11.784 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:34:11.784 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:34:11.784 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:34:11.784 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:34:11.784 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:34:11.784 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:34:11.784 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:34:11.784 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:11.772+0000 7f3dfdcdf700 10 monclient: shutdown 2023-09-30T01:34:11.784 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:11.772+0000 7f3dfdcdf700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:34:11.784 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:11.772+0000 7f3dfdcdf700 1 -- 172.21.15.88:0/3045760454 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f3df8097a40 msgr2=0x7f3df8132700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:34:11.784 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:11.772+0000 7f3dfdcdf700 1 --2- 172.21.15.88:0/3045760454 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f3df8097a40 0x7f3df8132700 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f3de80099e0 tx=0x7f3de8009250 comp rx=0 tx=0).stop 2023-09-30T01:34:11.784 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:11.772+0000 7f3dfdcdf700 1 -- 172.21.15.88:0/3045760454 shutdown_connections 2023-09-30T01:34:11.785 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:11.772+0000 7f3dfdcdf700 1 --2- 172.21.15.88:0/3045760454 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7f3df81311c0 0x7f3df8132c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:34:11.785 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:11.772+0000 7f3dfdcdf700 1 --2- 172.21.15.88:0/3045760454 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f3df8097a40 0x7f3df8132700 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:34:11.785 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:11.772+0000 7f3dfdcdf700 1 -- 172.21.15.88:0/3045760454 >> 172.21.15.88:0/3045760454 conn(0x7f3df806e5e0 msgr2=0x7f3df806e9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:34:11.785 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:11.772+0000 7f3dfdcdf700 1 -- 172.21.15.88:0/3045760454 shutdown_connections 2023-09-30T01:34:11.785 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:11.772+0000 7f3dfdcdf700 1 -- 172.21.15.88:0/3045760454 wait complete. 2023-09-30T01:34:11.785 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:11.772+0000 7f3dfdcdf700 10 monclient: build_initial_monmap 2023-09-30T01:34:11.785 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:11.772+0000 7f3dfdcdf700 10 monclient: monmap: 2023-09-30T01:34:11.785 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2023-09-30T01:34:11.785 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:34:11.785 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 0.000000 2023-09-30T01:34:11.785 INFO:teuthology.orchestra.run.smithi088.stderr:created 0.000000 2023-09-30T01:34:11.785 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2023-09-30T01:34:11.785 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 1 2023-09-30T01:34:11.786 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.noname-a 2023-09-30T01:34:11.786 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.noname-f 2023-09-30T01:34:11.786 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.noname-b 2023-09-30T01:34:11.786 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.noname-g 2023-09-30T01:34:11.786 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.noname-c 2023-09-30T01:34:11.786 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.noname-h 2023-09-30T01:34:11.786 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.noname-d 2023-09-30T01:34:11.786 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.noname-i 2023-09-30T01:34:11.786 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.noname-e 2023-09-30T01:34:11.786 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:34:11.791 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:11.772+0000 7f3dfdcdf700 1 Processor -- start 2023-09-30T01:34:11.791 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:11.772+0000 7f3dfdcdf700 1 -- start start 2023-09-30T01:34:11.791 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:11.772+0000 7f3dfdcdf700 10 monclient: init 2023-09-30T01:34:11.791 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:11.772+0000 7f3dfdcdf700 10 monclient: _reopen_session rank -1 2023-09-30T01:34:11.791 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:11.776+0000 7f3dfdcdf700 10 monclient: _add_conns ranks=[1,7,2,6,3,0,8,5,4] 2023-09-30T01:34:11.792 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:11.776+0000 7f3dfdcdf700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3df8097a40 0x7f3df811cd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:34:11.792 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:11.776+0000 7f3dfdcdf700 10 monclient(hunting): picked mon.noname-f con 0x7f3df8097a40 addr [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] 2023-09-30T01:34:11.792 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:11.776+0000 7f3dfdcdf700 1 --2- >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7f3df81311c0 0x7f3df811d290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:34:11.792 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:11.776+0000 7f3dfdcdf700 10 monclient(hunting): picked mon.noname-i con 0x7f3df81311c0 addr [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] 2023-09-30T01:34:11.792 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:11.776+0000 7f3dfdcdf700 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f3df81332e0 0x7f3df811d7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:34:11.792 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:11.776+0000 7f3dfdcdf700 10 monclient(hunting): picked mon.noname-b con 0x7f3df81332e0 addr [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] 2023-09-30T01:34:11.792 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:11.776+0000 7f3dfdcdf700 10 monclient(hunting): start opening mon connection 2023-09-30T01:34:11.792 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:11.776+0000 7f3dfdcdf700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3df811f520 con 0x7f3df8097a40 2023-09-30T01:34:11.792 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:11.776+0000 7f3dfdcdf700 10 monclient(hunting): start opening mon connection 2023-09-30T01:34:11.792 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:11.776+0000 7f3dfdcdf700 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3df8099c30 con 0x7f3df81332e0 2023-09-30T01:34:11.792 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:11.776+0000 7f3dfdcdf700 10 monclient(hunting): start opening mon connection 2023-09-30T01:34:11.793 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:11.776+0000 7f3dfdcdf700 1 -- --> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] -- mon_getmap magic: 0 v1 -- 0x7f3df80996b0 con 0x7f3df81311c0 2023-09-30T01:34:11.793 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:11.776+0000 7f3dfdcdf700 10 monclient(hunting): _renew_subs 2023-09-30T01:34:11.793 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:11.776+0000 7f3dfdcdf700 10 monclient(hunting): authenticate will time out at 923.816259s 2023-09-30T01:34:11.793 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:11.776+0000 7f3dfd4de700 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f3df81332e0 0x7f3df811d7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:34:11.793 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:11.776+0000 7f3dfd4de700 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f3df81332e0 0x7f3df811d7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3301/0 says I am v2:172.21.15.88:47402/0 (socket says 172.21.15.88:47402) 2023-09-30T01:34:11.793 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:11.776+0000 7f3dfd4de700 1 -- 172.21.15.88:0/1899070093 learned_addr learned my addr 172.21.15.88:0/1899070093 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-09-30T01:34:11.793 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:11.776+0000 7f3dfd4de700 10 monclient(hunting): get_auth_request con 0x7f3df81332e0 auth_method 0 2023-09-30T01:34:11.793 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:11.776+0000 7f3dfd4de700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:34:11.793 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:11.776+0000 7f3dfd4de700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:34:11.793 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:11.776+0000 7f3dfd4de700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:34:11.793 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:11.776+0000 7f3df7fff700 1 --2- 172.21.15.88:0/1899070093 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7f3df81311c0 0x7f3df811d290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:34:11.794 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:11.776+0000 7f3dfccdd700 1 --2- 172.21.15.88:0/1899070093 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3df8097a40 0x7f3df811cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:34:11.794 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:11.776+0000 7f3dfd4de700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:34:11.794 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:11.776+0000 7f3dfd4de700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:34:11.794 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:11.776+0000 7f3dfd4de700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:34:11.794 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:11.776+0000 7f3df7fff700 10 monclient(hunting): get_auth_request con 0x7f3df81311c0 auth_method 0 2023-09-30T01:34:11.794 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:11.776+0000 7f3df7fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:34:11.794 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:11.776+0000 7f3df7fff700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:34:11.794 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:11.776+0000 7f3df7fff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:34:11.794 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:11.776+0000 7f3dfccdd700 10 monclient(hunting): get_auth_request con 0x7f3df8097a40 auth_method 0 2023-09-30T01:34:11.794 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:11.776+0000 7f3dfccdd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:34:11.794 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:11.776+0000 7f3dfccdd700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:34:11.794 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:11.776+0000 7f3dfccdd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:34:11.794 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:11.776+0000 7f3df7fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:34:11.795 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:11.776+0000 7f3df7fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:34:11.795 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:11.776+0000 7f3df7fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:34:11.795 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:11.776+0000 7f3dfccdd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:34:11.795 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:11.776+0000 7f3dfccdd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:34:11.795 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:11.776+0000 7f3dfccdd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:34:11.795 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:11.776+0000 7f3dfd4de700 10 monclient(hunting): handle_auth_done global_id 4439 payload 824 2023-09-30T01:34:11.795 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:11.776+0000 7f3dfd4de700 1 -- 172.21.15.88:0/1899070093 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7f3df81311c0 msgr2=0x7f3df811d290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:34:11.795 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:11.776+0000 7f3dfd4de700 1 --2- 172.21.15.88:0/1899070093 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7f3df81311c0 0x7f3df811d290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:34:11.795 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:11.776+0000 7f3dfd4de700 1 -- 172.21.15.88:0/1899070093 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3df8097a40 msgr2=0x7f3df811cd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:34:11.795 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:11.776+0000 7f3dfd4de700 1 --2- 172.21.15.88:0/1899070093 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3df8097a40 0x7f3df811cd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:34:11.795 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:11.776+0000 7f3dfd4de700 10 monclient: _finish_hunting 0 2023-09-30T01:34:11.795 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:11.776+0000 7f3dfd4de700 1 monclient: found mon.noname-b 2023-09-30T01:34:11.796 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:11.776+0000 7f3dfd4de700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:34:11.796 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:11.776+0000 7f3dfd4de700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.88:3301/0 2023-09-30T01:34:11.796 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:11.776+0000 7f3dfd4de700 1 -- 172.21.15.88:0/1899070093 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3de80094f0 con 0x7f3df81332e0 2023-09-30T01:34:11.796 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:11.776+0000 7f3dfd4de700 1 --2- 172.21.15.88:0/1899070093 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f3df81332e0 0x7f3df811d7d0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f3df000c4f0 tx=0x7f3df0007f50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:34:11.796 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:11.776+0000 7f3df4ff9700 1 -- 172.21.15.88:0/1899070093 <== mon.2 v2:172.21.15.88:3301/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f3df000ce60 con 0x7f3df81332e0 2023-09-30T01:34:11.796 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:11.776+0000 7f3df4ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:34:11.796 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:11.776+0000 7f3df4ff9700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T01:34:11.796 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:11.776+0000 7f3df4ff9700 10 monclient: dump: 2023-09-30T01:34:11.796 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:34:11.796 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:34:11.796 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:34:11.796 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:34:11.796 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:34:11.797 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:34:11.797 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:34:11.797 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:34:11.797 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:34:11.797 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:34:11.797 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:34:11.797 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:34:11.797 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:34:11.797 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:34:11.797 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:34:11.797 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:34:11.797 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:11.776+0000 7f3df4ff9700 10 monclient: _finish_auth 0 2023-09-30T01:34:11.797 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:11.776+0000 7f3df4ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:34:11.798 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:11.776+0000 7f3dfdcdf700 5 monclient: authenticate success, global_id 4439 2023-09-30T01:34:11.798 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:11.776+0000 7f3df4ff9700 1 -- 172.21.15.88:0/1899070093 <== mon.2 v2:172.21.15.88:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3df0000fe0 con 0x7f3df81332e0 2023-09-30T01:34:11.798 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:11.776+0000 7f3dfdcdf700 10 monclient: _renew_subs 2023-09-30T01:34:11.798 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:11.776+0000 7f3dfdcdf700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.88:3301/0 2023-09-30T01:34:11.798 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:11.776+0000 7f3dfdcdf700 1 -- 172.21.15.88:0/1899070093 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3df811eed0 con 0x7f3df81332e0 2023-09-30T01:34:11.798 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:11.776+0000 7f3df4ff9700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:34:11.798 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:11.776+0000 7f3df4ff9700 1 -- 172.21.15.88:0/1899070093 <== mon.2 v2:172.21.15.88:3301/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f3df0011e90 con 0x7f3df81332e0 2023-09-30T01:34:11.798 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:11.776+0000 7f3df4ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:34:11.798 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:11.776+0000 7f3df4ff9700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T01:34:11.798 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:11.776+0000 7f3df6ffd700 4 set_mon_vals no callback set 2023-09-30T01:34:11.798 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:11.776+0000 7f3df4ff9700 10 monclient: dump: 2023-09-30T01:34:11.798 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:34:11.799 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:34:11.799 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:34:11.799 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:34:11.799 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:34:11.799 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:34:11.799 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:34:11.799 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:34:11.799 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:34:11.799 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:34:11.799 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:34:11.799 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:34:11.799 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:34:11.799 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:34:11.800 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:34:11.800 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:34:11.800 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:11.776+0000 7f3dfdcdf700 10 monclient: _renew_subs 2023-09-30T01:34:11.800 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:11.776+0000 7f3dfdcdf700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.88:3301/0 2023-09-30T01:34:11.800 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:11.776+0000 7f3dfdcdf700 1 -- 172.21.15.88:0/1899070093 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3df8120160 con 0x7f3df81332e0 2023-09-30T01:34:11.800 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:11.776+0000 7f3df4ff9700 1 -- 172.21.15.88:0/1899070093 <== mon.2 v2:172.21.15.88:3301/0 4 ==== mgrmap(e 5) v1 ==== 40144+0+0 (secure 0 0 0) 0x7f3df0011590 con 0x7f3df81332e0 2023-09-30T01:34:11.800 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:11.776+0000 7f3df4ff9700 1 --2- 172.21.15.88:0/1899070093 >> v2:172.21.15.107:6812/29926 conn(0x7f3dd8033390 0x7f3dd8035840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:34:11.800 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:11.776+0000 7f3df4ff9700 1 -- 172.21.15.88:0/1899070093 <== mon.2 v2:172.21.15.88:3301/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 3502+0+0 (secure 0 0 0) 0x7f3df00422a0 con 0x7f3df81332e0 2023-09-30T01:34:11.800 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:11.776+0000 7f3dfdcdf700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:34:11.800 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:11.776+0000 7f3dfdcdf700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:34:11.800 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:11.776+0000 7f3dfdcdf700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.88:3301/0 2023-09-30T01:34:11.800 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:11.776+0000 7f3dfccdd700 1 --2- 172.21.15.88:0/1899070093 >> v2:172.21.15.107:6812/29926 conn(0x7f3dd8033390 0x7f3dd8035840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:34:11.800 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:11.776+0000 7f3dfdcdf700 1 -- 172.21.15.88:0/1899070093 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3df8071960 con 0x7f3df81332e0 2023-09-30T01:34:11.801 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:11.776+0000 7f3dfccdd700 10 monclient: get_auth_request con 0x7f3dd8033390 auth_method 0 2023-09-30T01:34:11.801 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:11.780+0000 7f3dfccdd700 1 --2- 172.21.15.88:0/1899070093 >> v2:172.21.15.107:6812/29926 conn(0x7f3dd8033390 0x7f3dd8035840 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f3de800bfd0 tx=0x7f3de8017000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:34:11.801 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:11.784+0000 7f3df4ff9700 1 -- 172.21.15.88:0/1899070093 <== mon.2 v2:172.21.15.88:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f3df0017030 con 0x7f3df81332e0 2023-09-30T01:34:11.801 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:11.784+0000 7f3df4ff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:34:11.801 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:11.784+0000 7f3df4ff9700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:34:11.982 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:11.976+0000 7f3dfdcdf700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T01:34:11.983 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:11.976+0000 7f3dfdcdf700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:34:11.983 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:11.976+0000 7f3dfdcdf700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.88:3301/0 2023-09-30T01:34:11.983 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:11.976+0000 7f3dfdcdf700 1 -- 172.21.15.88:0/1899070093 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f3df8067b60 con 0x7f3df81332e0 2023-09-30T01:34:11.983 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:11.980+0000 7f3df4ff9700 1 -- 172.21.15.88:0/1899070093 <== mon.2 v2:172.21.15.88:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f3df0017030 con 0x7f3df81332e0 2023-09-30T01:34:11.983 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:11.980+0000 7f3df4ff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:34:11.983 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:11.980+0000 7f3df4ff9700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:34:11.984 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-09-30T01:34:11.984 INFO:teuthology.orchestra.run.smithi088.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:34:11.985 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:11.980+0000 7f3dfdcdf700 1 -- 172.21.15.88:0/1899070093 >> v2:172.21.15.107:6812/29926 conn(0x7f3dd8033390 msgr2=0x7f3dd8035840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:34:11.986 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:11.980+0000 7f3dfdcdf700 1 --2- 172.21.15.88:0/1899070093 >> v2:172.21.15.107:6812/29926 conn(0x7f3dd8033390 0x7f3dd8035840 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f3de800bfd0 tx=0x7f3de8017000 comp rx=0 tx=0).stop 2023-09-30T01:34:11.986 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:11.980+0000 7f3dfdcdf700 10 monclient: shutdown 2023-09-30T01:34:11.986 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:11.980+0000 7f3dfdcdf700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:34:11.986 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:11.980+0000 7f3dfdcdf700 1 -- 172.21.15.88:0/1899070093 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f3df81332e0 msgr2=0x7f3df811d7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:34:11.986 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:11.980+0000 7f3dfdcdf700 1 --2- 172.21.15.88:0/1899070093 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f3df81332e0 0x7f3df811d7d0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f3df000c4f0 tx=0x7f3df0007f50 comp rx=0 tx=0).stop 2023-09-30T01:34:11.986 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:11.980+0000 7f3dfdcdf700 1 -- 172.21.15.88:0/1899070093 shutdown_connections 2023-09-30T01:34:11.986 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:11.980+0000 7f3dfdcdf700 1 --2- 172.21.15.88:0/1899070093 >> v2:172.21.15.107:6812/29926 conn(0x7f3dd8033390 0x7f3dd8035840 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:34:11.986 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:11.980+0000 7f3dfdcdf700 1 --2- 172.21.15.88:0/1899070093 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f3df81332e0 0x7f3df811d7d0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:34:11.986 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:11.980+0000 7f3dfdcdf700 1 --2- 172.21.15.88:0/1899070093 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7f3df81311c0 0x7f3df811d290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:34:11.986 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:11.980+0000 7f3dfdcdf700 1 --2- 172.21.15.88:0/1899070093 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3df8097a40 0x7f3df811cd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:34:11.986 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:11.980+0000 7f3dfdcdf700 1 -- 172.21.15.88:0/1899070093 >> 172.21.15.88:0/1899070093 conn(0x7f3df806e5e0 msgr2=0x7f3df8134bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:34:11.986 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:11.980+0000 7f3dfdcdf700 1 -- 172.21.15.88:0/1899070093 shutdown_connections 2023-09-30T01:34:11.987 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:11.980+0000 7f3dfdcdf700 1 -- 172.21.15.88:0/1899070093 wait complete. 2023-09-30T01:34:14.997 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:34:15.109 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:15.104+0000 7f130b55c700 10 monclient: get_monmap_and_config 2023-09-30T01:34:15.109 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:15.104+0000 7f130b55c700 10 monclient: build_initial_monmap 2023-09-30T01:34:15.109 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:15.104+0000 7f130b55c700 10 monclient: monmap: 2023-09-30T01:34:15.109 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2023-09-30T01:34:15.110 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:34:15.110 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:34:15.109698+0000 2023-09-30T01:34:15.110 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:34:15.109698+0000 2023-09-30T01:34:15.110 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2023-09-30T01:34:15.110 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:34:15.110 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a-legacy 2023-09-30T01:34:15.110 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.107:6789/0 mon.noname-f-legacy 2023-09-30T01:34:15.110 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.noname-b 2023-09-30T01:34:15.110 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.noname-g 2023-09-30T01:34:15.110 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.noname-c 2023-09-30T01:34:15.110 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.noname-h 2023-09-30T01:34:15.111 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.noname-d 2023-09-30T01:34:15.111 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.noname-i 2023-09-30T01:34:15.111 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.noname-e 2023-09-30T01:34:15.111 INFO:teuthology.orchestra.run.smithi088.stderr:9: v2:172.21.15.88:3300/0 mon.noname-a 2023-09-30T01:34:15.111 INFO:teuthology.orchestra.run.smithi088.stderr:10: v2:172.21.15.107:3300/0 mon.noname-f 2023-09-30T01:34:15.111 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:34:15.120 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:15.104+0000 7f130b55c700 1 Processor -- start 2023-09-30T01:34:15.120 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:15.104+0000 7f130b55c700 1 -- start start 2023-09-30T01:34:15.120 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:15.104+0000 7f130b55c700 10 monclient: init 2023-09-30T01:34:15.120 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:15.104+0000 7f130b55c700 10 monclient: _reopen_session rank -1 2023-09-30T01:34:15.120 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:15.104+0000 7f130b55c700 10 monclient: _add_conns ranks=[4,7,8,5,10,1,0,2,3,9,6] 2023-09-30T01:34:15.120 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:15.104+0000 7f130b55c700 1 --2- >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f13040a35f0 0x7f1304133330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:34:15.120 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:15.104+0000 7f130b55c700 10 monclient(hunting): picked mon.noname-c con 0x7f13040a35f0 addr [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] 2023-09-30T01:34:15.120 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:15.104+0000 7f130b55c700 1 --2- >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7f1304133870 0x7f1304133c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:34:15.120 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:15.104+0000 7f130b55c700 10 monclient(hunting): picked mon.noname-i con 0x7f1304133870 addr [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] 2023-09-30T01:34:15.120 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:15.104+0000 7f130b55c700 1 --2- >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f13040a46f0 0x7f13040a4b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:34:15.121 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:15.104+0000 7f130b55c700 10 monclient(hunting): picked mon.noname-e con 0x7f13040a46f0 addr [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] 2023-09-30T01:34:15.121 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:15.104+0000 7f130b55c700 10 monclient(hunting): start opening mon connection 2023-09-30T01:34:15.121 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:15.104+0000 7f130b55c700 1 -- --> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] -- mon_getmap magic: 0 v1 -- 0x7f1304097ad0 con 0x7f13040a35f0 2023-09-30T01:34:15.121 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:15.104+0000 7f130b55c700 10 monclient(hunting): start opening mon connection 2023-09-30T01:34:15.121 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:15.104+0000 7f130b55c700 1 -- --> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] -- mon_getmap magic: 0 v1 -- 0x7f13041341c0 con 0x7f1304133870 2023-09-30T01:34:15.121 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:15.104+0000 7f130b55c700 10 monclient(hunting): start opening mon connection 2023-09-30T01:34:15.121 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:15.104+0000 7f130b55c700 1 -- --> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] -- mon_getmap magic: 0 v1 -- 0x7f13040a0940 con 0x7f13040a46f0 2023-09-30T01:34:15.121 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:15.104+0000 7f130b55c700 10 monclient(hunting): _renew_subs 2023-09-30T01:34:15.121 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:15.104+0000 7f130b55c700 10 monclient(hunting): authenticate will time out at 927.147526s 2023-09-30T01:34:15.121 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:15.104+0000 7f1309af9700 1 --2- >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f13040a46f0 0x7f13040a4b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:34:15.121 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:15.104+0000 7f13092f8700 1 --2- >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f13040a35f0 0x7f1304133330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:34:15.121 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:15.104+0000 7f1309af9700 1 --2- >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f13040a46f0 0x7f13040a4b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3304/0 says I am v2:172.21.15.88:50316/0 (socket says 172.21.15.88:50316) 2023-09-30T01:34:15.122 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:15.104+0000 7f1309af9700 1 -- 172.21.15.88:0/2729620644 learned_addr learned my addr 172.21.15.88:0/2729620644 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-09-30T01:34:15.122 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:15.104+0000 7f1309af9700 10 monclient(hunting): get_auth_request con 0x7f13040a46f0 auth_method 0 2023-09-30T01:34:15.122 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:15.104+0000 7f1309af9700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:34:15.122 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:15.104+0000 7f1309af9700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:34:15.122 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:15.104+0000 7f1309af9700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:34:15.122 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:15.104+0000 7f13092f8700 10 monclient(hunting): get_auth_request con 0x7f13040a35f0 auth_method 0 2023-09-30T01:34:15.122 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:15.104+0000 7f13092f8700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:34:15.122 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:15.104+0000 7f13092f8700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:34:15.122 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:15.104+0000 7f13092f8700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:34:15.122 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:15.108+0000 7f1309af9700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:34:15.122 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:15.108+0000 7f1309af9700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:34:15.123 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:15.104+0000 7f1308af7700 1 --2- 172.21.15.88:0/2729620644 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7f1304133870 0x7f1304133c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:34:15.123 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:15.108+0000 7f1309af9700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:34:15.123 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:15.108+0000 7f1308af7700 10 monclient(hunting): get_auth_request con 0x7f1304133870 auth_method 0 2023-09-30T01:34:15.123 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:15.108+0000 7f1308af7700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:34:15.123 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:15.108+0000 7f1308af7700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:34:15.123 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:15.108+0000 7f1308af7700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:34:15.123 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:15.108+0000 7f13092f8700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:34:15.123 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:15.108+0000 7f13092f8700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:34:15.123 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:15.108+0000 7f13092f8700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:34:15.124 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:15.108+0000 7f1308af7700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:34:15.124 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:15.108+0000 7f1308af7700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:34:15.124 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:15.108+0000 7f1308af7700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:34:15.124 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:15.108+0000 7f13092f8700 10 monclient(hunting): handle_auth_done global_id 4486 payload 274 2023-09-30T01:34:15.124 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:15.108+0000 7f13092f8700 1 -- 172.21.15.88:0/2729620644 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f13040a46f0 msgr2=0x7f13040a4b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:34:15.124 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:15.108+0000 7f13092f8700 1 --2- 172.21.15.88:0/2729620644 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f13040a46f0 0x7f13040a4b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:34:15.124 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:15.108+0000 7f13092f8700 1 -- 172.21.15.88:0/2729620644 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7f1304133870 msgr2=0x7f1304133c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:34:15.124 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:15.108+0000 7f13092f8700 1 --2- 172.21.15.88:0/2729620644 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7f1304133870 0x7f1304133c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:34:15.124 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:15.108+0000 7f13092f8700 10 monclient: _finish_hunting 0 2023-09-30T01:34:15.125 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:15.108+0000 7f13092f8700 1 monclient: found mon.noname-c 2023-09-30T01:34:15.125 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:15.108+0000 7f13092f8700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:34:15.125 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:15.108+0000 7f13092f8700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.88:3302/0 2023-09-30T01:34:15.125 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:15.108+0000 7f13092f8700 1 -- 172.21.15.88:0/2729620644 --> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f130405ecb0 con 0x7f13040a35f0 2023-09-30T01:34:15.125 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:15.108+0000 7f13092f8700 1 --2- 172.21.15.88:0/2729620644 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f13040a35f0 0x7f1304133330 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f12f8009940 tx=0x7f12f80091a0 comp rx=0 tx=0).ready entity=mon.4 client_cookie=35a1d93952f4c3a server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:34:15.125 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:15.108+0000 7f12f7fff700 1 -- 172.21.15.88:0/2729620644 <== mon.4 v2:172.21.15.88:3302/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f12f8011ab0 con 0x7f13040a35f0 2023-09-30T01:34:15.125 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:15.108+0000 7f12f7fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:34:15.125 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:15.108+0000 7f12f7fff700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T01:34:15.125 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:15.108+0000 7f12f7fff700 10 monclient: dump: 2023-09-30T01:34:15.125 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:34:15.126 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:34:15.126 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:34:15.126 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:34:15.126 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:34:15.126 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:34:15.126 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:34:15.126 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:34:15.126 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:34:15.126 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:34:15.126 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:34:15.126 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:34:15.126 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:34:15.127 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:34:15.127 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:34:15.127 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:34:15.127 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:15.108+0000 7f12f7fff700 10 monclient: _finish_auth 0 2023-09-30T01:34:15.127 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:15.108+0000 7f12f7fff700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:34:15.127 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:15.108+0000 7f130b55c700 5 monclient: authenticate success, global_id 4486 2023-09-30T01:34:15.127 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:15.108+0000 7f130b55c700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:34:15.127 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:15.108+0000 7f12f7fff700 1 -- 172.21.15.88:0/2729620644 <== mon.4 v2:172.21.15.88:3302/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f12f8011c10 con 0x7f13040a35f0 2023-09-30T01:34:15.127 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:15.108+0000 7f12f7fff700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:34:15.127 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:15.108+0000 7f130b55c700 10 monclient: get_monmap_and_config success 2023-09-30T01:34:15.127 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:15.108+0000 7f130b55c700 4 set_mon_vals no callback set 2023-09-30T01:34:15.127 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:15.108+0000 7f12f7fff700 1 -- 172.21.15.88:0/2729620644 <== mon.4 v2:172.21.15.88:3302/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f12f8011d80 con 0x7f13040a35f0 2023-09-30T01:34:15.127 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:15.108+0000 7f12f7fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:34:15.128 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:15.108+0000 7f12f7fff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T01:34:15.128 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:15.108+0000 7f12f7fff700 10 monclient: dump: 2023-09-30T01:34:15.128 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:34:15.128 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:34:15.128 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:34:15.128 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:34:15.128 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:34:15.128 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:34:15.128 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:34:15.128 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:34:15.128 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:34:15.128 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:34:15.129 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:34:15.129 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:34:15.129 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:34:15.129 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:34:15.129 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:34:15.129 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:34:15.129 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:15.108+0000 7f130b55c700 10 monclient: shutdown 2023-09-30T01:34:15.129 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:15.108+0000 7f130b55c700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:34:15.129 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:15.108+0000 7f130b55c700 1 -- 172.21.15.88:0/2729620644 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f13040a35f0 msgr2=0x7f1304133330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:34:15.129 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:15.108+0000 7f130b55c700 1 --2- 172.21.15.88:0/2729620644 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f13040a35f0 0x7f1304133330 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f12f8009940 tx=0x7f12f80091a0 comp rx=0 tx=0).stop 2023-09-30T01:34:15.129 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:15.108+0000 7f130b55c700 1 -- 172.21.15.88:0/2729620644 shutdown_connections 2023-09-30T01:34:15.129 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:15.108+0000 7f130b55c700 1 --2- 172.21.15.88:0/2729620644 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7f1304133870 0x7f1304133c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:34:15.130 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:15.108+0000 7f130b55c700 1 --2- 172.21.15.88:0/2729620644 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f13040a46f0 0x7f13040a4b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:34:15.130 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:15.108+0000 7f130b55c700 1 --2- 172.21.15.88:0/2729620644 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f13040a35f0 0x7f1304133330 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:34:15.130 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:15.108+0000 7f130b55c700 1 -- 172.21.15.88:0/2729620644 >> 172.21.15.88:0/2729620644 conn(0x7f130406e5e0 msgr2=0x7f130406e9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:34:15.130 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:15.108+0000 7f130b55c700 1 -- 172.21.15.88:0/2729620644 shutdown_connections 2023-09-30T01:34:15.130 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:15.108+0000 7f130b55c700 1 -- 172.21.15.88:0/2729620644 wait complete. 2023-09-30T01:34:15.130 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:15.108+0000 7f130b55c700 10 monclient: build_initial_monmap 2023-09-30T01:34:15.130 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:15.108+0000 7f130b55c700 10 monclient: monmap: 2023-09-30T01:34:15.130 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2023-09-30T01:34:15.130 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:34:15.130 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 0.000000 2023-09-30T01:34:15.131 INFO:teuthology.orchestra.run.smithi088.stderr:created 0.000000 2023-09-30T01:34:15.131 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2023-09-30T01:34:15.131 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 1 2023-09-30T01:34:15.131 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.noname-a 2023-09-30T01:34:15.131 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.noname-f 2023-09-30T01:34:15.131 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.noname-b 2023-09-30T01:34:15.131 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.noname-g 2023-09-30T01:34:15.131 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.noname-c 2023-09-30T01:34:15.131 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.noname-h 2023-09-30T01:34:15.131 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.noname-d 2023-09-30T01:34:15.131 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.noname-i 2023-09-30T01:34:15.132 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.noname-e 2023-09-30T01:34:15.132 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:34:15.132 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:15.108+0000 7f130b55c700 1 Processor -- start 2023-09-30T01:34:15.132 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:15.108+0000 7f130b55c700 1 -- start start 2023-09-30T01:34:15.132 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:15.108+0000 7f130b55c700 10 monclient: init 2023-09-30T01:34:15.132 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:15.108+0000 7f130b55c700 10 monclient: _reopen_session rank -1 2023-09-30T01:34:15.132 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:15.108+0000 7f130b55c700 10 monclient: _add_conns ranks=[2,0,3,6,7,1,5,4,8] 2023-09-30T01:34:15.132 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:15.108+0000 7f130b55c700 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f13040a35f0 0x7f13041318b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:34:15.133 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:15.108+0000 7f130b55c700 10 monclient(hunting): picked mon.noname-b con 0x7f13040a35f0 addr [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] 2023-09-30T01:34:15.133 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:15.108+0000 7f130b55c700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f13040a46f0 0x7f130411d290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:34:15.133 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:15.108+0000 7f130b55c700 10 monclient(hunting): picked mon.noname-a con 0x7f13040a46f0 addr [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] 2023-09-30T01:34:15.133 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:15.108+0000 7f130b55c700 1 --2- >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7f1304133870 0x7f130411d7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:34:15.133 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:15.108+0000 7f130b55c700 10 monclient(hunting): picked mon.noname-g con 0x7f1304133870 addr [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] 2023-09-30T01:34:15.133 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:15.108+0000 7f130b55c700 10 monclient(hunting): start opening mon connection 2023-09-30T01:34:15.133 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:15.108+0000 7f130b55c700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13040a0b70 con 0x7f13040a46f0 2023-09-30T01:34:15.133 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:15.108+0000 7f130b55c700 10 monclient(hunting): start opening mon connection 2023-09-30T01:34:15.133 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:15.108+0000 7f130b55c700 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1304097d90 con 0x7f13040a35f0 2023-09-30T01:34:15.133 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:15.108+0000 7f130b55c700 10 monclient(hunting): start opening mon connection 2023-09-30T01:34:15.133 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:15.108+0000 7f130b55c700 1 -- --> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1304120790 con 0x7f1304133870 2023-09-30T01:34:15.133 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:15.108+0000 7f130b55c700 10 monclient(hunting): _renew_subs 2023-09-30T01:34:15.134 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:15.108+0000 7f130b55c700 10 monclient(hunting): authenticate will time out at 927.151195s 2023-09-30T01:34:15.134 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:15.108+0000 7f13092f8700 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f13040a35f0 0x7f13041318b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:34:15.134 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:15.108+0000 7f13092f8700 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f13040a35f0 0x7f13041318b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3301/0 says I am v2:172.21.15.88:47414/0 (socket says 172.21.15.88:47414) 2023-09-30T01:34:15.134 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:15.108+0000 7f13092f8700 1 -- 172.21.15.88:0/1053346788 learned_addr learned my addr 172.21.15.88:0/1053346788 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-09-30T01:34:15.134 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:15.108+0000 7f1308af7700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f13040a46f0 0x7f130411d290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:34:15.134 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:15.108+0000 7f13092f8700 10 monclient(hunting): get_auth_request con 0x7f13040a35f0 auth_method 0 2023-09-30T01:34:15.134 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:15.108+0000 7f13092f8700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:34:15.135 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:15.108+0000 7f13092f8700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:34:15.135 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:15.108+0000 7f13092f8700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:34:15.135 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:15.108+0000 7f1308af7700 10 monclient(hunting): get_auth_request con 0x7f13040a46f0 auth_method 0 2023-09-30T01:34:15.135 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:15.108+0000 7f1308af7700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:34:15.135 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:15.108+0000 7f1308af7700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:34:15.135 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:15.108+0000 7f1308af7700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:34:15.135 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:15.108+0000 7f1309af9700 1 --2- 172.21.15.88:0/1053346788 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7f1304133870 0x7f130411d7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:34:15.135 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:15.108+0000 7f13092f8700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:34:15.136 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:15.108+0000 7f13092f8700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:34:15.136 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:15.108+0000 7f13092f8700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:34:15.136 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:15.108+0000 7f1309af9700 10 monclient(hunting): get_auth_request con 0x7f1304133870 auth_method 0 2023-09-30T01:34:15.136 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:15.108+0000 7f1309af9700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:34:15.136 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:15.108+0000 7f1309af9700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:34:15.136 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:15.108+0000 7f1309af9700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:34:15.136 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:15.108+0000 7f1308af7700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:34:15.136 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:15.108+0000 7f1308af7700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:34:15.137 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:15.108+0000 7f1308af7700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:34:15.137 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:15.108+0000 7f1309af9700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:34:15.137 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:15.108+0000 7f1309af9700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:34:15.137 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:15.108+0000 7f1309af9700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:34:15.137 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:15.112+0000 7f13092f8700 10 monclient(hunting): handle_auth_done global_id 4448 payload 824 2023-09-30T01:34:15.137 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:15.112+0000 7f13092f8700 1 -- 172.21.15.88:0/1053346788 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7f1304133870 msgr2=0x7f130411d7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:34:15.137 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:15.112+0000 7f13092f8700 1 --2- 172.21.15.88:0/1053346788 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7f1304133870 0x7f130411d7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:34:15.137 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:15.112+0000 7f13092f8700 1 -- 172.21.15.88:0/1053346788 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f13040a46f0 msgr2=0x7f130411d290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:34:15.138 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:15.112+0000 7f13092f8700 1 --2- 172.21.15.88:0/1053346788 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f13040a46f0 0x7f130411d290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:34:15.138 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:15.112+0000 7f13092f8700 10 monclient: _finish_hunting 0 2023-09-30T01:34:15.138 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:15.112+0000 7f13092f8700 1 monclient: found mon.noname-b 2023-09-30T01:34:15.138 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:15.112+0000 7f13092f8700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:34:15.138 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:15.112+0000 7f13092f8700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.88:3301/0 2023-09-30T01:34:15.138 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:15.112+0000 7f13092f8700 1 -- 172.21.15.88:0/1053346788 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12f8009420 con 0x7f13040a35f0 2023-09-30T01:34:15.138 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:15.112+0000 7f1308af7700 1 --2- 172.21.15.88:0/1053346788 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f13040a46f0 0x7f130411d290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:34:15.138 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:15.112+0000 7f13092f8700 1 --2- 172.21.15.88:0/1053346788 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f13040a35f0 0x7f13041318b0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f13040a7800 tx=0x7f12f8017000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:34:15.139 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:15.112+0000 7f12f57fa700 1 -- 172.21.15.88:0/1053346788 <== mon.2 v2:172.21.15.88:3301/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f12f801f070 con 0x7f13040a35f0 2023-09-30T01:34:15.139 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:15.112+0000 7f12f57fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:34:15.139 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:15.112+0000 7f12f57fa700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T01:34:15.139 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:15.112+0000 7f12f57fa700 10 monclient: dump: 2023-09-30T01:34:15.139 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:34:15.139 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:34:15.139 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:34:15.140 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:34:15.140 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:34:15.140 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:34:15.140 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:34:15.140 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:34:15.140 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:34:15.140 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:34:15.140 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:34:15.141 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:34:15.141 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:34:15.141 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:34:15.141 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:34:15.141 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:34:15.141 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:15.112+0000 7f12f57fa700 10 monclient: _finish_auth 0 2023-09-30T01:34:15.141 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:15.112+0000 7f12f57fa700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:34:15.141 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:15.112+0000 7f130b55c700 5 monclient: authenticate success, global_id 4448 2023-09-30T01:34:15.141 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:15.112+0000 7f12f57fa700 1 -- 172.21.15.88:0/1053346788 <== mon.2 v2:172.21.15.88:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f12f8006a90 con 0x7f13040a35f0 2023-09-30T01:34:15.142 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:15.112+0000 7f12f57fa700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:34:15.142 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:15.112+0000 7f12f57fa700 1 -- 172.21.15.88:0/1053346788 <== mon.2 v2:172.21.15.88:3301/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f12f801ad50 con 0x7f13040a35f0 2023-09-30T01:34:15.142 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:15.112+0000 7f130b55c700 10 monclient: _renew_subs 2023-09-30T01:34:15.142 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:15.112+0000 7f130b55c700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.88:3301/0 2023-09-30T01:34:15.142 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:15.112+0000 7f130b55c700 1 -- 172.21.15.88:0/1053346788 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f13041359f0 con 0x7f13040a35f0 2023-09-30T01:34:15.142 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:15.112+0000 7f12f57fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:34:15.142 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:15.112+0000 7f12f57fa700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T01:34:15.143 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:15.112+0000 7f12f77fe700 4 set_mon_vals no callback set 2023-09-30T01:34:15.143 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:15.112+0000 7f12f57fa700 10 monclient: dump: 2023-09-30T01:34:15.143 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:34:15.143 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:34:15.143 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:34:15.143 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:34:15.143 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:34:15.143 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:34:15.144 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:34:15.144 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:34:15.144 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:34:15.144 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:34:15.144 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:34:15.144 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:34:15.144 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:34:15.144 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:34:15.145 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:34:15.145 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:34:15.145 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:15.112+0000 7f130b55c700 10 monclient: _renew_subs 2023-09-30T01:34:15.145 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:15.112+0000 7f130b55c700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.88:3301/0 2023-09-30T01:34:15.145 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:15.112+0000 7f130b55c700 1 -- 172.21.15.88:0/1053346788 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1304064b80 con 0x7f13040a35f0 2023-09-30T01:34:15.145 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:15.112+0000 7f130b55c700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:34:15.145 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:15.112+0000 7f12f57fa700 1 -- 172.21.15.88:0/1053346788 <== mon.2 v2:172.21.15.88:3301/0 4 ==== mgrmap(e 5) v1 ==== 40144+0+0 (secure 0 0 0) 0x7f12f8023070 con 0x7f13040a35f0 2023-09-30T01:34:15.145 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:15.112+0000 7f12f57fa700 1 --2- 172.21.15.88:0/1053346788 >> v2:172.21.15.107:6812/29926 conn(0x7f12e8033360 0x7f12e8035810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:34:15.146 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:15.112+0000 7f12f57fa700 1 -- 172.21.15.88:0/1053346788 <== mon.2 v2:172.21.15.88:3301/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 3502+0+0 (secure 0 0 0) 0x7f12f8044150 con 0x7f13040a35f0 2023-09-30T01:34:15.146 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:15.112+0000 7f130b55c700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:34:15.146 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:15.112+0000 7f130b55c700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.88:3301/0 2023-09-30T01:34:15.146 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:15.112+0000 7f130b55c700 1 -- 172.21.15.88:0/1053346788 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f13041210d0 con 0x7f13040a35f0 2023-09-30T01:34:15.146 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:15.112+0000 7f1308af7700 1 --2- 172.21.15.88:0/1053346788 >> v2:172.21.15.107:6812/29926 conn(0x7f12e8033360 0x7f12e8035810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:34:15.146 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:15.112+0000 7f1308af7700 10 monclient: get_auth_request con 0x7f12e8033360 auth_method 0 2023-09-30T01:34:15.146 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:15.116+0000 7f1308af7700 1 --2- 172.21.15.88:0/1053346788 >> v2:172.21.15.107:6812/29926 conn(0x7f12e8033360 0x7f12e8035810 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f13040a35c0 tx=0x7f13000091a0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:34:15.147 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:15.120+0000 7f12f57fa700 1 -- 172.21.15.88:0/1053346788 <== mon.2 v2:172.21.15.88:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f12f802a080 con 0x7f13040a35f0 2023-09-30T01:34:15.147 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:15.120+0000 7f12f57fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:34:15.147 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:15.120+0000 7f12f57fa700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:34:15.320 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:15.316+0000 7f130b55c700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T01:34:15.320 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:15.316+0000 7f130b55c700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:34:15.320 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:15.316+0000 7f130b55c700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.88:3301/0 2023-09-30T01:34:15.320 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:15.316+0000 7f130b55c700 1 -- 172.21.15.88:0/1053346788 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f13040710d0 con 0x7f13040a35f0 2023-09-30T01:34:15.321 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:15.316+0000 7f12f57fa700 1 -- 172.21.15.88:0/1053346788 <== mon.2 v2:172.21.15.88:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f12f8015020 con 0x7f13040a35f0 2023-09-30T01:34:15.321 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:15.316+0000 7f12f57fa700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:34:15.321 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:15.316+0000 7f12f57fa700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:34:15.321 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-09-30T01:34:15.322 INFO:teuthology.orchestra.run.smithi088.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:34:15.325 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:15.320+0000 7f130b55c700 1 -- 172.21.15.88:0/1053346788 >> v2:172.21.15.107:6812/29926 conn(0x7f12e8033360 msgr2=0x7f12e8035810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:34:15.325 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:15.320+0000 7f130b55c700 1 --2- 172.21.15.88:0/1053346788 >> v2:172.21.15.107:6812/29926 conn(0x7f12e8033360 0x7f12e8035810 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f13040a35c0 tx=0x7f13000091a0 comp rx=0 tx=0).stop 2023-09-30T01:34:15.326 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:15.320+0000 7f130b55c700 10 monclient: shutdown 2023-09-30T01:34:15.326 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:15.320+0000 7f130b55c700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:34:15.326 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:15.320+0000 7f130b55c700 1 -- 172.21.15.88:0/1053346788 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f13040a35f0 msgr2=0x7f13041318b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:34:15.326 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:15.320+0000 7f130b55c700 1 --2- 172.21.15.88:0/1053346788 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f13040a35f0 0x7f13041318b0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f13040a7800 tx=0x7f12f8017000 comp rx=0 tx=0).stop 2023-09-30T01:34:15.326 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:15.320+0000 7f130b55c700 1 -- 172.21.15.88:0/1053346788 shutdown_connections 2023-09-30T01:34:15.326 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:15.320+0000 7f130b55c700 1 --2- 172.21.15.88:0/1053346788 >> v2:172.21.15.107:6812/29926 conn(0x7f12e8033360 0x7f12e8035810 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:34:15.326 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:15.320+0000 7f130b55c700 1 --2- 172.21.15.88:0/1053346788 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7f1304133870 0x7f130411d7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:34:15.326 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:15.320+0000 7f130b55c700 1 --2- 172.21.15.88:0/1053346788 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f13040a46f0 0x7f130411d290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:34:15.326 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:15.320+0000 7f130b55c700 1 --2- 172.21.15.88:0/1053346788 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f13040a35f0 0x7f13041318b0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:34:15.326 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:15.320+0000 7f130b55c700 1 -- 172.21.15.88:0/1053346788 >> 172.21.15.88:0/1053346788 conn(0x7f130406e5e0 msgr2=0x7f13040a1150 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:34:15.326 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:15.320+0000 7f130b55c700 1 -- 172.21.15.88:0/1053346788 shutdown_connections 2023-09-30T01:34:15.327 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:15.320+0000 7f130b55c700 1 -- 172.21.15.88:0/1053346788 wait complete. 2023-09-30T01:34:18.342 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:34:18.453 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:18.448+0000 7ff623b43700 10 monclient: get_monmap_and_config 2023-09-30T01:34:18.453 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:18.448+0000 7ff623b43700 10 monclient: build_initial_monmap 2023-09-30T01:34:18.453 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:18.448+0000 7ff623b43700 10 monclient: monmap: 2023-09-30T01:34:18.453 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2023-09-30T01:34:18.453 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:34:18.454 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:34:18.453752+0000 2023-09-30T01:34:18.454 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:34:18.453752+0000 2023-09-30T01:34:18.454 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2023-09-30T01:34:18.454 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:34:18.454 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a-legacy 2023-09-30T01:34:18.454 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.107:6789/0 mon.noname-f-legacy 2023-09-30T01:34:18.454 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.noname-b 2023-09-30T01:34:18.454 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.noname-g 2023-09-30T01:34:18.454 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.noname-c 2023-09-30T01:34:18.454 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.noname-h 2023-09-30T01:34:18.454 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.noname-d 2023-09-30T01:34:18.455 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.noname-i 2023-09-30T01:34:18.455 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.noname-e 2023-09-30T01:34:18.455 INFO:teuthology.orchestra.run.smithi088.stderr:9: v2:172.21.15.88:3300/0 mon.noname-a 2023-09-30T01:34:18.455 INFO:teuthology.orchestra.run.smithi088.stderr:10: v2:172.21.15.107:3300/0 mon.noname-f 2023-09-30T01:34:18.455 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:34:18.460 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:18.448+0000 7ff623b43700 1 Processor -- start 2023-09-30T01:34:18.460 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:18.448+0000 7ff623b43700 1 -- start start 2023-09-30T01:34:18.461 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:18.448+0000 7ff623b43700 10 monclient: init 2023-09-30T01:34:18.461 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:18.452+0000 7ff623b43700 10 monclient: _reopen_session rank -1 2023-09-30T01:34:18.461 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:18.452+0000 7ff623b43700 10 monclient: _add_conns ranks=[3,0,4,5,1,7,8,6,9,10,2] 2023-09-30T01:34:18.461 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:18.452+0000 7ff623b43700 1 --2- >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7ff61c06dbb0 0x7ff61c06d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:34:18.461 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:18.452+0000 7ff623b43700 10 monclient(hunting): picked mon.noname-g con 0x7ff61c06dbb0 addr [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] 2023-09-30T01:34:18.461 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:18.452+0000 7ff623b43700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ff61c06df80 addr v1:172.21.15.88:6789/0 2023-09-30T01:34:18.461 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:18.452+0000 7ff623b43700 1 --2- >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7ff61c06e350 0x7ff61c06ad80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:34:18.461 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:18.452+0000 7ff623b43700 10 monclient(hunting): picked mon.noname-c con 0x7ff61c06e350 addr [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] 2023-09-30T01:34:18.461 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:18.452+0000 7ff623b43700 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff61c05ecb0 con 0x7ff61c06df80 2023-09-30T01:34:18.462 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:18.452+0000 7ff623b43700 10 monclient(hunting): start opening mon connection 2023-09-30T01:34:18.462 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:18.452+0000 7ff623b43700 1 -- --> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff61c06c8f0 con 0x7ff61c06dbb0 2023-09-30T01:34:18.462 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:18.452+0000 7ff623b43700 10 monclient(hunting): start opening mon connection 2023-09-30T01:34:18.462 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:18.452+0000 7ff623b43700 1 -- --> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] -- mon_getmap magic: 0 v1 -- 0x7ff61c06c610 con 0x7ff61c06e350 2023-09-30T01:34:18.462 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:18.452+0000 7ff623b43700 10 monclient(hunting): _renew_subs 2023-09-30T01:34:18.462 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:18.452+0000 7ff623b43700 10 monclient(hunting): authenticate will time out at 930.492474s 2023-09-30T01:34:18.462 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:18.452+0000 7ff6210de700 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7ff61c06df80 0x7ff61c06a670 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:51932/0 (socket says 172.21.15.88:51932) 2023-09-30T01:34:18.462 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:18.452+0000 7ff6210de700 1 -- 172.21.15.88:0/631797461 learned_addr learned my addr 172.21.15.88:0/631797461 (peer_addr_for_me v1:172.21.15.88:0/0) 2023-09-30T01:34:18.462 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:18.452+0000 7ff6220e0700 1 --2- 172.21.15.88:0/631797461 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7ff61c06e350 0x7ff61c06ad80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:34:18.462 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:18.452+0000 7ff6220e0700 10 monclient(hunting): get_auth_request con 0x7ff61c06e350 auth_method 0 2023-09-30T01:34:18.462 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:18.452+0000 7ff6220e0700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:34:18.463 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:18.452+0000 7ff6220e0700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:34:18.463 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:18.452+0000 7ff6220e0700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:34:18.463 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:18.452+0000 7ff6218df700 1 --2- 172.21.15.88:0/631797461 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7ff61c06dbb0 0x7ff61c06d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:34:18.463 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:18.452+0000 7ff6220e0700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:34:18.463 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:18.452+0000 7ff6220e0700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:34:18.463 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:18.452+0000 7ff6220e0700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:34:18.463 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:18.452+0000 7ff6218df700 10 monclient(hunting): get_auth_request con 0x7ff61c06dbb0 auth_method 0 2023-09-30T01:34:18.463 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:18.452+0000 7ff6218df700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:34:18.463 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:18.452+0000 7ff6218df700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:34:18.463 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:18.452+0000 7ff6218df700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:34:18.463 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:18.452+0000 7ff6208dd700 1 -- 172.21.15.88:0/631797461 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3173217680 0 0) 0x7ff61c05ecb0 con 0x7ff61c06df80 2023-09-30T01:34:18.463 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:18.452+0000 7ff6208dd700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:34:18.464 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:18.452+0000 7ff6208dd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:34:18.464 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:18.452+0000 7ff6208dd700 10 monclient(hunting): my global_id is 4608 2023-09-30T01:34:18.464 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:18.452+0000 7ff6208dd700 1 -- 172.21.15.88:0/631797461 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff608003570 con 0x7ff61c06df80 2023-09-30T01:34:18.464 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:18.452+0000 7ff6218df700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:34:18.464 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:18.452+0000 7ff6218df700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:34:18.464 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:18.452+0000 7ff6218df700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:34:18.464 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:18.452+0000 7ff6220e0700 10 monclient(hunting): handle_auth_done global_id 4495 payload 274 2023-09-30T01:34:18.464 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:18.452+0000 7ff6220e0700 1 -- 172.21.15.88:0/631797461 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7ff61c06dbb0 msgr2=0x7ff61c06d5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:34:18.464 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:18.452+0000 7ff6220e0700 1 --2- 172.21.15.88:0/631797461 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7ff61c06dbb0 0x7ff61c06d5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:34:18.464 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:18.452+0000 7ff6220e0700 1 -- 172.21.15.88:0/631797461 >> v1:172.21.15.88:6789/0 conn(0x7ff61c06df80 legacy=0x7ff61c06a670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:34:18.464 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:18.452+0000 7ff6220e0700 10 monclient: _finish_hunting 0 2023-09-30T01:34:18.464 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:18.452+0000 7ff6220e0700 1 monclient: found mon.noname-c 2023-09-30T01:34:18.465 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:18.452+0000 7ff6220e0700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:34:18.465 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:18.452+0000 7ff6220e0700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.88:3302/0 2023-09-30T01:34:18.465 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:18.452+0000 7ff6220e0700 1 -- 172.21.15.88:0/631797461 --> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff61c06e7b0 con 0x7ff61c06e350 2023-09-30T01:34:18.465 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:18.452+0000 7ff6220e0700 1 --2- 172.21.15.88:0/631797461 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7ff61c06e350 0x7ff61c06ad80 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7ff614009e90 tx=0x7ff614009780 comp rx=0 tx=0).ready entity=mon.4 client_cookie=38d8ca755e456413 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:34:18.465 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:18.452+0000 7ff6208dd700 1 -- 172.21.15.88:0/631797461 <== mon.4 v2:172.21.15.88:3302/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7ff61400e070 con 0x7ff61c06e350 2023-09-30T01:34:18.465 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:18.452+0000 7ff6208dd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:34:18.465 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:18.452+0000 7ff6208dd700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T01:34:18.465 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:18.452+0000 7ff6208dd700 10 monclient: dump: 2023-09-30T01:34:18.465 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:34:18.465 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:34:18.465 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:34:18.465 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:34:18.466 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:34:18.466 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:34:18.466 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:34:18.466 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:34:18.466 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:34:18.466 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:34:18.466 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:34:18.466 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:34:18.466 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:34:18.466 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:34:18.466 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:34:18.467 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:34:18.467 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:18.452+0000 7ff6208dd700 10 monclient: _finish_auth 0 2023-09-30T01:34:18.467 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:18.452+0000 7ff6208dd700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:34:18.467 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:18.452+0000 7ff623b43700 5 monclient: authenticate success, global_id 4495 2023-09-30T01:34:18.467 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:18.452+0000 7ff6208dd700 1 -- 172.21.15.88:0/631797461 <== mon.4 v2:172.21.15.88:3302/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff614012430 con 0x7ff61c06e350 2023-09-30T01:34:18.467 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:18.452+0000 7ff623b43700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:34:18.467 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:18.452+0000 7ff6208dd700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:34:18.467 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:18.452+0000 7ff623b43700 10 monclient: get_monmap_and_config success 2023-09-30T01:34:18.467 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:18.452+0000 7ff6208dd700 1 -- 172.21.15.88:0/631797461 <== mon.4 v2:172.21.15.88:3302/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7ff614012a20 con 0x7ff61c06e350 2023-09-30T01:34:18.467 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:18.452+0000 7ff623b43700 4 set_mon_vals no callback set 2023-09-30T01:34:18.467 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:18.452+0000 7ff6208dd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:34:18.467 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:18.452+0000 7ff6208dd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T01:34:18.468 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:18.452+0000 7ff6208dd700 10 monclient: dump: 2023-09-30T01:34:18.468 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:34:18.468 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:34:18.468 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:34:18.468 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:34:18.468 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:34:18.468 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:34:18.468 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:34:18.468 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:34:18.468 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:34:18.468 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:34:18.468 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:34:18.469 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:34:18.469 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:34:18.469 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:34:18.469 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:34:18.469 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:34:18.469 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:18.452+0000 7ff623b43700 10 monclient: shutdown 2023-09-30T01:34:18.469 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:18.452+0000 7ff623b43700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:34:18.469 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:18.452+0000 7ff623b43700 1 -- 172.21.15.88:0/631797461 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7ff61c06e350 msgr2=0x7ff61c06ad80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:34:18.469 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:18.452+0000 7ff623b43700 1 --2- 172.21.15.88:0/631797461 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7ff61c06e350 0x7ff61c06ad80 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7ff614009e90 tx=0x7ff614009780 comp rx=0 tx=0).stop 2023-09-30T01:34:18.469 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:18.452+0000 7ff623b43700 1 -- 172.21.15.88:0/631797461 shutdown_connections 2023-09-30T01:34:18.469 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:18.452+0000 7ff623b43700 1 --2- 172.21.15.88:0/631797461 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7ff61c06e350 0x7ff61c06ad80 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:34:18.469 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:18.452+0000 7ff623b43700 1 --2- 172.21.15.88:0/631797461 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7ff61c06dbb0 0x7ff61c06d5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:34:18.470 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:18.452+0000 7ff623b43700 1 -- 172.21.15.88:0/631797461 >> 172.21.15.88:0/631797461 conn(0x7ff61c0a01d0 msgr2=0x7ff61c062c70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:34:18.470 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:18.452+0000 7ff623b43700 1 -- 172.21.15.88:0/631797461 shutdown_connections 2023-09-30T01:34:18.470 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:18.452+0000 7ff623b43700 1 -- 172.21.15.88:0/631797461 wait complete. 2023-09-30T01:34:18.470 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:18.456+0000 7ff623b43700 10 monclient: build_initial_monmap 2023-09-30T01:34:18.470 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:18.456+0000 7ff623b43700 10 monclient: monmap: 2023-09-30T01:34:18.470 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2023-09-30T01:34:18.470 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:34:18.470 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 0.000000 2023-09-30T01:34:18.470 INFO:teuthology.orchestra.run.smithi088.stderr:created 0.000000 2023-09-30T01:34:18.470 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2023-09-30T01:34:18.470 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 1 2023-09-30T01:34:18.470 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.noname-a 2023-09-30T01:34:18.471 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.noname-f 2023-09-30T01:34:18.471 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.noname-b 2023-09-30T01:34:18.471 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.noname-g 2023-09-30T01:34:18.471 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.noname-c 2023-09-30T01:34:18.471 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.noname-h 2023-09-30T01:34:18.471 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.noname-d 2023-09-30T01:34:18.471 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.noname-i 2023-09-30T01:34:18.471 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.noname-e 2023-09-30T01:34:18.471 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:34:18.471 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:18.456+0000 7ff623b43700 1 Processor -- start 2023-09-30T01:34:18.471 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:18.456+0000 7ff623b43700 1 -- start start 2023-09-30T01:34:18.471 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:18.456+0000 7ff623b43700 10 monclient: init 2023-09-30T01:34:18.475 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:18.456+0000 7ff623b43700 10 monclient: _reopen_session rank -1 2023-09-30T01:34:18.475 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:18.456+0000 7ff623b43700 10 monclient: _add_conns ranks=[8,7,0,3,2,5,6,1,4] 2023-09-30T01:34:18.475 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:18.456+0000 7ff623b43700 1 --2- >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7ff61c06dbb0 0x7ff61c140c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:34:18.476 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:18.456+0000 7ff623b43700 10 monclient(hunting): picked mon.noname-e con 0x7ff61c06dbb0 addr [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] 2023-09-30T01:34:18.476 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:18.456+0000 7ff623b43700 1 --2- >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7ff61c06df80 0x7ff61c13c2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:34:18.476 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:18.456+0000 7ff623b43700 10 monclient(hunting): picked mon.noname-i con 0x7ff61c06df80 addr [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] 2023-09-30T01:34:18.476 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:18.456+0000 7ff623b43700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff61c06e350 0x7ff61c13cc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:34:18.476 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:18.456+0000 7ff623b43700 10 monclient(hunting): picked mon.noname-a con 0x7ff61c06e350 addr [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] 2023-09-30T01:34:18.476 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:18.456+0000 7ff623b43700 10 monclient(hunting): start opening mon connection 2023-09-30T01:34:18.476 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:18.456+0000 7ff623b43700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff61c121930 con 0x7ff61c06e350 2023-09-30T01:34:18.476 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:18.456+0000 7ff623b43700 10 monclient(hunting): start opening mon connection 2023-09-30T01:34:18.476 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:18.456+0000 7ff623b43700 1 -- --> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] -- mon_getmap magic: 0 v1 -- 0x7ff61c121650 con 0x7ff61c06df80 2023-09-30T01:34:18.476 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:18.456+0000 7ff623b43700 10 monclient(hunting): start opening mon connection 2023-09-30T01:34:18.476 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:18.456+0000 7ff623b43700 1 -- --> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] -- mon_getmap magic: 0 v1 -- 0x7ff61c121370 con 0x7ff61c06dbb0 2023-09-30T01:34:18.476 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:18.456+0000 7ff623b43700 10 monclient(hunting): _renew_subs 2023-09-30T01:34:18.477 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:18.456+0000 7ff623b43700 10 monclient(hunting): authenticate will time out at 930.498440s 2023-09-30T01:34:18.477 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:18.456+0000 7ff6218df700 1 --2- >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7ff61c06dbb0 0x7ff61c140c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:34:18.477 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:18.456+0000 7ff6220e0700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff61c06e350 0x7ff61c13cc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:34:18.477 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:18.456+0000 7ff6218df700 1 --2- >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7ff61c06dbb0 0x7ff61c140c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3304/0 says I am v2:172.21.15.88:50322/0 (socket says 172.21.15.88:50322) 2023-09-30T01:34:18.477 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:18.456+0000 7ff6218df700 1 -- 172.21.15.88:0/602285511 learned_addr learned my addr 172.21.15.88:0/602285511 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-09-30T01:34:18.477 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:18.456+0000 7ff6218df700 10 monclient(hunting): get_auth_request con 0x7ff61c06dbb0 auth_method 0 2023-09-30T01:34:18.477 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:18.456+0000 7ff6218df700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:34:18.477 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:18.456+0000 7ff6218df700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:34:18.477 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:18.456+0000 7ff6218df700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:34:18.477 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:18.456+0000 7ff6220e0700 10 monclient(hunting): get_auth_request con 0x7ff61c06e350 auth_method 0 2023-09-30T01:34:18.477 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:18.456+0000 7ff6220e0700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:34:18.478 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:18.456+0000 7ff6220e0700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:34:18.478 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:18.456+0000 7ff6220e0700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:34:18.478 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:18.456+0000 7ff6220e0700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:34:18.478 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:18.456+0000 7ff6220e0700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:34:18.478 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:18.456+0000 7ff6220e0700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:34:18.478 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:18.456+0000 7ff6218df700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:34:18.478 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:18.456+0000 7ff6218df700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:34:18.478 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:18.456+0000 7ff6218df700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:34:18.478 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:18.456+0000 7ff6210de700 1 --2- 172.21.15.88:0/602285511 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7ff61c06df80 0x7ff61c13c2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:34:18.478 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:18.456+0000 7ff6210de700 10 monclient(hunting): get_auth_request con 0x7ff61c06df80 auth_method 0 2023-09-30T01:34:18.478 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:18.456+0000 7ff6210de700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:34:18.478 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:18.456+0000 7ff6210de700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:34:18.479 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:18.456+0000 7ff6210de700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:34:18.479 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:18.456+0000 7ff6210de700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:34:18.479 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:18.456+0000 7ff6210de700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:34:18.479 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:18.456+0000 7ff6210de700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:34:18.479 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:18.456+0000 7ff6218df700 10 monclient(hunting): handle_auth_done global_id 4535 payload 824 2023-09-30T01:34:18.479 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:18.456+0000 7ff6218df700 1 -- 172.21.15.88:0/602285511 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7ff61c06df80 msgr2=0x7ff61c13c2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:34:18.479 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:18.456+0000 7ff6218df700 1 --2- 172.21.15.88:0/602285511 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7ff61c06df80 0x7ff61c13c2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:34:18.479 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:18.456+0000 7ff6218df700 1 -- 172.21.15.88:0/602285511 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff61c06e350 msgr2=0x7ff61c13cc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:34:18.479 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:18.456+0000 7ff6218df700 1 --2- 172.21.15.88:0/602285511 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff61c06e350 0x7ff61c13cc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:34:18.479 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:18.456+0000 7ff6218df700 10 monclient: _finish_hunting 0 2023-09-30T01:34:18.479 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:18.456+0000 7ff6218df700 1 monclient: found mon.noname-e 2023-09-30T01:34:18.479 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:18.456+0000 7ff6218df700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:34:18.480 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:18.456+0000 7ff6218df700 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.88:3304/0 2023-09-30T01:34:18.480 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:18.456+0000 7ff6218df700 1 -- 172.21.15.88:0/602285511 --> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff614009a20 con 0x7ff61c06dbb0 2023-09-30T01:34:18.480 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:18.456+0000 7ff6220e0700 1 --2- 172.21.15.88:0/602285511 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff61c06e350 0x7ff61c13cc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:34:18.480 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:18.456+0000 7ff6218df700 1 --2- 172.21.15.88:0/602285511 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7ff61c06dbb0 0x7ff61c140c60 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7ff61800ec80 tx=0x7ff6180091a0 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:34:18.480 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:18.460+0000 7ff611ffb700 1 -- 172.21.15.88:0/602285511 <== mon.8 v2:172.21.15.88:3304/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7ff618009690 con 0x7ff61c06dbb0 2023-09-30T01:34:18.480 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:18.460+0000 7ff611ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:34:18.480 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:18.460+0000 7ff611ffb700 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2023-09-30T01:34:18.480 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:18.460+0000 7ff611ffb700 10 monclient: dump: 2023-09-30T01:34:18.480 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:34:18.480 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:34:18.480 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:34:18.480 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:34:18.481 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:34:18.481 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:34:18.481 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:34:18.481 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:34:18.481 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:34:18.481 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:34:18.481 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:34:18.481 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:34:18.481 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:34:18.481 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:34:18.481 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:34:18.481 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:34:18.482 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:18.460+0000 7ff611ffb700 10 monclient: _finish_auth 0 2023-09-30T01:34:18.482 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:18.460+0000 7ff611ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:34:18.482 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:18.460+0000 7ff623b43700 5 monclient: authenticate success, global_id 4535 2023-09-30T01:34:18.482 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:18.460+0000 7ff611ffb700 1 -- 172.21.15.88:0/602285511 <== mon.8 v2:172.21.15.88:3304/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff61800ccd0 con 0x7ff61c06dbb0 2023-09-30T01:34:18.482 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:18.460+0000 7ff611ffb700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:34:18.482 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:18.460+0000 7ff611ffb700 1 -- 172.21.15.88:0/602285511 <== mon.8 v2:172.21.15.88:3304/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7ff61800ce40 con 0x7ff61c06dbb0 2023-09-30T01:34:18.482 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:18.460+0000 7ff623b43700 10 monclient: _renew_subs 2023-09-30T01:34:18.482 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:18.460+0000 7ff623b43700 10 monclient: _send_mon_message to mon.e at v2:172.21.15.88:3304/0 2023-09-30T01:34:18.482 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:18.460+0000 7ff623b43700 1 -- 172.21.15.88:0/602285511 --> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff61c09e290 con 0x7ff61c06dbb0 2023-09-30T01:34:18.482 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:18.460+0000 7ff613fff700 4 set_mon_vals no callback set 2023-09-30T01:34:18.482 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:18.460+0000 7ff611ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:34:18.482 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:18.460+0000 7ff611ffb700 10 monclient: got monmap 1 from mon.e (according to old e1) 2023-09-30T01:34:18.483 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:18.460+0000 7ff611ffb700 10 monclient: dump: 2023-09-30T01:34:18.483 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:34:18.483 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:34:18.483 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:34:18.483 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:34:18.483 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:34:18.483 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:34:18.483 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:34:18.483 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:34:18.483 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:34:18.483 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:34:18.483 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:34:18.484 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:34:18.484 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:34:18.484 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:34:18.484 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:34:18.484 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:34:18.484 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:18.460+0000 7ff623b43700 10 monclient: _renew_subs 2023-09-30T01:34:18.484 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:18.460+0000 7ff623b43700 10 monclient: _send_mon_message to mon.e at v2:172.21.15.88:3304/0 2023-09-30T01:34:18.484 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:18.460+0000 7ff623b43700 1 -- 172.21.15.88:0/602285511 --> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff61c064b80 con 0x7ff61c06dbb0 2023-09-30T01:34:18.484 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:18.460+0000 7ff611ffb700 1 -- 172.21.15.88:0/602285511 <== mon.8 v2:172.21.15.88:3304/0 4 ==== mgrmap(e 5) v1 ==== 40144+0+0 (secure 0 0 0) 0x7ff61800c850 con 0x7ff61c06dbb0 2023-09-30T01:34:18.484 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:18.460+0000 7ff623b43700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:34:18.484 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:18.460+0000 7ff623b43700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:34:18.484 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:18.460+0000 7ff623b43700 10 monclient: _send_mon_message to mon.e at v2:172.21.15.88:3304/0 2023-09-30T01:34:18.485 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:18.460+0000 7ff623b43700 1 -- 172.21.15.88:0/602285511 --> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff61c067fe0 con 0x7ff61c06dbb0 2023-09-30T01:34:18.485 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:18.460+0000 7ff611ffb700 1 --2- 172.21.15.88:0/602285511 >> v2:172.21.15.107:6812/29926 conn(0x7ff600033360 0x7ff600035810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:34:18.485 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:18.460+0000 7ff611ffb700 1 -- 172.21.15.88:0/602285511 <== mon.8 v2:172.21.15.88:3304/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 3502+0+0 (secure 0 0 0) 0x7ff6180436c0 con 0x7ff61c06dbb0 2023-09-30T01:34:18.485 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:18.460+0000 7ff6210de700 1 --2- 172.21.15.88:0/602285511 >> v2:172.21.15.107:6812/29926 conn(0x7ff600033360 0x7ff600035810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:34:18.485 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:18.460+0000 7ff6210de700 10 monclient: get_auth_request con 0x7ff600033360 auth_method 0 2023-09-30T01:34:18.485 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:18.460+0000 7ff6210de700 1 --2- 172.21.15.88:0/602285511 >> v2:172.21.15.107:6812/29926 conn(0x7ff600033360 0x7ff600035810 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7ff61c09d650 tx=0x7ff60c009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:34:18.485 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:18.468+0000 7ff611ffb700 1 -- 172.21.15.88:0/602285511 <== mon.8 v2:172.21.15.88:3304/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7ff61c067fe0 con 0x7ff61c06dbb0 2023-09-30T01:34:18.485 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:18.468+0000 7ff611ffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:34:18.485 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:18.468+0000 7ff611ffb700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:34:18.677 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:18.672+0000 7ff623b43700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T01:34:18.678 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:18.672+0000 7ff623b43700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:34:18.678 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:18.672+0000 7ff623b43700 10 monclient: _send_mon_message to mon.e at v2:172.21.15.88:3304/0 2023-09-30T01:34:18.678 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:18.672+0000 7ff623b43700 1 -- 172.21.15.88:0/602285511 --> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7ff61c067fe0 con 0x7ff61c06dbb0 2023-09-30T01:34:18.678 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:18.672+0000 7ff611ffb700 1 -- 172.21.15.88:0/602285511 <== mon.8 v2:172.21.15.88:3304/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7ff61c067fe0 con 0x7ff61c06dbb0 2023-09-30T01:34:18.678 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:18.672+0000 7ff611ffb700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:34:18.679 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:18.672+0000 7ff611ffb700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:34:18.679 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-09-30T01:34:18.679 INFO:teuthology.orchestra.run.smithi088.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:34:18.680 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:18.676+0000 7ff623b43700 1 -- 172.21.15.88:0/602285511 >> v2:172.21.15.107:6812/29926 conn(0x7ff600033360 msgr2=0x7ff600035810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:34:18.681 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:18.676+0000 7ff623b43700 1 --2- 172.21.15.88:0/602285511 >> v2:172.21.15.107:6812/29926 conn(0x7ff600033360 0x7ff600035810 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7ff61c09d650 tx=0x7ff60c009040 comp rx=0 tx=0).stop 2023-09-30T01:34:18.681 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:18.676+0000 7ff623b43700 10 monclient: shutdown 2023-09-30T01:34:18.681 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:18.676+0000 7ff623b43700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:34:18.681 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:18.676+0000 7ff623b43700 1 -- 172.21.15.88:0/602285511 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7ff61c06dbb0 msgr2=0x7ff61c140c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:34:18.681 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:18.676+0000 7ff623b43700 1 --2- 172.21.15.88:0/602285511 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7ff61c06dbb0 0x7ff61c140c60 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7ff61800ec80 tx=0x7ff6180091a0 comp rx=0 tx=0).stop 2023-09-30T01:34:18.681 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:18.676+0000 7ff623b43700 1 -- 172.21.15.88:0/602285511 shutdown_connections 2023-09-30T01:34:18.681 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:18.676+0000 7ff623b43700 1 --2- 172.21.15.88:0/602285511 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff61c06e350 0x7ff61c13cc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:34:18.681 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:18.676+0000 7ff623b43700 1 --2- 172.21.15.88:0/602285511 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7ff61c06df80 0x7ff61c13c2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:34:18.681 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:18.676+0000 7ff623b43700 1 --2- 172.21.15.88:0/602285511 >> v2:172.21.15.107:6812/29926 conn(0x7ff600033360 0x7ff600035810 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:34:18.682 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:18.676+0000 7ff623b43700 1 --2- 172.21.15.88:0/602285511 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7ff61c06dbb0 0x7ff61c140c60 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:34:18.682 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:18.676+0000 7ff623b43700 1 -- 172.21.15.88:0/602285511 >> 172.21.15.88:0/602285511 conn(0x7ff61c0a01d0 msgr2=0x7ff61c0a0b00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:34:18.682 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:18.676+0000 7ff623b43700 1 -- 172.21.15.88:0/602285511 shutdown_connections 2023-09-30T01:34:18.682 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:18.676+0000 7ff623b43700 1 -- 172.21.15.88:0/602285511 wait complete. 2023-09-30T01:34:21.693 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:34:21.806 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:21.800+0000 7ff1ae08a700 10 monclient: get_monmap_and_config 2023-09-30T01:34:21.806 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:21.800+0000 7ff1ae08a700 10 monclient: build_initial_monmap 2023-09-30T01:34:21.806 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:21.800+0000 7ff1ae08a700 10 monclient: monmap: 2023-09-30T01:34:21.806 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2023-09-30T01:34:21.806 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:34:21.806 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:34:21.806197+0000 2023-09-30T01:34:21.806 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:34:21.806197+0000 2023-09-30T01:34:21.806 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2023-09-30T01:34:21.806 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:34:21.807 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a-legacy 2023-09-30T01:34:21.807 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.107:6789/0 mon.noname-f-legacy 2023-09-30T01:34:21.807 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.noname-b 2023-09-30T01:34:21.807 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.noname-g 2023-09-30T01:34:21.807 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.noname-c 2023-09-30T01:34:21.807 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.noname-h 2023-09-30T01:34:21.807 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.noname-d 2023-09-30T01:34:21.807 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.noname-i 2023-09-30T01:34:21.807 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.noname-e 2023-09-30T01:34:21.807 INFO:teuthology.orchestra.run.smithi088.stderr:9: v2:172.21.15.88:3300/0 mon.noname-a 2023-09-30T01:34:21.807 INFO:teuthology.orchestra.run.smithi088.stderr:10: v2:172.21.15.107:3300/0 mon.noname-f 2023-09-30T01:34:21.807 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:34:21.814 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:21.800+0000 7ff1ae08a700 1 Processor -- start 2023-09-30T01:34:21.815 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:21.800+0000 7ff1ae08a700 1 -- start start 2023-09-30T01:34:21.815 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:21.800+0000 7ff1ae08a700 10 monclient: init 2023-09-30T01:34:21.815 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:21.800+0000 7ff1ae08a700 10 monclient: _reopen_session rank -1 2023-09-30T01:34:21.815 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:21.800+0000 7ff1ae08a700 10 monclient: _add_conns ranks=[0,4,7,1,10,2,9,3,6,5,8] 2023-09-30T01:34:21.815 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:21.800+0000 7ff1ae08a700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ff1a8097a40 addr v1:172.21.15.88:6789/0 2023-09-30T01:34:21.815 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:21.800+0000 7ff1ae08a700 1 --2- >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7ff1a8132e10 0x7ff1a8138650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:34:21.815 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:21.800+0000 7ff1ae08a700 10 monclient(hunting): picked mon.noname-c con 0x7ff1a8132e10 addr [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] 2023-09-30T01:34:21.815 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:21.800+0000 7ff1ae08a700 1 --2- >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7ff1a81311c0 0x7ff1a813aba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:34:21.815 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:21.800+0000 7ff1ae08a700 10 monclient(hunting): picked mon.noname-i con 0x7ff1a81311c0 addr [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] 2023-09-30T01:34:21.815 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:21.800+0000 7ff1ae08a700 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1a804bb20 con 0x7ff1a8097a40 2023-09-30T01:34:21.815 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:21.800+0000 7ff1ae08a700 10 monclient(hunting): start opening mon connection 2023-09-30T01:34:21.816 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:21.800+0000 7ff1ae08a700 1 -- --> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] -- mon_getmap magic: 0 v1 -- 0x7ff1a809a5d0 con 0x7ff1a8132e10 2023-09-30T01:34:21.816 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:21.800+0000 7ff1ae08a700 10 monclient(hunting): start opening mon connection 2023-09-30T01:34:21.816 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:21.800+0000 7ff1ae08a700 1 -- --> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] -- mon_getmap magic: 0 v1 -- 0x7ff1a8097e10 con 0x7ff1a81311c0 2023-09-30T01:34:21.816 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:21.800+0000 7ff1ae08a700 10 monclient(hunting): _renew_subs 2023-09-30T01:34:21.816 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:21.800+0000 7ff1ae08a700 10 monclient(hunting): authenticate will time out at 933.843699s 2023-09-30T01:34:21.816 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:21.800+0000 7ff1ac887700 1 --2- >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7ff1a8132e10 0x7ff1a8138650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:34:21.816 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:21.800+0000 7ff1ad088700 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7ff1a8097a40 0x7ff1a8132700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:43292/0 (socket says 172.21.15.88:43292) 2023-09-30T01:34:21.816 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:21.800+0000 7ff1ad088700 1 -- 172.21.15.88:0/805825905 learned_addr learned my addr 172.21.15.88:0/805825905 (peer_addr_for_me v1:172.21.15.88:0/0) 2023-09-30T01:34:21.816 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:21.800+0000 7ff1ac887700 10 monclient(hunting): get_auth_request con 0x7ff1a8132e10 auth_method 0 2023-09-30T01:34:21.816 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:21.800+0000 7ff1ac887700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:34:21.816 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:21.800+0000 7ff1ac887700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:34:21.816 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:21.800+0000 7ff1ac887700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:34:21.816 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:21.804+0000 7ff1ac887700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:34:21.817 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:21.804+0000 7ff1ac887700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:34:21.817 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:21.804+0000 7ff1ad889700 1 --2- 172.21.15.88:0/805825905 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7ff1a81311c0 0x7ff1a813aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:34:21.817 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:21.804+0000 7ff1ac887700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:34:21.817 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:21.804+0000 7ff1ad889700 10 monclient(hunting): get_auth_request con 0x7ff1a81311c0 auth_method 0 2023-09-30T01:34:21.817 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:21.804+0000 7ff1ad889700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:34:21.817 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:21.804+0000 7ff1ad889700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:34:21.817 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:21.804+0000 7ff1ad889700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:34:21.817 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:21.804+0000 7ff1ad889700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:34:21.817 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:21.804+0000 7ff1ad889700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:34:21.817 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:21.804+0000 7ff1ad889700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:34:21.817 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:21.804+0000 7ff1ac887700 10 monclient(hunting): handle_auth_done global_id 4504 payload 274 2023-09-30T01:34:21.817 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:21.804+0000 7ff1ac887700 1 -- 172.21.15.88:0/805825905 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7ff1a81311c0 msgr2=0x7ff1a813aba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:34:21.818 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:21.804+0000 7ff1ac887700 1 --2- 172.21.15.88:0/805825905 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7ff1a81311c0 0x7ff1a813aba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:34:21.818 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:21.804+0000 7ff19ffff700 1 -- 172.21.15.88:0/805825905 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1287434303 0 0) 0x7ff1a804bb20 con 0x7ff1a8097a40 2023-09-30T01:34:21.818 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:21.804+0000 7ff1ac887700 1 -- 172.21.15.88:0/805825905 >> v1:172.21.15.88:6789/0 conn(0x7ff1a8097a40 legacy=0x7ff1a8132700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:34:21.818 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:21.804+0000 7ff1ac887700 10 monclient: _finish_hunting 0 2023-09-30T01:34:21.818 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:21.804+0000 7ff1ac887700 1 monclient: found mon.noname-c 2023-09-30T01:34:21.818 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:21.804+0000 7ff1ac887700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:34:21.818 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:21.804+0000 7ff1ac887700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.88:3302/0 2023-09-30T01:34:21.818 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:21.804+0000 7ff1ac887700 1 -- 172.21.15.88:0/805825905 --> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1a8098d40 con 0x7ff1a8132e10 2023-09-30T01:34:21.818 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:21.804+0000 7ff19ffff700 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2023-09-30T01:34:21.818 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:21.804+0000 7ff1ac887700 1 --2- 172.21.15.88:0/805825905 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7ff1a8132e10 0x7ff1a8138650 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7ff1a4009e90 tx=0x7ff1a4009780 comp rx=0 tx=0).ready entity=mon.4 client_cookie=815fc6a36857245d server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:34:21.818 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:21.804+0000 7ff19ffff700 1 -- 172.21.15.88:0/805825905 <== mon.4 v2:172.21.15.88:3302/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7ff1a400e070 con 0x7ff1a8132e10 2023-09-30T01:34:21.818 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:21.804+0000 7ff19ffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:34:21.819 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:21.804+0000 7ff19ffff700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T01:34:21.819 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:21.804+0000 7ff19ffff700 10 monclient: dump: 2023-09-30T01:34:21.819 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:34:21.819 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:34:21.819 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:34:21.819 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:34:21.819 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:34:21.819 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:34:21.819 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:34:21.819 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:34:21.819 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:34:21.819 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:34:21.820 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:34:21.820 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:34:21.820 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:34:21.820 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:34:21.820 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:34:21.820 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:34:21.820 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:21.804+0000 7ff19ffff700 10 monclient: _finish_auth 0 2023-09-30T01:34:21.820 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:21.804+0000 7ff19ffff700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:34:21.820 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:21.804+0000 7ff19ffff700 1 -- 172.21.15.88:0/805825905 <== mon.4 v2:172.21.15.88:3302/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff1a400bc30 con 0x7ff1a8132e10 2023-09-30T01:34:21.820 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:21.804+0000 7ff1ae08a700 5 monclient: authenticate success, global_id 4504 2023-09-30T01:34:21.820 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:21.804+0000 7ff1ae08a700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:34:21.820 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:21.804+0000 7ff19ffff700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:34:21.820 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:21.804+0000 7ff1ae08a700 10 monclient: get_monmap_and_config success 2023-09-30T01:34:21.821 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:21.804+0000 7ff1ae08a700 4 set_mon_vals no callback set 2023-09-30T01:34:21.821 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:21.804+0000 7ff19ffff700 1 -- 172.21.15.88:0/805825905 <== mon.4 v2:172.21.15.88:3302/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7ff1a400bda0 con 0x7ff1a8132e10 2023-09-30T01:34:21.821 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:21.804+0000 7ff19ffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:34:21.821 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:21.804+0000 7ff19ffff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T01:34:21.821 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:21.804+0000 7ff19ffff700 10 monclient: dump: 2023-09-30T01:34:21.821 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:34:21.821 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:34:21.821 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:34:21.821 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:34:21.821 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:34:21.821 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:34:21.821 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:34:21.822 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:34:21.822 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:34:21.822 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:34:21.822 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:34:21.822 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:34:21.822 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:34:21.822 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:34:21.822 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:34:21.822 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:34:21.822 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:21.804+0000 7ff1ae08a700 10 monclient: shutdown 2023-09-30T01:34:21.822 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:21.804+0000 7ff1ae08a700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:34:21.822 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:21.804+0000 7ff1ae08a700 1 -- 172.21.15.88:0/805825905 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7ff1a8132e10 msgr2=0x7ff1a8138650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:34:21.822 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:21.804+0000 7ff1ae08a700 1 --2- 172.21.15.88:0/805825905 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7ff1a8132e10 0x7ff1a8138650 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7ff1a4009e90 tx=0x7ff1a4009780 comp rx=0 tx=0).stop 2023-09-30T01:34:21.823 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:21.804+0000 7ff1ae08a700 1 -- 172.21.15.88:0/805825905 shutdown_connections 2023-09-30T01:34:21.823 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:21.804+0000 7ff1ae08a700 1 --2- 172.21.15.88:0/805825905 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7ff1a81311c0 0x7ff1a813aba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:34:21.823 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:21.804+0000 7ff1ae08a700 1 --2- 172.21.15.88:0/805825905 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7ff1a8132e10 0x7ff1a8138650 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:34:21.823 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:21.804+0000 7ff1ae08a700 1 -- 172.21.15.88:0/805825905 >> 172.21.15.88:0/805825905 conn(0x7ff1a806e5e0 msgr2=0x7ff1a806e9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:34:21.823 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:21.804+0000 7ff1ae08a700 1 -- 172.21.15.88:0/805825905 shutdown_connections 2023-09-30T01:34:21.823 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:21.804+0000 7ff1ae08a700 1 -- 172.21.15.88:0/805825905 wait complete. 2023-09-30T01:34:21.823 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:21.804+0000 7ff1ae08a700 10 monclient: build_initial_monmap 2023-09-30T01:34:21.823 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:21.804+0000 7ff1ae08a700 10 monclient: monmap: 2023-09-30T01:34:21.823 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2023-09-30T01:34:21.823 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:34:21.823 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 0.000000 2023-09-30T01:34:21.823 INFO:teuthology.orchestra.run.smithi088.stderr:created 0.000000 2023-09-30T01:34:21.824 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2023-09-30T01:34:21.824 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 1 2023-09-30T01:34:21.824 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.noname-a 2023-09-30T01:34:21.824 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.noname-f 2023-09-30T01:34:21.824 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.noname-b 2023-09-30T01:34:21.824 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.noname-g 2023-09-30T01:34:21.824 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.noname-c 2023-09-30T01:34:21.824 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.noname-h 2023-09-30T01:34:21.824 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.noname-d 2023-09-30T01:34:21.824 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.noname-i 2023-09-30T01:34:21.824 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.noname-e 2023-09-30T01:34:21.824 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:34:21.824 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:21.804+0000 7ff1ae08a700 1 Processor -- start 2023-09-30T01:34:21.825 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:21.804+0000 7ff1ae08a700 1 -- start start 2023-09-30T01:34:21.825 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:21.804+0000 7ff1ae08a700 10 monclient: init 2023-09-30T01:34:21.825 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:21.804+0000 7ff1ae08a700 10 monclient: _reopen_session rank -1 2023-09-30T01:34:21.825 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:21.804+0000 7ff1ae08a700 10 monclient: _add_conns ranks=[4,0,5,3,6,1,8,2,7] 2023-09-30T01:34:21.825 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:21.804+0000 7ff1ae08a700 1 --2- >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7ff1a8097a40 0x7ff1a812fff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:34:21.825 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:21.804+0000 7ff1ae08a700 10 monclient(hunting): picked mon.noname-c con 0x7ff1a8097a40 addr [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] 2023-09-30T01:34:21.825 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:21.804+0000 7ff1ae08a700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff1a81311c0 0x7ff1a811cd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:34:21.825 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:21.804+0000 7ff1ae08a700 10 monclient(hunting): picked mon.noname-a con 0x7ff1a81311c0 addr [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] 2023-09-30T01:34:21.825 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:21.804+0000 7ff1ae08a700 1 --2- >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7ff1a8132e10 0x7ff1a811d290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:34:21.825 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:21.804+0000 7ff1ae08a700 10 monclient(hunting): picked mon.noname-h con 0x7ff1a8132e10 addr [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] 2023-09-30T01:34:21.825 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:21.804+0000 7ff1ae08a700 10 monclient(hunting): start opening mon connection 2023-09-30T01:34:21.825 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:21.804+0000 7ff1ae08a700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1a8099c30 con 0x7ff1a81311c0 2023-09-30T01:34:21.826 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:21.804+0000 7ff1ae08a700 10 monclient(hunting): start opening mon connection 2023-09-30T01:34:21.826 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:21.804+0000 7ff1ae08a700 1 -- --> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] -- mon_getmap magic: 0 v1 -- 0x7ff1a80a3dc0 con 0x7ff1a8097a40 2023-09-30T01:34:21.826 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:21.804+0000 7ff1ae08a700 10 monclient(hunting): start opening mon connection 2023-09-30T01:34:21.826 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:21.804+0000 7ff1ae08a700 1 -- --> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] -- mon_getmap magic: 0 v1 -- 0x7ff1a80996b0 con 0x7ff1a8132e10 2023-09-30T01:34:21.826 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:21.804+0000 7ff1ae08a700 10 monclient(hunting): _renew_subs 2023-09-30T01:34:21.826 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:21.804+0000 7ff1ae08a700 10 monclient(hunting): authenticate will time out at 933.847853s 2023-09-30T01:34:21.826 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:21.808+0000 7ff1ad088700 1 --2- >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7ff1a8097a40 0x7ff1a812fff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:34:21.826 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:21.808+0000 7ff1ac887700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff1a81311c0 0x7ff1a811cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:34:21.826 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:21.808+0000 7ff1ad088700 1 --2- >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7ff1a8097a40 0x7ff1a812fff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3302/0 says I am v2:172.21.15.88:40314/0 (socket says 172.21.15.88:40314) 2023-09-30T01:34:21.826 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:21.808+0000 7ff1ac887700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff1a81311c0 0x7ff1a811cd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:40362/0 (socket says 172.21.15.88:40362) 2023-09-30T01:34:21.826 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:21.808+0000 7ff1ad088700 1 -- 172.21.15.88:0/4120860452 learned_addr learned my addr 172.21.15.88:0/4120860452 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-09-30T01:34:21.827 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:21.808+0000 7ff1ad088700 10 monclient(hunting): get_auth_request con 0x7ff1a8097a40 auth_method 0 2023-09-30T01:34:21.827 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:21.808+0000 7ff1ad088700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:34:21.827 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:21.808+0000 7ff1ad088700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:34:21.827 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:21.808+0000 7ff1ad088700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:34:21.827 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:21.808+0000 7ff1ad889700 1 --2- 172.21.15.88:0/4120860452 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7ff1a8132e10 0x7ff1a811d290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:34:21.827 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:21.808+0000 7ff1ac887700 10 monclient(hunting): get_auth_request con 0x7ff1a81311c0 auth_method 0 2023-09-30T01:34:21.827 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:21.808+0000 7ff1ac887700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:34:21.827 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:21.808+0000 7ff1ac887700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:34:21.827 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:21.808+0000 7ff1ac887700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:34:21.827 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:21.808+0000 7ff1ad889700 10 monclient(hunting): get_auth_request con 0x7ff1a8132e10 auth_method 0 2023-09-30T01:34:21.827 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:21.808+0000 7ff1ad889700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:34:21.827 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:21.808+0000 7ff1ad889700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:34:21.828 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:21.808+0000 7ff1ad889700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:34:21.828 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:21.808+0000 7ff1ad088700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:34:21.828 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:21.808+0000 7ff1ad088700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:34:21.828 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:21.808+0000 7ff1ad088700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:34:21.828 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:21.808+0000 7ff1ac887700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:34:21.828 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:21.808+0000 7ff1ac887700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:34:21.828 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:21.808+0000 7ff1ac887700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:34:21.828 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:21.808+0000 7ff1ad889700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:34:21.828 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:21.808+0000 7ff1ad889700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:34:21.828 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:21.808+0000 7ff1ad889700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:34:21.828 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:21.808+0000 7ff1ad088700 10 monclient(hunting): handle_auth_done global_id 4513 payload 824 2023-09-30T01:34:21.828 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:21.808+0000 7ff1ad088700 1 -- 172.21.15.88:0/4120860452 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7ff1a8132e10 msgr2=0x7ff1a811d290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:34:21.829 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:21.808+0000 7ff1ad088700 1 --2- 172.21.15.88:0/4120860452 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7ff1a8132e10 0x7ff1a811d290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:34:21.829 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:21.808+0000 7ff1ad088700 1 -- 172.21.15.88:0/4120860452 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff1a81311c0 msgr2=0x7ff1a811cd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:34:21.829 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:21.808+0000 7ff1ad088700 1 --2- 172.21.15.88:0/4120860452 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff1a81311c0 0x7ff1a811cd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:34:21.829 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:21.808+0000 7ff1ad088700 10 monclient: _finish_hunting 0 2023-09-30T01:34:21.829 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:21.808+0000 7ff1ad088700 1 monclient: found mon.noname-c 2023-09-30T01:34:21.829 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:21.808+0000 7ff1ad088700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:34:21.829 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:21.808+0000 7ff1ad088700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.88:3302/0 2023-09-30T01:34:21.829 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:21.808+0000 7ff1ad088700 1 -- 172.21.15.88:0/4120860452 --> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1a4009a20 con 0x7ff1a8097a40 2023-09-30T01:34:21.829 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:21.808+0000 7ff1ad088700 1 --2- 172.21.15.88:0/4120860452 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7ff1a8097a40 0x7ff1a812fff0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7ff19800c840 tx=0x7ff198007d90 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:34:21.829 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:21.808+0000 7ff19d7fa700 1 -- 172.21.15.88:0/4120860452 <== mon.4 v2:172.21.15.88:3302/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7ff1980070d0 con 0x7ff1a8097a40 2023-09-30T01:34:21.829 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:21.808+0000 7ff19d7fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:34:21.830 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:21.808+0000 7ff19d7fa700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T01:34:21.830 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:21.808+0000 7ff19d7fa700 10 monclient: dump: 2023-09-30T01:34:21.830 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:34:21.830 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:34:21.830 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:34:21.830 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:34:21.830 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:34:21.830 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:34:21.830 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:34:21.830 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:34:21.830 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:34:21.831 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:34:21.831 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:34:21.831 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:34:21.831 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:34:21.831 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:34:21.831 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:34:21.831 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:34:21.831 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:21.808+0000 7ff19d7fa700 10 monclient: _finish_auth 0 2023-09-30T01:34:21.831 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:21.808+0000 7ff19d7fa700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:34:21.831 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:21.808+0000 7ff1ae08a700 5 monclient: authenticate success, global_id 4513 2023-09-30T01:34:21.831 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:21.808+0000 7ff19d7fa700 1 -- 172.21.15.88:0/4120860452 <== mon.4 v2:172.21.15.88:3302/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff198007250 con 0x7ff1a8097a40 2023-09-30T01:34:21.831 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:21.808+0000 7ff1ae08a700 10 monclient: _renew_subs 2023-09-30T01:34:21.832 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:21.808+0000 7ff1ae08a700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.88:3302/0 2023-09-30T01:34:21.832 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:21.808+0000 7ff1ae08a700 1 -- 172.21.15.88:0/4120860452 --> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff1a8120140 con 0x7ff1a8097a40 2023-09-30T01:34:21.832 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:21.808+0000 7ff19d7fa700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:34:21.832 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:21.808+0000 7ff19d7fa700 1 -- 172.21.15.88:0/4120860452 <== mon.4 v2:172.21.15.88:3302/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7ff198011360 con 0x7ff1a8097a40 2023-09-30T01:34:21.832 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:21.808+0000 7ff19d7fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:34:21.832 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:21.808+0000 7ff19f7fe700 4 set_mon_vals no callback set 2023-09-30T01:34:21.832 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:21.808+0000 7ff19d7fa700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T01:34:21.832 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:21.808+0000 7ff19d7fa700 10 monclient: dump: 2023-09-30T01:34:21.832 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:34:21.832 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:34:21.832 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:34:21.832 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:34:21.833 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:34:21.833 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:34:21.833 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:34:21.833 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:34:21.833 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:34:21.833 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:34:21.833 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:34:21.833 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:34:21.833 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:34:21.833 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:34:21.833 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:34:21.833 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:34:21.833 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:21.808+0000 7ff1ae08a700 10 monclient: _renew_subs 2023-09-30T01:34:21.834 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:21.808+0000 7ff1ae08a700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.88:3302/0 2023-09-30T01:34:21.834 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:21.808+0000 7ff1ae08a700 1 -- 172.21.15.88:0/4120860452 --> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff1a811e610 con 0x7ff1a8097a40 2023-09-30T01:34:21.834 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:21.808+0000 7ff1ae08a700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:34:21.834 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:21.808+0000 7ff1ae08a700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:34:21.834 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:21.808+0000 7ff1ae08a700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.88:3302/0 2023-09-30T01:34:21.834 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:21.808+0000 7ff19d7fa700 1 -- 172.21.15.88:0/4120860452 <== mon.4 v2:172.21.15.88:3302/0 4 ==== mgrmap(e 5) v1 ==== 40144+0+0 (secure 0 0 0) 0x7ff198005b30 con 0x7ff1a8097a40 2023-09-30T01:34:21.834 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:21.808+0000 7ff1ae08a700 1 -- 172.21.15.88:0/4120860452 --> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff1a8071960 con 0x7ff1a8097a40 2023-09-30T01:34:21.834 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:21.808+0000 7ff19d7fa700 1 --2- 172.21.15.88:0/4120860452 >> v2:172.21.15.107:6812/29926 conn(0x7ff18c033360 0x7ff18c035810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:34:21.834 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:21.808+0000 7ff19d7fa700 1 -- 172.21.15.88:0/4120860452 <== mon.4 v2:172.21.15.88:3302/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 3502+0+0 (secure 0 0 0) 0x7ff1980424a0 con 0x7ff1a8097a40 2023-09-30T01:34:21.835 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:21.812+0000 7ff1ac887700 1 --2- 172.21.15.88:0/4120860452 >> v2:172.21.15.107:6812/29926 conn(0x7ff18c033360 0x7ff18c035810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:34:21.835 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:21.812+0000 7ff1ac887700 10 monclient: get_auth_request con 0x7ff18c033360 auth_method 0 2023-09-30T01:34:21.835 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:21.812+0000 7ff1ac887700 1 --2- 172.21.15.88:0/4120860452 >> v2:172.21.15.107:6812/29926 conn(0x7ff18c033360 0x7ff18c035810 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7ff1a4009e60 tx=0x7ff1a4019000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:34:21.836 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:21.816+0000 7ff19d7fa700 1 -- 172.21.15.88:0/4120860452 <== mon.4 v2:172.21.15.88:3302/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7ff198018680 con 0x7ff1a8097a40 2023-09-30T01:34:21.836 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:21.816+0000 7ff19d7fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:34:21.836 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:21.816+0000 7ff19d7fa700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:34:22.016 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:22.012+0000 7ff1ae08a700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T01:34:22.016 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:22.012+0000 7ff1ae08a700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:34:22.016 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:22.012+0000 7ff1ae08a700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.88:3302/0 2023-09-30T01:34:22.017 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:22.012+0000 7ff1ae08a700 1 -- 172.21.15.88:0/4120860452 --> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7ff1a80a44f0 con 0x7ff1a8097a40 2023-09-30T01:34:22.017 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:22.012+0000 7ff19d7fa700 1 -- 172.21.15.88:0/4120860452 <== mon.4 v2:172.21.15.88:3302/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7ff198017020 con 0x7ff1a8097a40 2023-09-30T01:34:22.017 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:22.012+0000 7ff19d7fa700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:34:22.017 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:22.012+0000 7ff19d7fa700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:34:22.017 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-09-30T01:34:22.017 INFO:teuthology.orchestra.run.smithi088.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:34:22.019 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:22.012+0000 7ff1ae08a700 1 -- 172.21.15.88:0/4120860452 >> v2:172.21.15.107:6812/29926 conn(0x7ff18c033360 msgr2=0x7ff18c035810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:34:22.020 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:22.012+0000 7ff1ae08a700 1 --2- 172.21.15.88:0/4120860452 >> v2:172.21.15.107:6812/29926 conn(0x7ff18c033360 0x7ff18c035810 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7ff1a4009e60 tx=0x7ff1a4019000 comp rx=0 tx=0).stop 2023-09-30T01:34:22.020 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:22.012+0000 7ff1ae08a700 10 monclient: shutdown 2023-09-30T01:34:22.020 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:22.012+0000 7ff1ae08a700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:34:22.020 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:22.012+0000 7ff1ae08a700 1 -- 172.21.15.88:0/4120860452 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7ff1a8097a40 msgr2=0x7ff1a812fff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:34:22.020 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:22.012+0000 7ff1ae08a700 1 --2- 172.21.15.88:0/4120860452 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7ff1a8097a40 0x7ff1a812fff0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7ff19800c840 tx=0x7ff198007d90 comp rx=0 tx=0).stop 2023-09-30T01:34:22.021 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:22.012+0000 7ff1ae08a700 1 -- 172.21.15.88:0/4120860452 shutdown_connections 2023-09-30T01:34:22.021 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:22.012+0000 7ff1ae08a700 1 --2- 172.21.15.88:0/4120860452 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7ff1a8132e10 0x7ff1a811d290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:34:22.021 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:22.012+0000 7ff1ae08a700 1 --2- 172.21.15.88:0/4120860452 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff1a81311c0 0x7ff1a811cd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:34:22.021 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:22.012+0000 7ff1ae08a700 1 --2- 172.21.15.88:0/4120860452 >> v2:172.21.15.107:6812/29926 conn(0x7ff18c033360 0x7ff18c035810 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:34:22.021 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:22.012+0000 7ff1ae08a700 1 --2- 172.21.15.88:0/4120860452 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7ff1a8097a40 0x7ff1a812fff0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:34:22.022 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:22.012+0000 7ff1ae08a700 1 -- 172.21.15.88:0/4120860452 >> 172.21.15.88:0/4120860452 conn(0x7ff1a806e5e0 msgr2=0x7ff1a81339c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:34:22.022 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:22.012+0000 7ff1ae08a700 1 -- 172.21.15.88:0/4120860452 shutdown_connections 2023-09-30T01:34:22.022 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:22.012+0000 7ff1ae08a700 1 -- 172.21.15.88:0/4120860452 wait complete. 2023-09-30T01:34:25.035 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:34:25.145 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:25.140+0000 7f78df022700 10 monclient: get_monmap_and_config 2023-09-30T01:34:25.146 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:25.140+0000 7f78df022700 10 monclient: build_initial_monmap 2023-09-30T01:34:25.146 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:25.140+0000 7f78df022700 10 monclient: monmap: 2023-09-30T01:34:25.146 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2023-09-30T01:34:25.146 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:34:25.146 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:34:25.146115+0000 2023-09-30T01:34:25.146 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:34:25.146115+0000 2023-09-30T01:34:25.146 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2023-09-30T01:34:25.146 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:34:25.146 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a-legacy 2023-09-30T01:34:25.147 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.107:6789/0 mon.noname-f-legacy 2023-09-30T01:34:25.147 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.noname-b 2023-09-30T01:34:25.147 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.noname-g 2023-09-30T01:34:25.147 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.noname-c 2023-09-30T01:34:25.147 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.noname-h 2023-09-30T01:34:25.147 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.noname-d 2023-09-30T01:34:25.147 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.noname-i 2023-09-30T01:34:25.147 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.noname-e 2023-09-30T01:34:25.147 INFO:teuthology.orchestra.run.smithi088.stderr:9: v2:172.21.15.88:3300/0 mon.noname-a 2023-09-30T01:34:25.147 INFO:teuthology.orchestra.run.smithi088.stderr:10: v2:172.21.15.107:3300/0 mon.noname-f 2023-09-30T01:34:25.147 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:34:25.151 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:25.140+0000 7f78df022700 1 Processor -- start 2023-09-30T01:34:25.151 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:25.140+0000 7f78df022700 1 -- start start 2023-09-30T01:34:25.151 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:25.140+0000 7f78df022700 10 monclient: init 2023-09-30T01:34:25.151 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:25.144+0000 7f78df022700 10 monclient: _reopen_session rank -1 2023-09-30T01:34:25.151 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:25.144+0000 7f78df022700 10 monclient: _add_conns ranks=[1,0,3,7,4,5,6,8,2,10,9] 2023-09-30T01:34:25.151 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:25.144+0000 7f78df022700 10 monclient(hunting): picked mon.noname-f-legacy con 0x7f78d80a1340 addr v1:172.21.15.107:6789/0 2023-09-30T01:34:25.151 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:25.144+0000 7f78df022700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f78d806ede0 addr v1:172.21.15.88:6789/0 2023-09-30T01:34:25.151 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:25.144+0000 7f78df022700 1 --2- >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7f78d8137570 0x7f78d8139950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:34:25.151 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:25.144+0000 7f78df022700 10 monclient(hunting): picked mon.noname-g con 0x7f78d8137570 addr [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] 2023-09-30T01:34:25.152 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:25.144+0000 7f78df022700 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78d805ecb0 con 0x7f78d806ede0 2023-09-30T01:34:25.152 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:25.144+0000 7f78df022700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78d80a0090 con 0x7f78d80a1340 2023-09-30T01:34:25.152 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:25.144+0000 7f78df022700 10 monclient(hunting): start opening mon connection 2023-09-30T01:34:25.152 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:25.144+0000 7f78df022700 1 -- --> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78d8131f30 con 0x7f78d8137570 2023-09-30T01:34:25.152 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:25.144+0000 7f78df022700 10 monclient(hunting): _renew_subs 2023-09-30T01:34:25.152 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:25.144+0000 7f78df022700 10 monclient(hunting): authenticate will time out at 937.184563s 2023-09-30T01:34:25.152 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:25.144+0000 7f78d7fff700 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f78d806ede0 0x7f78d806f1b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:43300/0 (socket says 172.21.15.88:43300) 2023-09-30T01:34:25.152 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:25.144+0000 7f78d7fff700 1 -- 172.21.15.88:0/1428427299 learned_addr learned my addr 172.21.15.88:0/1428427299 (peer_addr_for_me v1:172.21.15.88:0/0) 2023-09-30T01:34:25.152 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:25.144+0000 7f78dd5bf700 1 --2- 172.21.15.88:0/1428427299 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7f78d8137570 0x7f78d8139950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:34:25.152 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:25.144+0000 7f78dd5bf700 10 monclient(hunting): get_auth_request con 0x7f78d8137570 auth_method 0 2023-09-30T01:34:25.152 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:25.144+0000 7f78dd5bf700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:34:25.152 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:25.144+0000 7f78dd5bf700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:34:25.153 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:25.144+0000 7f78dd5bf700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:34:25.153 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:25.144+0000 7f78dd5bf700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:34:25.153 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:25.144+0000 7f78dd5bf700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:34:25.153 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:25.144+0000 7f78dd5bf700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:34:25.153 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:25.144+0000 7f78d77fe700 1 -- 172.21.15.88:0/1428427299 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1206433737 0 0) 0x7f78d805ecb0 con 0x7f78d806ede0 2023-09-30T01:34:25.153 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:25.144+0000 7f78d77fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:34:25.153 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:25.144+0000 7f78d77fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:34:25.153 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:25.144+0000 7f78d77fe700 10 monclient(hunting): my global_id is 4644 2023-09-30T01:34:25.153 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:25.144+0000 7f78d77fe700 1 -- 172.21.15.88:0/1428427299 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f78c4003570 con 0x7f78d806ede0 2023-09-30T01:34:25.153 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:25.144+0000 7f78dd5bf700 10 monclient(hunting): handle_auth_done global_id 4440 payload 274 2023-09-30T01:34:25.153 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:25.144+0000 7f78dd5bf700 1 -- 172.21.15.88:0/1428427299 >> v1:172.21.15.107:6789/0 conn(0x7f78d80a1340 legacy=0x7f78d8131820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:34:25.153 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:25.144+0000 7f78dd5bf700 1 -- 172.21.15.88:0/1428427299 >> v1:172.21.15.88:6789/0 conn(0x7f78d806ede0 legacy=0x7f78d806f1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:34:25.154 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:25.144+0000 7f78dd5bf700 10 monclient: _finish_hunting 0 2023-09-30T01:34:25.154 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:25.144+0000 7f78dd5bf700 1 monclient: found mon.noname-g 2023-09-30T01:34:25.154 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:25.144+0000 7f78dd5bf700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:34:25.154 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:25.144+0000 7f78dd5bf700 10 monclient: _send_mon_message to mon.noname-g at v2:172.21.15.107:3301/0 2023-09-30T01:34:25.154 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:25.144+0000 7f78dd5bf700 1 -- 172.21.15.88:0/1428427299 --> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78d80a1710 con 0x7f78d8137570 2023-09-30T01:34:25.154 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:25.144+0000 7f78dd5bf700 1 --2- 172.21.15.88:0/1428427299 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7f78d8137570 0x7f78d8139950 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f78d0009e90 tx=0x7f78d0009780 comp rx=0 tx=0).ready entity=mon.3 client_cookie=763f52b295a100a2 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:34:25.154 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:25.144+0000 7f78d77fe700 1 -- 172.21.15.88:0/1428427299 <== mon.3 v2:172.21.15.107:3301/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f78d000e070 con 0x7f78d8137570 2023-09-30T01:34:25.154 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:25.144+0000 7f78d77fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:34:25.154 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:25.144+0000 7f78d77fe700 10 monclient: got monmap 1 from mon.noname-g (according to old e1) 2023-09-30T01:34:25.154 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:25.144+0000 7f78d77fe700 10 monclient: dump: 2023-09-30T01:34:25.154 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:34:25.155 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:34:25.155 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:34:25.155 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:34:25.155 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:34:25.155 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:34:25.155 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:34:25.155 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:34:25.155 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:34:25.155 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:34:25.155 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:34:25.155 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:34:25.155 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:34:25.155 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:34:25.156 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:34:25.156 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:34:25.156 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:25.144+0000 7f78d77fe700 10 monclient: _finish_auth 0 2023-09-30T01:34:25.156 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:25.144+0000 7f78d77fe700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:34:25.156 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:25.144+0000 7f78df022700 5 monclient: authenticate success, global_id 4440 2023-09-30T01:34:25.156 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:25.144+0000 7f78df022700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:34:25.156 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:25.144+0000 7f78d77fe700 1 -- 172.21.15.88:0/1428427299 <== mon.3 v2:172.21.15.107:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f78d000bc30 con 0x7f78d8137570 2023-09-30T01:34:25.156 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:25.144+0000 7f78d77fe700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:34:25.156 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:25.144+0000 7f78df022700 10 monclient: get_monmap_and_config success 2023-09-30T01:34:25.156 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:25.144+0000 7f78df022700 4 set_mon_vals no callback set 2023-09-30T01:34:25.156 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:25.144+0000 7f78d77fe700 1 -- 172.21.15.88:0/1428427299 <== mon.3 v2:172.21.15.107:3301/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f78d000bda0 con 0x7f78d8137570 2023-09-30T01:34:25.157 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:25.144+0000 7f78d77fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:34:25.157 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:25.144+0000 7f78d77fe700 10 monclient: got monmap 1 from mon.g (according to old e1) 2023-09-30T01:34:25.157 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:25.144+0000 7f78d77fe700 10 monclient: dump: 2023-09-30T01:34:25.157 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:34:25.157 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:34:25.157 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:34:25.157 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:34:25.157 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:34:25.157 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:34:25.157 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:34:25.157 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:34:25.157 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:34:25.157 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:34:25.158 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:34:25.158 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:34:25.158 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:34:25.158 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:34:25.158 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:34:25.158 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:34:25.158 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:25.144+0000 7f78df022700 10 monclient: shutdown 2023-09-30T01:34:25.158 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:25.144+0000 7f78df022700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:34:25.158 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:25.144+0000 7f78df022700 1 -- 172.21.15.88:0/1428427299 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7f78d8137570 msgr2=0x7f78d8139950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:34:25.158 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:25.144+0000 7f78df022700 1 --2- 172.21.15.88:0/1428427299 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7f78d8137570 0x7f78d8139950 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f78d0009e90 tx=0x7f78d0009780 comp rx=0 tx=0).stop 2023-09-30T01:34:25.158 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:25.144+0000 7f78df022700 1 -- 172.21.15.88:0/1428427299 shutdown_connections 2023-09-30T01:34:25.159 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:25.144+0000 7f78df022700 1 --2- 172.21.15.88:0/1428427299 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7f78d8137570 0x7f78d8139950 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:34:25.159 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:25.144+0000 7f78df022700 1 -- 172.21.15.88:0/1428427299 >> 172.21.15.88:0/1428427299 conn(0x7f78d8097a40 msgr2=0x7f78d8063450 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:34:25.159 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:25.144+0000 7f78df022700 1 -- 172.21.15.88:0/1428427299 shutdown_connections 2023-09-30T01:34:25.159 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:25.144+0000 7f78df022700 1 -- 172.21.15.88:0/1428427299 wait complete. 2023-09-30T01:34:25.159 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:25.144+0000 7f78df022700 10 monclient: build_initial_monmap 2023-09-30T01:34:25.159 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:25.148+0000 7f78df022700 10 monclient: monmap: 2023-09-30T01:34:25.159 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2023-09-30T01:34:25.159 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:34:25.159 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 0.000000 2023-09-30T01:34:25.159 INFO:teuthology.orchestra.run.smithi088.stderr:created 0.000000 2023-09-30T01:34:25.159 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2023-09-30T01:34:25.159 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 1 2023-09-30T01:34:25.160 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.noname-a 2023-09-30T01:34:25.160 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.noname-f 2023-09-30T01:34:25.160 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.noname-b 2023-09-30T01:34:25.160 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.noname-g 2023-09-30T01:34:25.160 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.noname-c 2023-09-30T01:34:25.160 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.noname-h 2023-09-30T01:34:25.160 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.noname-d 2023-09-30T01:34:25.160 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.noname-i 2023-09-30T01:34:25.160 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.noname-e 2023-09-30T01:34:25.160 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:34:25.164 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:25.148+0000 7f78df022700 1 Processor -- start 2023-09-30T01:34:25.164 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:25.148+0000 7f78df022700 1 -- start start 2023-09-30T01:34:25.165 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:25.148+0000 7f78df022700 10 monclient: init 2023-09-30T01:34:25.165 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:25.148+0000 7f78df022700 10 monclient: _reopen_session rank -1 2023-09-30T01:34:25.165 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:25.148+0000 7f78df022700 10 monclient: _add_conns ranks=[8,7,1,3,2,4,5,6,0] 2023-09-30T01:34:25.165 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:25.148+0000 7f78df022700 1 --2- >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f78d806ede0 0x7f78d8067fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:34:25.165 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:25.148+0000 7f78df022700 10 monclient(hunting): picked mon.noname-e con 0x7f78d806ede0 addr [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] 2023-09-30T01:34:25.165 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:25.148+0000 7f78df022700 1 --2- >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7f78d80a1340 0x7f78d8068500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:34:25.165 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:25.148+0000 7f78df022700 10 monclient(hunting): picked mon.noname-i con 0x7f78d80a1340 addr [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] 2023-09-30T01:34:25.165 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:25.148+0000 7f78df022700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f78d8137570 0x7f78d8068a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:34:25.165 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:25.148+0000 7f78df022700 10 monclient(hunting): picked mon.noname-f con 0x7f78d8137570 addr [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] 2023-09-30T01:34:25.165 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:25.148+0000 7f78df022700 10 monclient(hunting): start opening mon connection 2023-09-30T01:34:25.166 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:25.148+0000 7f78df022700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78d8069f50 con 0x7f78d8137570 2023-09-30T01:34:25.166 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:25.148+0000 7f78df022700 10 monclient(hunting): start opening mon connection 2023-09-30T01:34:25.166 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:25.148+0000 7f78df022700 1 -- --> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] -- mon_getmap magic: 0 v1 -- 0x7f78d806c9d0 con 0x7f78d80a1340 2023-09-30T01:34:25.166 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:25.148+0000 7f78df022700 10 monclient(hunting): start opening mon connection 2023-09-30T01:34:25.166 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:25.148+0000 7f78df022700 1 -- --> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] -- mon_getmap magic: 0 v1 -- 0x7f78d811bf30 con 0x7f78d806ede0 2023-09-30T01:34:25.166 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:25.148+0000 7f78df022700 10 monclient(hunting): _renew_subs 2023-09-30T01:34:25.166 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:25.148+0000 7f78df022700 10 monclient(hunting): authenticate will time out at 937.189216s 2023-09-30T01:34:25.166 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:25.148+0000 7f78dcdbe700 1 --2- >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f78d806ede0 0x7f78d8067fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:34:25.166 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:25.148+0000 7f78dd5bf700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f78d8137570 0x7f78d8068a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:34:25.166 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:25.148+0000 7f78d7fff700 1 --2- >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7f78d80a1340 0x7f78d8068500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:34:25.166 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:25.148+0000 7f78d7fff700 1 --2- >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7f78d80a1340 0x7f78d8068500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3303/0 says I am v2:172.21.15.88:53928/0 (socket says 172.21.15.88:53928) 2023-09-30T01:34:25.167 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:25.148+0000 7f78dcdbe700 1 --2- >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f78d806ede0 0x7f78d8067fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3304/0 says I am v2:172.21.15.88:39354/0 (socket says 172.21.15.88:39354) 2023-09-30T01:34:25.167 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:25.148+0000 7f78d7fff700 1 -- 172.21.15.88:0/2585224170 learned_addr learned my addr 172.21.15.88:0/2585224170 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-09-30T01:34:25.167 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:25.148+0000 7f78dd5bf700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f78d8137570 0x7f78d8068a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.88:37804/0 (socket says 172.21.15.88:37804) 2023-09-30T01:34:25.167 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:25.148+0000 7f78dd5bf700 10 monclient(hunting): get_auth_request con 0x7f78d8137570 auth_method 0 2023-09-30T01:34:25.167 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:25.148+0000 7f78dd5bf700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:34:25.167 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:25.148+0000 7f78dd5bf700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:34:25.167 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:25.148+0000 7f78dd5bf700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:34:25.167 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:25.148+0000 7f78d7fff700 10 monclient(hunting): get_auth_request con 0x7f78d80a1340 auth_method 0 2023-09-30T01:34:25.167 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:25.148+0000 7f78d7fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:34:25.167 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:25.148+0000 7f78d7fff700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:34:25.168 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:25.148+0000 7f78d7fff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:34:25.168 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:25.148+0000 7f78dcdbe700 10 monclient(hunting): get_auth_request con 0x7f78d806ede0 auth_method 0 2023-09-30T01:34:25.168 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:25.148+0000 7f78dcdbe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:34:25.168 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:25.148+0000 7f78dcdbe700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:34:25.168 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:25.148+0000 7f78dcdbe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:34:25.168 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:25.148+0000 7f78dd5bf700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:34:25.168 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:25.148+0000 7f78dd5bf700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:34:25.168 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:25.148+0000 7f78dd5bf700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:34:25.168 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:25.148+0000 7f78d7fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:34:25.168 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:25.148+0000 7f78d7fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:34:25.168 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:25.148+0000 7f78d7fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:34:25.169 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:25.148+0000 7f78dcdbe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:34:25.169 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:25.148+0000 7f78dcdbe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:34:25.169 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:25.148+0000 7f78dcdbe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:34:25.169 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:25.148+0000 7f78dcdbe700 10 monclient(hunting): handle_auth_done global_id 4544 payload 824 2023-09-30T01:34:25.169 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:25.148+0000 7f78dcdbe700 1 -- 172.21.15.88:0/2585224170 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7f78d80a1340 msgr2=0x7f78d8068500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:34:25.169 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:25.148+0000 7f78dcdbe700 1 --2- 172.21.15.88:0/2585224170 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7f78d80a1340 0x7f78d8068500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:34:25.169 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:25.148+0000 7f78dcdbe700 1 -- 172.21.15.88:0/2585224170 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f78d8137570 msgr2=0x7f78d8068a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:34:25.169 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:25.148+0000 7f78dcdbe700 1 --2- 172.21.15.88:0/2585224170 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f78d8137570 0x7f78d8068a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:34:25.169 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:25.148+0000 7f78dcdbe700 10 monclient: _finish_hunting 0 2023-09-30T01:34:25.169 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:25.148+0000 7f78dcdbe700 1 monclient: found mon.noname-e 2023-09-30T01:34:25.169 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:25.148+0000 7f78dcdbe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:34:25.170 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:25.148+0000 7f78dcdbe700 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.88:3304/0 2023-09-30T01:34:25.170 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:25.148+0000 7f78dcdbe700 1 -- 172.21.15.88:0/2585224170 --> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78d0009a20 con 0x7f78d806ede0 2023-09-30T01:34:25.170 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:25.148+0000 7f78d7fff700 1 --2- 172.21.15.88:0/2585224170 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7f78d80a1340 0x7f78d8068500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:34:25.170 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:25.148+0000 7f78dd5bf700 1 --2- 172.21.15.88:0/2585224170 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f78d8137570 0x7f78d8068a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:34:25.170 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:25.148+0000 7f78dcdbe700 1 --2- 172.21.15.88:0/2585224170 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f78d806ede0 0x7f78d8067fc0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f78c000c510 tx=0x7f78c0007e70 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:34:25.170 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:25.148+0000 7f78d4ff9700 1 -- 172.21.15.88:0/2585224170 <== mon.8 v2:172.21.15.88:3304/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f78c000ce80 con 0x7f78d806ede0 2023-09-30T01:34:25.170 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:25.148+0000 7f78d4ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:34:25.170 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:25.148+0000 7f78d4ff9700 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2023-09-30T01:34:25.170 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:25.148+0000 7f78d4ff9700 10 monclient: dump: 2023-09-30T01:34:25.170 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:34:25.170 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:34:25.171 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:34:25.171 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:34:25.171 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:34:25.171 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:34:25.171 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:34:25.171 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:34:25.171 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:34:25.171 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:34:25.171 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:34:25.171 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:34:25.171 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:34:25.172 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:34:25.172 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:34:25.172 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:34:25.172 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:25.148+0000 7f78d4ff9700 10 monclient: _finish_auth 0 2023-09-30T01:34:25.172 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:25.148+0000 7f78d4ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:34:25.172 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:25.148+0000 7f78df022700 5 monclient: authenticate success, global_id 4544 2023-09-30T01:34:25.172 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:25.148+0000 7f78d4ff9700 1 -- 172.21.15.88:0/2585224170 <== mon.8 v2:172.21.15.88:3304/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f78c0011a10 con 0x7f78d806ede0 2023-09-30T01:34:25.172 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:25.148+0000 7f78d4ff9700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:34:25.172 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:25.148+0000 7f78df022700 10 monclient: _renew_subs 2023-09-30T01:34:25.172 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:25.148+0000 7f78d4ff9700 1 -- 172.21.15.88:0/2585224170 <== mon.8 v2:172.21.15.88:3304/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f78c0011bc0 con 0x7f78d806ede0 2023-09-30T01:34:25.172 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:25.148+0000 7f78df022700 10 monclient: _send_mon_message to mon.e at v2:172.21.15.88:3304/0 2023-09-30T01:34:25.173 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:25.148+0000 7f78df022700 1 -- 172.21.15.88:0/2585224170 --> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f78d811ad60 con 0x7f78d806ede0 2023-09-30T01:34:25.173 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:25.148+0000 7f78d4ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:34:25.173 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:25.148+0000 7f78d6ffd700 4 set_mon_vals no callback set 2023-09-30T01:34:25.173 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:25.148+0000 7f78d4ff9700 10 monclient: got monmap 1 from mon.e (according to old e1) 2023-09-30T01:34:25.173 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:25.148+0000 7f78d4ff9700 10 monclient: dump: 2023-09-30T01:34:25.173 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:34:25.173 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:34:25.173 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:34:25.173 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:34:25.173 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:34:25.173 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:34:25.173 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:34:25.174 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:34:25.174 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:34:25.174 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:34:25.174 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:34:25.174 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:34:25.174 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:34:25.174 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:34:25.174 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:34:25.174 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:34:25.174 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:25.148+0000 7f78df022700 10 monclient: _renew_subs 2023-09-30T01:34:25.174 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:25.148+0000 7f78df022700 10 monclient: _send_mon_message to mon.e at v2:172.21.15.88:3304/0 2023-09-30T01:34:25.175 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:25.148+0000 7f78df022700 1 -- 172.21.15.88:0/2585224170 --> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f78d8064be0 con 0x7f78d806ede0 2023-09-30T01:34:25.175 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:25.152+0000 7f78d4ff9700 1 -- 172.21.15.88:0/2585224170 <== mon.8 v2:172.21.15.88:3304/0 4 ==== mgrmap(e 5) v1 ==== 40144+0+0 (secure 0 0 0) 0x7f78c0011d20 con 0x7f78d806ede0 2023-09-30T01:34:25.175 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:25.152+0000 7f78df022700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:34:25.175 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:25.152+0000 7f78df022700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:34:25.175 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:25.152+0000 7f78df022700 10 monclient: _send_mon_message to mon.e at v2:172.21.15.88:3304/0 2023-09-30T01:34:25.175 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:25.152+0000 7f78df022700 1 -- 172.21.15.88:0/2585224170 --> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f78d811c170 con 0x7f78d806ede0 2023-09-30T01:34:25.175 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:25.152+0000 7f78d4ff9700 1 --2- 172.21.15.88:0/2585224170 >> v2:172.21.15.107:6812/29926 conn(0x7f78b8033360 0x7f78b8035810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:34:25.175 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:25.152+0000 7f78d4ff9700 1 -- 172.21.15.88:0/2585224170 <== mon.8 v2:172.21.15.88:3304/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 3502+0+0 (secure 0 0 0) 0x7f78c0042320 con 0x7f78d806ede0 2023-09-30T01:34:25.175 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:25.152+0000 7f78d7fff700 1 --2- 172.21.15.88:0/2585224170 >> v2:172.21.15.107:6812/29926 conn(0x7f78b8033360 0x7f78b8035810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:34:25.175 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:25.152+0000 7f78d7fff700 10 monclient: get_auth_request con 0x7f78b8033360 auth_method 0 2023-09-30T01:34:25.175 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:25.152+0000 7f78d7fff700 1 --2- 172.21.15.88:0/2585224170 >> v2:172.21.15.107:6812/29926 conn(0x7f78b8033360 0x7f78b8035810 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f78c8007fd0 tx=0x7f78c8009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:34:25.176 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:25.160+0000 7f78d4ff9700 1 -- 172.21.15.88:0/2585224170 <== mon.8 v2:172.21.15.88:3304/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f78c0011820 con 0x7f78d806ede0 2023-09-30T01:34:25.176 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:25.160+0000 7f78d4ff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:34:25.176 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:25.160+0000 7f78d4ff9700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:34:25.360 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:25.356+0000 7f78df022700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T01:34:25.360 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:25.356+0000 7f78df022700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:34:25.360 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:25.356+0000 7f78df022700 10 monclient: _send_mon_message to mon.e at v2:172.21.15.88:3304/0 2023-09-30T01:34:25.360 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:25.356+0000 7f78df022700 1 -- 172.21.15.88:0/2585224170 --> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f78d8071960 con 0x7f78d806ede0 2023-09-30T01:34:25.360 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:25.356+0000 7f78d4ff9700 1 -- 172.21.15.88:0/2585224170 <== mon.8 v2:172.21.15.88:3304/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f78c0017030 con 0x7f78d806ede0 2023-09-30T01:34:25.361 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:25.356+0000 7f78d4ff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:34:25.361 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:25.356+0000 7f78d4ff9700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:34:25.361 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-09-30T01:34:25.361 INFO:teuthology.orchestra.run.smithi088.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:34:25.363 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:25.356+0000 7f78df022700 1 -- 172.21.15.88:0/2585224170 >> v2:172.21.15.107:6812/29926 conn(0x7f78b8033360 msgr2=0x7f78b8035810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:34:25.363 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:25.356+0000 7f78df022700 1 --2- 172.21.15.88:0/2585224170 >> v2:172.21.15.107:6812/29926 conn(0x7f78b8033360 0x7f78b8035810 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f78c8007fd0 tx=0x7f78c8009040 comp rx=0 tx=0).stop 2023-09-30T01:34:25.363 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:25.356+0000 7f78df022700 10 monclient: shutdown 2023-09-30T01:34:25.363 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:25.356+0000 7f78df022700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:34:25.363 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:25.356+0000 7f78df022700 1 -- 172.21.15.88:0/2585224170 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f78d806ede0 msgr2=0x7f78d8067fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:34:25.363 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:25.356+0000 7f78df022700 1 --2- 172.21.15.88:0/2585224170 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f78d806ede0 0x7f78d8067fc0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f78c000c510 tx=0x7f78c0007e70 comp rx=0 tx=0).stop 2023-09-30T01:34:25.363 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:25.356+0000 7f78df022700 1 -- 172.21.15.88:0/2585224170 shutdown_connections 2023-09-30T01:34:25.363 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:25.356+0000 7f78df022700 1 --2- 172.21.15.88:0/2585224170 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f78d8137570 0x7f78d8068a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:34:25.364 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:25.356+0000 7f78df022700 1 --2- 172.21.15.88:0/2585224170 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7f78d80a1340 0x7f78d8068500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:34:25.364 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:25.356+0000 7f78df022700 1 --2- 172.21.15.88:0/2585224170 >> v2:172.21.15.107:6812/29926 conn(0x7f78b8033360 0x7f78b8035810 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:34:25.364 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:25.356+0000 7f78df022700 1 --2- 172.21.15.88:0/2585224170 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f78d806ede0 0x7f78d8067fc0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:34:25.364 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:25.356+0000 7f78df022700 1 -- 172.21.15.88:0/2585224170 >> 172.21.15.88:0/2585224170 conn(0x7f78d8097a40 msgr2=0x7f78d80a3cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:34:25.364 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:25.356+0000 7f78df022700 1 -- 172.21.15.88:0/2585224170 shutdown_connections 2023-09-30T01:34:25.364 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:25.356+0000 7f78df022700 1 -- 172.21.15.88:0/2585224170 wait complete. 2023-09-30T01:34:28.376 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:34:28.487 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:28.480+0000 7f7e32605700 10 monclient: get_monmap_and_config 2023-09-30T01:34:28.503 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:28.480+0000 7f7e32605700 10 monclient: build_initial_monmap 2023-09-30T01:34:28.504 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:28.484+0000 7f7e32605700 10 monclient: monmap: 2023-09-30T01:34:28.504 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2023-09-30T01:34:28.504 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:34:28.504 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:34:28.487823+0000 2023-09-30T01:34:28.504 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:34:28.487823+0000 2023-09-30T01:34:28.504 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2023-09-30T01:34:28.504 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:34:28.504 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a-legacy 2023-09-30T01:34:28.504 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.107:6789/0 mon.noname-f-legacy 2023-09-30T01:34:28.504 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.noname-b 2023-09-30T01:34:28.505 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.noname-g 2023-09-30T01:34:28.505 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.noname-c 2023-09-30T01:34:28.505 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.noname-h 2023-09-30T01:34:28.505 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.noname-d 2023-09-30T01:34:28.505 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.noname-i 2023-09-30T01:34:28.505 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.noname-e 2023-09-30T01:34:28.505 INFO:teuthology.orchestra.run.smithi088.stderr:9: v2:172.21.15.88:3300/0 mon.noname-a 2023-09-30T01:34:28.505 INFO:teuthology.orchestra.run.smithi088.stderr:10: v2:172.21.15.107:3300/0 mon.noname-f 2023-09-30T01:34:28.505 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:34:28.512 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:28.484+0000 7f7e32605700 1 Processor -- start 2023-09-30T01:34:28.512 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:28.484+0000 7f7e32605700 1 -- start start 2023-09-30T01:34:28.512 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:28.484+0000 7f7e32605700 10 monclient: init 2023-09-30T01:34:28.512 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:28.484+0000 7f7e32605700 10 monclient: _reopen_session rank -1 2023-09-30T01:34:28.513 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:28.484+0000 7f7e32605700 10 monclient: _add_conns ranks=[6,7,1,4,0,3,8,2,10,5,9] 2023-09-30T01:34:28.513 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:28.484+0000 7f7e32605700 1 --2- >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7f7e2c12f7c0 0x7f7e2c12fe70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:34:28.513 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:28.484+0000 7f7e32605700 10 monclient(hunting): picked mon.noname-d con 0x7f7e2c12f7c0 addr [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] 2023-09-30T01:34:28.513 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:28.484+0000 7f7e32605700 1 --2- >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7f7e2c06e610 0x7f7e2c06ea20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:34:28.513 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:28.484+0000 7f7e32605700 10 monclient(hunting): picked mon.noname-i con 0x7f7e2c06e610 addr [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] 2023-09-30T01:34:28.513 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:28.484+0000 7f7e32605700 10 monclient(hunting): picked mon.noname-f-legacy con 0x7f7e2c09d580 addr v1:172.21.15.107:6789/0 2023-09-30T01:34:28.513 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:28.484+0000 7f7e32605700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e2c04bb20 con 0x7f7e2c09d580 2023-09-30T01:34:28.513 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:28.484+0000 7f7e32605700 10 monclient(hunting): start opening mon connection 2023-09-30T01:34:28.513 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:28.484+0000 7f7e32605700 1 -- --> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] -- mon_getmap magic: 0 v1 -- 0x7f7e2c06bdf0 con 0x7f7e2c12f7c0 2023-09-30T01:34:28.513 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:28.484+0000 7f7e32605700 10 monclient(hunting): start opening mon connection 2023-09-30T01:34:28.513 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:28.484+0000 7f7e32605700 1 -- --> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] -- mon_getmap magic: 0 v1 -- 0x7f7e2c06a4c0 con 0x7f7e2c06e610 2023-09-30T01:34:28.513 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:28.484+0000 7f7e32605700 10 monclient(hunting): _renew_subs 2023-09-30T01:34:28.514 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:28.484+0000 7f7e32605700 10 monclient(hunting): authenticate will time out at 940.527044s 2023-09-30T01:34:28.514 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:28.484+0000 7f7e31603700 1 --2- >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7f7e2c12f7c0 0x7f7e2c12fe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:34:28.514 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:28.484+0000 7f7e31603700 1 --2- >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7f7e2c12f7c0 0x7f7e2c12fe70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3303/0 says I am v2:172.21.15.88:54988/0 (socket says 172.21.15.88:54988) 2023-09-30T01:34:28.514 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:28.484+0000 7f7e31603700 1 -- 172.21.15.88:0/2403683999 learned_addr learned my addr 172.21.15.88:0/2403683999 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-09-30T01:34:28.514 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:28.484+0000 7f7e31603700 10 monclient(hunting): get_auth_request con 0x7f7e2c12f7c0 auth_method 0 2023-09-30T01:34:28.514 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:28.484+0000 7f7e31603700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:34:28.514 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:28.484+0000 7f7e31603700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:34:28.514 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:28.484+0000 7f7e31603700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:34:28.514 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:28.484+0000 7f7e30e02700 1 --2- 172.21.15.88:0/2403683999 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7f7e2c06e610 0x7f7e2c06ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:34:28.514 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:28.484+0000 7f7e31603700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:34:28.514 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:28.484+0000 7f7e31603700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:34:28.515 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:28.484+0000 7f7e31603700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:34:28.515 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:28.484+0000 7f7e30e02700 10 monclient(hunting): get_auth_request con 0x7f7e2c06e610 auth_method 0 2023-09-30T01:34:28.515 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:28.484+0000 7f7e30e02700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:34:28.515 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:28.484+0000 7f7e30e02700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:34:28.515 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:28.484+0000 7f7e30e02700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:34:28.515 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:28.484+0000 7f7e30e02700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:34:28.515 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:28.484+0000 7f7e30e02700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:34:28.515 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:28.484+0000 7f7e30e02700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:34:28.515 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:28.484+0000 7f7e31603700 10 monclient(hunting): handle_auth_done global_id 4407 payload 274 2023-09-30T01:34:28.515 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:28.484+0000 7f7e31603700 1 -- 172.21.15.88:0/2403683999 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7f7e2c06e610 msgr2=0x7f7e2c06ea20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:34:28.515 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:28.484+0000 7f7e31603700 1 --2- 172.21.15.88:0/2403683999 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7f7e2c06e610 0x7f7e2c06ea20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:34:28.515 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:28.484+0000 7f7e31603700 1 -- 172.21.15.88:0/2403683999 >> v1:172.21.15.107:6789/0 conn(0x7f7e2c09d580 legacy=0x7f7e2c09d9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:34:28.516 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:28.484+0000 7f7e31603700 10 monclient: _finish_hunting 0 2023-09-30T01:34:28.516 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:28.484+0000 7f7e31603700 1 monclient: found mon.noname-d 2023-09-30T01:34:28.516 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:28.484+0000 7f7e31603700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:34:28.516 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:28.484+0000 7f7e31603700 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.88:3303/0 2023-09-30T01:34:28.516 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:28.484+0000 7f7e31603700 1 -- 172.21.15.88:0/2403683999 --> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e2c12fb90 con 0x7f7e2c12f7c0 2023-09-30T01:34:28.516 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:28.488+0000 7f7e31603700 1 --2- 172.21.15.88:0/2403683999 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7f7e2c12f7c0 0x7f7e2c12fe70 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f7e1c00abb0 tx=0x7f7e1c00a3f0 comp rx=0 tx=0).ready entity=mon.6 client_cookie=dc0e0cddcca8c462 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:34:28.516 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:28.488+0000 7f7e23fff700 1 -- 172.21.15.88:0/2403683999 <== mon.6 v2:172.21.15.88:3303/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f7e1c00ce80 con 0x7f7e2c12f7c0 2023-09-30T01:34:28.516 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:28.488+0000 7f7e23fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:34:28.516 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:28.488+0000 7f7e23fff700 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2023-09-30T01:34:28.516 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:28.488+0000 7f7e23fff700 10 monclient: dump: 2023-09-30T01:34:28.516 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:34:28.517 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:34:28.517 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:34:28.517 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:34:28.517 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:34:28.517 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:34:28.517 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:34:28.517 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:34:28.517 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:34:28.517 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:34:28.517 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:34:28.517 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:34:28.517 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:34:28.518 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:34:28.518 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:34:28.518 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:34:28.518 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:28.488+0000 7f7e23fff700 10 monclient: _finish_auth 0 2023-09-30T01:34:28.518 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:28.488+0000 7f7e23fff700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:34:28.518 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:28.488+0000 7f7e32605700 5 monclient: authenticate success, global_id 4407 2023-09-30T01:34:28.518 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:28.488+0000 7f7e23fff700 1 -- 172.21.15.88:0/2403683999 <== mon.6 v2:172.21.15.88:3303/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e1c018400 con 0x7f7e2c12f7c0 2023-09-30T01:34:28.518 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:28.488+0000 7f7e32605700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:34:28.518 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:28.488+0000 7f7e23fff700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:34:28.518 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:28.488+0000 7f7e32605700 10 monclient: get_monmap_and_config success 2023-09-30T01:34:28.518 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:28.488+0000 7f7e32605700 4 set_mon_vals no callback set 2023-09-30T01:34:28.519 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:28.488+0000 7f7e23fff700 1 -- 172.21.15.88:0/2403683999 <== mon.6 v2:172.21.15.88:3303/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f7e1c0189f0 con 0x7f7e2c12f7c0 2023-09-30T01:34:28.519 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:28.488+0000 7f7e23fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:34:28.519 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:28.488+0000 7f7e23fff700 10 monclient: got monmap 1 from mon.d (according to old e1) 2023-09-30T01:34:28.519 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:28.488+0000 7f7e23fff700 10 monclient: dump: 2023-09-30T01:34:28.519 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:34:28.519 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:34:28.519 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:34:28.519 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:34:28.519 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:34:28.519 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:34:28.519 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:34:28.519 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:34:28.520 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:34:28.520 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:34:28.520 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:34:28.520 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:34:28.520 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:34:28.520 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:34:28.520 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:34:28.520 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:34:28.520 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:28.488+0000 7f7e32605700 10 monclient: shutdown 2023-09-30T01:34:28.520 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:28.488+0000 7f7e32605700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:34:28.520 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:28.488+0000 7f7e32605700 1 -- 172.21.15.88:0/2403683999 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7f7e2c12f7c0 msgr2=0x7f7e2c12fe70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:34:28.521 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:28.488+0000 7f7e32605700 1 --2- 172.21.15.88:0/2403683999 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7f7e2c12f7c0 0x7f7e2c12fe70 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f7e1c00abb0 tx=0x7f7e1c00a3f0 comp rx=0 tx=0).stop 2023-09-30T01:34:28.521 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:28.488+0000 7f7e32605700 1 -- 172.21.15.88:0/2403683999 shutdown_connections 2023-09-30T01:34:28.521 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:28.488+0000 7f7e32605700 1 --2- 172.21.15.88:0/2403683999 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7f7e2c06e610 0x7f7e2c06ea20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:34:28.521 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:28.488+0000 7f7e32605700 1 --2- 172.21.15.88:0/2403683999 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7f7e2c12f7c0 0x7f7e2c12fe70 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:34:28.521 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:28.488+0000 7f7e32605700 1 -- 172.21.15.88:0/2403683999 >> 172.21.15.88:0/2403683999 conn(0x7f7e2c063610 msgr2=0x7f7e2c0a2b90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:34:28.521 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:28.488+0000 7f7e32605700 1 -- 172.21.15.88:0/2403683999 shutdown_connections 2023-09-30T01:34:28.521 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:28.488+0000 7f7e32605700 1 -- 172.21.15.88:0/2403683999 wait complete. 2023-09-30T01:34:28.521 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:28.488+0000 7f7e32605700 10 monclient: build_initial_monmap 2023-09-30T01:34:28.521 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:28.488+0000 7f7e32605700 10 monclient: monmap: 2023-09-30T01:34:28.521 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2023-09-30T01:34:28.521 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:34:28.521 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 0.000000 2023-09-30T01:34:28.522 INFO:teuthology.orchestra.run.smithi088.stderr:created 0.000000 2023-09-30T01:34:28.522 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2023-09-30T01:34:28.522 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 1 2023-09-30T01:34:28.522 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.noname-a 2023-09-30T01:34:28.522 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.noname-f 2023-09-30T01:34:28.522 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.noname-b 2023-09-30T01:34:28.522 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.noname-g 2023-09-30T01:34:28.522 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.noname-c 2023-09-30T01:34:28.522 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.noname-h 2023-09-30T01:34:28.522 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.noname-d 2023-09-30T01:34:28.522 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.noname-i 2023-09-30T01:34:28.522 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.noname-e 2023-09-30T01:34:28.523 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:34:28.523 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:28.488+0000 7f7e32605700 1 Processor -- start 2023-09-30T01:34:28.523 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:28.488+0000 7f7e32605700 1 -- start start 2023-09-30T01:34:28.523 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:28.488+0000 7f7e32605700 10 monclient: init 2023-09-30T01:34:28.523 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:28.488+0000 7f7e32605700 10 monclient: _reopen_session rank -1 2023-09-30T01:34:28.523 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:28.488+0000 7f7e32605700 10 monclient: _add_conns ranks=[7,5,1,6,2,4,8,0,3] 2023-09-30T01:34:28.523 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:28.488+0000 7f7e32605700 1 --2- >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7f7e2c06e610 0x7f7e2c12cc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:34:28.523 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:28.488+0000 7f7e32605700 10 monclient(hunting): picked mon.noname-i con 0x7f7e2c06e610 addr [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] 2023-09-30T01:34:28.523 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:28.488+0000 7f7e32605700 1 --2- >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7f7e2c09d580 0x7f7e2c097a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:34:28.523 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:28.488+0000 7f7e32605700 10 monclient(hunting): picked mon.noname-h con 0x7f7e2c09d580 addr [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] 2023-09-30T01:34:28.523 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:28.488+0000 7f7e32605700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7e2c12f7c0 0x7f7e2c097f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:34:28.523 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:28.488+0000 7f7e32605700 10 monclient(hunting): picked mon.noname-f con 0x7f7e2c12f7c0 addr [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] 2023-09-30T01:34:28.524 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:28.488+0000 7f7e32605700 10 monclient(hunting): start opening mon connection 2023-09-30T01:34:28.524 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:28.488+0000 7f7e32605700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e2c11ebf0 con 0x7f7e2c12f7c0 2023-09-30T01:34:28.524 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:28.488+0000 7f7e32605700 10 monclient(hunting): start opening mon connection 2023-09-30T01:34:28.524 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:28.488+0000 7f7e32605700 1 -- --> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] -- mon_getmap magic: 0 v1 -- 0x7f7e2c06b200 con 0x7f7e2c09d580 2023-09-30T01:34:28.524 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:28.488+0000 7f7e32605700 10 monclient(hunting): start opening mon connection 2023-09-30T01:34:28.524 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:28.488+0000 7f7e32605700 1 -- --> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] -- mon_getmap magic: 0 v1 -- 0x7f7e2c06ac80 con 0x7f7e2c06e610 2023-09-30T01:34:28.524 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:28.488+0000 7f7e32605700 10 monclient(hunting): _renew_subs 2023-09-30T01:34:28.524 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:28.488+0000 7f7e32605700 10 monclient(hunting): authenticate will time out at 940.531136s 2023-09-30T01:34:28.524 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:28.488+0000 7f7e31603700 1 --2- >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7f7e2c06e610 0x7f7e2c12cc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:34:28.524 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:28.488+0000 7f7e31e04700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7e2c12f7c0 0x7f7e2c097f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:34:28.524 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:28.488+0000 7f7e31e04700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7e2c12f7c0 0x7f7e2c097f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.88:37814/0 (socket says 172.21.15.88:37814) 2023-09-30T01:34:28.524 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:28.488+0000 7f7e31e04700 1 -- 172.21.15.88:0/2593692923 learned_addr learned my addr 172.21.15.88:0/2593692923 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-09-30T01:34:28.525 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:28.488+0000 7f7e31e04700 10 monclient(hunting): get_auth_request con 0x7f7e2c12f7c0 auth_method 0 2023-09-30T01:34:28.525 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:28.488+0000 7f7e31e04700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:34:28.525 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:28.488+0000 7f7e31e04700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:34:28.525 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:28.488+0000 7f7e31603700 1 --2- >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7f7e2c06e610 0x7f7e2c12cc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3303/0 says I am v2:172.21.15.88:53948/0 (socket says 172.21.15.88:53948) 2023-09-30T01:34:28.525 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:28.488+0000 7f7e30e02700 1 --2- >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7f7e2c09d580 0x7f7e2c097a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:34:28.525 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:28.488+0000 7f7e31e04700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:34:28.525 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:28.488+0000 7f7e31603700 10 monclient(hunting): get_auth_request con 0x7f7e2c06e610 auth_method 0 2023-09-30T01:34:28.525 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:28.488+0000 7f7e31603700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:34:28.525 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:28.488+0000 7f7e31603700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:34:28.525 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:28.488+0000 7f7e31603700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:34:28.525 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:28.488+0000 7f7e30e02700 10 monclient(hunting): get_auth_request con 0x7f7e2c09d580 auth_method 0 2023-09-30T01:34:28.526 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:28.488+0000 7f7e30e02700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:34:28.526 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:28.488+0000 7f7e30e02700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:34:28.526 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:28.488+0000 7f7e30e02700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:34:28.526 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:28.488+0000 7f7e31e04700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:34:28.526 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:28.488+0000 7f7e31e04700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:34:28.526 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:28.488+0000 7f7e31e04700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:34:28.526 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:28.488+0000 7f7e31603700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:34:28.526 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:28.488+0000 7f7e31603700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:34:28.526 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:28.488+0000 7f7e31603700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:34:28.526 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:28.488+0000 7f7e30e02700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:34:28.526 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:28.488+0000 7f7e30e02700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:34:28.526 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:28.488+0000 7f7e30e02700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:34:28.527 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:28.492+0000 7f7e31603700 10 monclient(hunting): handle_auth_done global_id 4516 payload 824 2023-09-30T01:34:28.527 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:28.492+0000 7f7e31603700 1 -- 172.21.15.88:0/2593692923 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7f7e2c09d580 msgr2=0x7f7e2c097a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:34:28.527 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:28.492+0000 7f7e31603700 1 --2- 172.21.15.88:0/2593692923 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7f7e2c09d580 0x7f7e2c097a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:34:28.527 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:28.492+0000 7f7e31603700 1 -- 172.21.15.88:0/2593692923 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7e2c12f7c0 msgr2=0x7f7e2c097f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:34:28.527 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:28.492+0000 7f7e31603700 1 --2- 172.21.15.88:0/2593692923 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7e2c12f7c0 0x7f7e2c097f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:34:28.527 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:28.492+0000 7f7e31603700 10 monclient: _finish_hunting 0 2023-09-30T01:34:28.527 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:28.492+0000 7f7e31603700 1 monclient: found mon.noname-i 2023-09-30T01:34:28.527 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:28.492+0000 7f7e31603700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:34:28.527 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:28.492+0000 7f7e31603700 10 monclient: _send_mon_message to mon.noname-i at v2:172.21.15.107:3303/0 2023-09-30T01:34:28.527 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:28.492+0000 7f7e31603700 1 -- 172.21.15.88:0/2593692923 --> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e1c00a690 con 0x7f7e2c06e610 2023-09-30T01:34:28.527 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:28.492+0000 7f7e31e04700 1 --2- 172.21.15.88:0/2593692923 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7e2c12f7c0 0x7f7e2c097f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:34:28.527 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:28.492+0000 7f7e31603700 1 --2- 172.21.15.88:0/2593692923 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7f7e2c06e610 0x7f7e2c12cc80 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f7e1c014000 tx=0x7f7e1c01a000 comp rx=0 tx=0).ready entity=mon.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:34:28.527 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:28.492+0000 7f7e217fa700 1 -- 172.21.15.88:0/2593692923 <== mon.7 v2:172.21.15.107:3303/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f7e1c006a90 con 0x7f7e2c06e610 2023-09-30T01:34:28.528 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:28.492+0000 7f7e217fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:34:28.528 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:28.492+0000 7f7e217fa700 10 monclient: got monmap 1 from mon.noname-i (according to old e1) 2023-09-30T01:34:28.528 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:28.492+0000 7f7e217fa700 10 monclient: dump: 2023-09-30T01:34:28.528 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:34:28.528 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:34:28.528 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:34:28.528 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:34:28.528 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:34:28.528 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:34:28.528 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:34:28.528 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:34:28.528 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:34:28.529 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:34:28.529 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:34:28.529 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:34:28.529 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:34:28.529 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:34:28.529 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:34:28.529 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:34:28.529 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:28.492+0000 7f7e217fa700 10 monclient: _finish_auth 0 2023-09-30T01:34:28.529 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:28.492+0000 7f7e217fa700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:34:28.529 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:28.492+0000 7f7e217fa700 1 -- 172.21.15.88:0/2593692923 <== mon.7 v2:172.21.15.107:3303/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e1c010870 con 0x7f7e2c06e610 2023-09-30T01:34:28.529 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:28.492+0000 7f7e217fa700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:34:28.529 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:28.492+0000 7f7e217fa700 1 -- 172.21.15.88:0/2593692923 <== mon.7 v2:172.21.15.107:3303/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f7e1c010e60 con 0x7f7e2c06e610 2023-09-30T01:34:28.530 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:28.492+0000 7f7e217fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:34:28.530 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:28.492+0000 7f7e217fa700 10 monclient: got monmap 1 from mon.i (according to old e1) 2023-09-30T01:34:28.530 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:28.492+0000 7f7e217fa700 10 monclient: dump: 2023-09-30T01:34:28.530 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:34:28.530 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:34:28.530 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:34:28.530 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:34:28.530 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:34:28.530 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:34:28.530 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:34:28.530 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:34:28.530 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:34:28.530 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:34:28.531 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:34:28.531 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:34:28.531 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:34:28.531 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:34:28.531 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:34:28.531 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:34:28.531 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:28.492+0000 7f7e32605700 5 monclient: authenticate success, global_id 4516 2023-09-30T01:34:28.531 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:28.492+0000 7f7e32605700 10 monclient: _renew_subs 2023-09-30T01:34:28.531 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:28.492+0000 7f7e22ffd700 4 set_mon_vals no callback set 2023-09-30T01:34:28.531 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:28.492+0000 7f7e32605700 10 monclient: _send_mon_message to mon.i at v2:172.21.15.107:3303/0 2023-09-30T01:34:28.531 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:28.492+0000 7f7e32605700 1 -- 172.21.15.88:0/2593692923 --> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e2c11fcb0 con 0x7f7e2c06e610 2023-09-30T01:34:28.531 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:28.492+0000 7f7e32605700 10 monclient: _renew_subs 2023-09-30T01:34:28.531 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:28.492+0000 7f7e32605700 10 monclient: _send_mon_message to mon.i at v2:172.21.15.107:3303/0 2023-09-30T01:34:28.532 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:28.492+0000 7f7e32605700 1 -- 172.21.15.88:0/2593692923 --> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e2c120970 con 0x7f7e2c06e610 2023-09-30T01:34:28.532 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:28.492+0000 7f7e32605700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:34:28.532 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:28.492+0000 7f7e217fa700 1 -- 172.21.15.88:0/2593692923 <== mon.7 v2:172.21.15.107:3303/0 4 ==== mgrmap(e 5) v1 ==== 40144+0+0 (secure 0 0 0) 0x7f7e1c006c30 con 0x7f7e2c06e610 2023-09-30T01:34:28.532 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:28.492+0000 7f7e32605700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:34:28.532 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:28.492+0000 7f7e32605700 10 monclient: _send_mon_message to mon.i at v2:172.21.15.107:3303/0 2023-09-30T01:34:28.532 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:28.492+0000 7f7e32605700 1 -- 172.21.15.88:0/2593692923 --> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e2c11fe40 con 0x7f7e2c06e610 2023-09-30T01:34:28.532 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:28.492+0000 7f7e217fa700 1 --2- 172.21.15.88:0/2593692923 >> v2:172.21.15.107:6812/29926 conn(0x7f7e10033390 0x7f7e10035840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:34:28.532 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:28.492+0000 7f7e217fa700 1 -- 172.21.15.88:0/2593692923 <== mon.7 v2:172.21.15.107:3303/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 3502+0+0 (secure 0 0 0) 0x7f7e1c044810 con 0x7f7e2c06e610 2023-09-30T01:34:28.532 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:28.492+0000 7f7e30e02700 1 --2- 172.21.15.88:0/2593692923 >> v2:172.21.15.107:6812/29926 conn(0x7f7e10033390 0x7f7e10035840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:34:28.532 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:28.492+0000 7f7e30e02700 10 monclient: get_auth_request con 0x7f7e10033390 auth_method 0 2023-09-30T01:34:28.532 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:28.496+0000 7f7e30e02700 1 --2- 172.21.15.88:0/2593692923 >> v2:172.21.15.107:6812/29926 conn(0x7f7e10033390 0x7f7e10035840 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f7e2c0a37a0 tx=0x7f7e280091a0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:34:28.532 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:28.504+0000 7f7e217fa700 1 -- 172.21.15.88:0/2593692923 <== mon.7 v2:172.21.15.107:3303/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f7e1c00f960 con 0x7f7e2c06e610 2023-09-30T01:34:28.533 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:28.504+0000 7f7e217fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:34:28.533 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:28.504+0000 7f7e217fa700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:34:28.705 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:28.700+0000 7f7e32605700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T01:34:28.705 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:28.700+0000 7f7e32605700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:34:28.705 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:28.700+0000 7f7e32605700 10 monclient: _send_mon_message to mon.i at v2:172.21.15.107:3303/0 2023-09-30T01:34:28.705 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:28.700+0000 7f7e32605700 1 -- 172.21.15.88:0/2593692923 --> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f7e2c071960 con 0x7f7e2c06e610 2023-09-30T01:34:28.705 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:28.700+0000 7f7e217fa700 1 -- 172.21.15.88:0/2593692923 <== mon.7 v2:172.21.15.107:3303/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f7e1c017360 con 0x7f7e2c06e610 2023-09-30T01:34:28.706 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:28.700+0000 7f7e217fa700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:34:28.706 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:28.700+0000 7f7e217fa700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:34:28.706 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-09-30T01:34:28.706 INFO:teuthology.orchestra.run.smithi088.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:34:28.710 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:28.704+0000 7f7e32605700 1 -- 172.21.15.88:0/2593692923 >> v2:172.21.15.107:6812/29926 conn(0x7f7e10033390 msgr2=0x7f7e10035840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:34:28.710 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:28.704+0000 7f7e32605700 1 --2- 172.21.15.88:0/2593692923 >> v2:172.21.15.107:6812/29926 conn(0x7f7e10033390 0x7f7e10035840 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f7e2c0a37a0 tx=0x7f7e280091a0 comp rx=0 tx=0).stop 2023-09-30T01:34:28.710 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:28.704+0000 7f7e32605700 10 monclient: shutdown 2023-09-30T01:34:28.710 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:28.704+0000 7f7e32605700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:34:28.710 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:28.704+0000 7f7e32605700 1 -- 172.21.15.88:0/2593692923 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7f7e2c06e610 msgr2=0x7f7e2c12cc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:34:28.710 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:28.704+0000 7f7e32605700 1 --2- 172.21.15.88:0/2593692923 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7f7e2c06e610 0x7f7e2c12cc80 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f7e1c014000 tx=0x7f7e1c01a000 comp rx=0 tx=0).stop 2023-09-30T01:34:28.711 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:28.704+0000 7f7e32605700 1 -- 172.21.15.88:0/2593692923 shutdown_connections 2023-09-30T01:34:28.711 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:28.704+0000 7f7e32605700 1 --2- 172.21.15.88:0/2593692923 >> v2:172.21.15.107:6812/29926 conn(0x7f7e10033390 0x7f7e10035840 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:34:28.711 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:28.704+0000 7f7e32605700 1 --2- 172.21.15.88:0/2593692923 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7e2c12f7c0 0x7f7e2c097f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:34:28.711 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:28.704+0000 7f7e32605700 1 --2- 172.21.15.88:0/2593692923 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7f7e2c09d580 0x7f7e2c097a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:34:28.711 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:28.704+0000 7f7e32605700 1 --2- 172.21.15.88:0/2593692923 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7f7e2c06e610 0x7f7e2c12cc80 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:34:28.711 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:28.704+0000 7f7e32605700 1 -- 172.21.15.88:0/2593692923 >> 172.21.15.88:0/2593692923 conn(0x7f7e2c063610 msgr2=0x7f7e2c12fe70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:34:28.711 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:28.704+0000 7f7e32605700 1 -- 172.21.15.88:0/2593692923 shutdown_connections 2023-09-30T01:34:28.711 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:28.704+0000 7f7e32605700 1 -- 172.21.15.88:0/2593692923 wait complete. 2023-09-30T01:34:31.726 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:34:31.835 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:31.828+0000 7fed5184d700 10 monclient: get_monmap_and_config 2023-09-30T01:34:31.835 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:31.828+0000 7fed5184d700 10 monclient: build_initial_monmap 2023-09-30T01:34:31.835 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:31.828+0000 7fed5184d700 10 monclient: monmap: 2023-09-30T01:34:31.835 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2023-09-30T01:34:31.835 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:34:31.835 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:34:31.835054+0000 2023-09-30T01:34:31.835 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:34:31.835054+0000 2023-09-30T01:34:31.835 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2023-09-30T01:34:31.835 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:34:31.835 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a-legacy 2023-09-30T01:34:31.835 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.107:6789/0 mon.noname-f-legacy 2023-09-30T01:34:31.836 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.noname-b 2023-09-30T01:34:31.836 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.noname-g 2023-09-30T01:34:31.836 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.noname-c 2023-09-30T01:34:31.836 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.noname-h 2023-09-30T01:34:31.836 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.noname-d 2023-09-30T01:34:31.836 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.noname-i 2023-09-30T01:34:31.836 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.noname-e 2023-09-30T01:34:31.836 INFO:teuthology.orchestra.run.smithi088.stderr:9: v2:172.21.15.88:3300/0 mon.noname-a 2023-09-30T01:34:31.836 INFO:teuthology.orchestra.run.smithi088.stderr:10: v2:172.21.15.107:3300/0 mon.noname-f 2023-09-30T01:34:31.836 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:34:31.845 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:31.832+0000 7fed5184d700 1 Processor -- start 2023-09-30T01:34:31.845 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:31.832+0000 7fed5184d700 1 -- start start 2023-09-30T01:34:31.845 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:31.832+0000 7fed5184d700 10 monclient: init 2023-09-30T01:34:31.845 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:31.832+0000 7fed5184d700 10 monclient: _reopen_session rank -1 2023-09-30T01:34:31.845 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:31.832+0000 7fed5184d700 10 monclient: _add_conns ranks=[6,5,2,7,10,8,0,1,9,3,4] 2023-09-30T01:34:31.845 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:31.832+0000 7fed5184d700 1 --2- >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7fed4c0a3270 0x7fed4c0a0160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:34:31.845 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:31.832+0000 7fed5184d700 10 monclient(hunting): picked mon.noname-d con 0x7fed4c0a3270 addr [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] 2023-09-30T01:34:31.846 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:31.832+0000 7fed5184d700 1 --2- >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7fed4c0a4b00 0x7fed4c0a06a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:34:31.846 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:31.832+0000 7fed5184d700 10 monclient(hunting): picked mon.noname-h con 0x7fed4c0a4b00 addr [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] 2023-09-30T01:34:31.846 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:31.832+0000 7fed5184d700 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fed4c1333c0 0x7fed4c1337b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:34:31.846 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:31.832+0000 7fed5184d700 10 monclient(hunting): picked mon.noname-b con 0x7fed4c1333c0 addr [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] 2023-09-30T01:34:31.846 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:31.832+0000 7fed5184d700 10 monclient(hunting): start opening mon connection 2023-09-30T01:34:31.846 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:31.832+0000 7fed5184d700 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed4c0a4ed0 con 0x7fed4c1333c0 2023-09-30T01:34:31.846 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:31.832+0000 7fed5184d700 10 monclient(hunting): start opening mon connection 2023-09-30T01:34:31.846 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:31.832+0000 7fed5184d700 1 -- --> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] -- mon_getmap magic: 0 v1 -- 0x7fed4c0a6330 con 0x7fed4c0a4b00 2023-09-30T01:34:31.846 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:31.832+0000 7fed5184d700 10 monclient(hunting): start opening mon connection 2023-09-30T01:34:31.846 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:31.832+0000 7fed5184d700 1 -- --> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] -- mon_getmap magic: 0 v1 -- 0x7fed4c098610 con 0x7fed4c0a3270 2023-09-30T01:34:31.846 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:31.832+0000 7fed5184d700 10 monclient(hunting): _renew_subs 2023-09-30T01:34:31.846 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:31.832+0000 7fed5184d700 10 monclient(hunting): authenticate will time out at 943.872952s 2023-09-30T01:34:31.847 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:31.832+0000 7fed4affd700 1 --2- >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7fed4c0a3270 0x7fed4c0a0160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:34:31.847 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:31.832+0000 7fed4b7fe700 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fed4c1333c0 0x7fed4c1337b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:34:31.847 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:31.832+0000 7fed4b7fe700 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fed4c1333c0 0x7fed4c1337b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3301/0 says I am v2:172.21.15.88:59860/0 (socket says 172.21.15.88:59860) 2023-09-30T01:34:31.847 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:31.832+0000 7fed4affd700 1 --2- >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7fed4c0a3270 0x7fed4c0a0160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3303/0 says I am v2:172.21.15.88:36404/0 (socket says 172.21.15.88:36404) 2023-09-30T01:34:31.847 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:31.832+0000 7fed4b7fe700 1 -- 172.21.15.88:0/3434928764 learned_addr learned my addr 172.21.15.88:0/3434928764 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-09-30T01:34:31.847 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:31.832+0000 7fed4affd700 10 monclient(hunting): get_auth_request con 0x7fed4c0a3270 auth_method 0 2023-09-30T01:34:31.847 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:31.832+0000 7fed4affd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:34:31.847 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:31.832+0000 7fed4affd700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:34:31.847 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:31.832+0000 7fed4affd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:34:31.847 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:31.832+0000 7fed4b7fe700 10 monclient(hunting): get_auth_request con 0x7fed4c1333c0 auth_method 0 2023-09-30T01:34:31.847 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:31.832+0000 7fed4a7fc700 1 --2- 172.21.15.88:0/3434928764 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7fed4c0a4b00 0x7fed4c0a06a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:34:31.847 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:31.832+0000 7fed4b7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:34:31.848 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:31.832+0000 7fed4b7fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:34:31.848 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:31.832+0000 7fed4b7fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:34:31.848 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:31.832+0000 7fed4a7fc700 10 monclient(hunting): get_auth_request con 0x7fed4c0a4b00 auth_method 0 2023-09-30T01:34:31.848 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:31.832+0000 7fed4a7fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:34:31.848 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:31.832+0000 7fed4a7fc700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:34:31.848 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:31.832+0000 7fed4a7fc700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:34:31.848 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:31.832+0000 7fed4affd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:34:31.848 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:31.832+0000 7fed4affd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:34:31.848 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:31.832+0000 7fed4affd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:34:31.848 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:31.832+0000 7fed4b7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:34:31.848 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:31.832+0000 7fed4b7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:34:31.848 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:31.832+0000 7fed4b7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:34:31.848 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:31.832+0000 7fed4a7fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:34:31.849 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:31.832+0000 7fed4a7fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:34:31.849 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:31.832+0000 7fed4a7fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:34:31.849 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:31.832+0000 7fed4b7fe700 10 monclient(hunting): handle_auth_done global_id 4457 payload 274 2023-09-30T01:34:31.849 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:31.832+0000 7fed4b7fe700 1 -- 172.21.15.88:0/3434928764 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7fed4c0a3270 msgr2=0x7fed4c0a0160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:34:31.849 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:31.832+0000 7fed4b7fe700 1 --2- 172.21.15.88:0/3434928764 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7fed4c0a3270 0x7fed4c0a0160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:34:31.849 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:31.832+0000 7fed4b7fe700 1 -- 172.21.15.88:0/3434928764 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7fed4c0a4b00 msgr2=0x7fed4c0a06a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:34:31.849 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:31.832+0000 7fed4b7fe700 1 --2- 172.21.15.88:0/3434928764 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7fed4c0a4b00 0x7fed4c0a06a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:34:31.849 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:31.832+0000 7fed4b7fe700 10 monclient: _finish_hunting 0 2023-09-30T01:34:31.849 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:31.832+0000 7fed4b7fe700 1 monclient: found mon.noname-b 2023-09-30T01:34:31.849 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:31.832+0000 7fed4b7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:34:31.849 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:31.832+0000 7fed4b7fe700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.88:3301/0 2023-09-30T01:34:31.849 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:31.832+0000 7fed4b7fe700 1 -- 172.21.15.88:0/3434928764 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed4c05ecb0 con 0x7fed4c1333c0 2023-09-30T01:34:31.850 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:31.832+0000 7fed4affd700 1 --2- 172.21.15.88:0/3434928764 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7fed4c0a3270 0x7fed4c0a0160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:34:31.850 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:31.832+0000 7fed4a7fc700 1 --2- 172.21.15.88:0/3434928764 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7fed4c0a4b00 0x7fed4c0a06a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:34:31.850 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:31.832+0000 7fed4b7fe700 1 --2- 172.21.15.88:0/3434928764 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fed4c1333c0 0x7fed4c1337b0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fed44009f50 tx=0x7fed44009840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=290c29aa96d6f830 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:34:31.850 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:31.832+0000 7fed49ffb700 1 -- 172.21.15.88:0/3434928764 <== mon.2 v2:172.21.15.88:3301/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7fed4400e070 con 0x7fed4c1333c0 2023-09-30T01:34:31.850 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:31.832+0000 7fed49ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:34:31.850 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:31.832+0000 7fed49ffb700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T01:34:31.850 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:31.832+0000 7fed49ffb700 10 monclient: dump: 2023-09-30T01:34:31.850 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:34:31.850 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:34:31.850 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:34:31.850 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:34:31.850 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:34:31.851 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:34:31.851 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:34:31.851 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:34:31.851 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:34:31.851 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:34:31.851 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:34:31.851 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:34:31.851 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:34:31.851 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:34:31.851 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:34:31.851 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:34:31.851 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:31.832+0000 7fed49ffb700 10 monclient: _finish_auth 0 2023-09-30T01:34:31.851 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:31.832+0000 7fed49ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:34:31.852 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:31.832+0000 7fed5184d700 5 monclient: authenticate success, global_id 4457 2023-09-30T01:34:31.852 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:31.832+0000 7fed5184d700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:34:31.852 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:31.832+0000 7fed49ffb700 1 -- 172.21.15.88:0/3434928764 <== mon.2 v2:172.21.15.88:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed4400bd70 con 0x7fed4c1333c0 2023-09-30T01:34:31.852 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:31.832+0000 7fed49ffb700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:34:31.852 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:31.832+0000 7fed5184d700 10 monclient: get_monmap_and_config success 2023-09-30T01:34:31.852 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:31.832+0000 7fed5184d700 4 set_mon_vals no callback set 2023-09-30T01:34:31.852 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:31.832+0000 7fed49ffb700 1 -- 172.21.15.88:0/3434928764 <== mon.2 v2:172.21.15.88:3301/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7fed440092e0 con 0x7fed4c1333c0 2023-09-30T01:34:31.852 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:31.832+0000 7fed49ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:34:31.852 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:31.832+0000 7fed49ffb700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T01:34:31.852 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:31.832+0000 7fed49ffb700 10 monclient: dump: 2023-09-30T01:34:31.852 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:34:31.852 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:34:31.852 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:34:31.853 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:34:31.853 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:34:31.853 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:34:31.853 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:34:31.853 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:34:31.853 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:34:31.853 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:34:31.853 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:34:31.853 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:34:31.853 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:34:31.853 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:34:31.853 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:34:31.853 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:34:31.854 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:31.832+0000 7fed5184d700 10 monclient: shutdown 2023-09-30T01:34:31.854 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:31.832+0000 7fed5184d700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:34:31.854 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:31.832+0000 7fed5184d700 1 -- 172.21.15.88:0/3434928764 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fed4c1333c0 msgr2=0x7fed4c1337b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:34:31.854 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:31.832+0000 7fed5184d700 1 --2- 172.21.15.88:0/3434928764 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fed4c1333c0 0x7fed4c1337b0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fed44009f50 tx=0x7fed44009840 comp rx=0 tx=0).stop 2023-09-30T01:34:31.854 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:31.832+0000 7fed5184d700 1 -- 172.21.15.88:0/3434928764 shutdown_connections 2023-09-30T01:34:31.854 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:31.832+0000 7fed5184d700 1 --2- 172.21.15.88:0/3434928764 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fed4c1333c0 0x7fed4c1337b0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:34:31.854 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:31.832+0000 7fed5184d700 1 --2- 172.21.15.88:0/3434928764 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7fed4c0a4b00 0x7fed4c0a06a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:34:31.854 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:31.832+0000 7fed5184d700 1 --2- 172.21.15.88:0/3434928764 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7fed4c0a3270 0x7fed4c0a0160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:34:31.854 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:31.832+0000 7fed5184d700 1 -- 172.21.15.88:0/3434928764 >> 172.21.15.88:0/3434928764 conn(0x7fed4c0a67f0 msgr2=0x7fed4c0a6bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:34:31.854 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:31.832+0000 7fed5184d700 1 -- 172.21.15.88:0/3434928764 shutdown_connections 2023-09-30T01:34:31.855 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:31.832+0000 7fed5184d700 1 -- 172.21.15.88:0/3434928764 wait complete. 2023-09-30T01:34:31.855 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:31.836+0000 7fed5184d700 10 monclient: build_initial_monmap 2023-09-30T01:34:31.855 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:31.836+0000 7fed5184d700 10 monclient: monmap: 2023-09-30T01:34:31.855 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2023-09-30T01:34:31.855 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:34:31.855 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 0.000000 2023-09-30T01:34:31.856 INFO:teuthology.orchestra.run.smithi088.stderr:created 0.000000 2023-09-30T01:34:31.857 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2023-09-30T01:34:31.857 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 1 2023-09-30T01:34:31.857 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.noname-a 2023-09-30T01:34:31.857 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.noname-f 2023-09-30T01:34:31.857 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.noname-b 2023-09-30T01:34:31.857 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.noname-g 2023-09-30T01:34:31.857 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.noname-c 2023-09-30T01:34:31.857 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.noname-h 2023-09-30T01:34:31.857 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.noname-d 2023-09-30T01:34:31.857 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.noname-i 2023-09-30T01:34:31.858 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.noname-e 2023-09-30T01:34:31.858 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:34:31.858 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:31.836+0000 7fed5184d700 1 Processor -- start 2023-09-30T01:34:31.858 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:31.836+0000 7fed5184d700 1 -- start start 2023-09-30T01:34:31.858 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:31.836+0000 7fed5184d700 10 monclient: init 2023-09-30T01:34:31.858 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:31.836+0000 7fed5184d700 10 monclient: _reopen_session rank -1 2023-09-30T01:34:31.858 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:31.836+0000 7fed5184d700 10 monclient: _add_conns ranks=[3,2,5,8,4,7,0,6,1] 2023-09-30T01:34:31.858 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:31.836+0000 7fed5184d700 1 --2- >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7fed4c0a3270 0x7fed4c09e390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:34:31.858 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:31.836+0000 7fed5184d700 10 monclient(hunting): picked mon.noname-g con 0x7fed4c0a3270 addr [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] 2023-09-30T01:34:31.858 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:31.836+0000 7fed5184d700 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fed4c0a4b00 0x7fed4c09e8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:34:31.858 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:31.836+0000 7fed5184d700 10 monclient(hunting): picked mon.noname-b con 0x7fed4c0a4b00 addr [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] 2023-09-30T01:34:31.858 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:31.836+0000 7fed5184d700 1 --2- >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7fed4c1333c0 0x7fed4c09ee10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:34:31.859 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:31.836+0000 7fed5184d700 10 monclient(hunting): picked mon.noname-h con 0x7fed4c1333c0 addr [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] 2023-09-30T01:34:31.859 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:31.836+0000 7fed5184d700 10 monclient(hunting): start opening mon connection 2023-09-30T01:34:31.859 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:31.836+0000 7fed5184d700 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed4c11e410 con 0x7fed4c0a4b00 2023-09-30T01:34:31.859 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:31.836+0000 7fed5184d700 10 monclient(hunting): start opening mon connection 2023-09-30T01:34:31.859 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:31.836+0000 7fed5184d700 1 -- --> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed4c11db70 con 0x7fed4c0a3270 2023-09-30T01:34:31.859 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:31.836+0000 7fed5184d700 10 monclient(hunting): start opening mon connection 2023-09-30T01:34:31.859 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:31.836+0000 7fed5184d700 1 -- --> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] -- mon_getmap magic: 0 v1 -- 0x7fed4c11e130 con 0x7fed4c1333c0 2023-09-30T01:34:31.859 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:31.836+0000 7fed5184d700 10 monclient(hunting): _renew_subs 2023-09-30T01:34:31.859 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:31.836+0000 7fed5184d700 10 monclient(hunting): authenticate will time out at 943.877968s 2023-09-30T01:34:31.859 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:31.836+0000 7fed4a7fc700 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fed4c0a4b00 0x7fed4c09e8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:34:31.859 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:31.836+0000 7fed4a7fc700 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fed4c0a4b00 0x7fed4c09e8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3301/0 says I am v2:172.21.15.88:59862/0 (socket says 172.21.15.88:59862) 2023-09-30T01:34:31.859 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:31.836+0000 7fed4a7fc700 1 -- 172.21.15.88:0/895041046 learned_addr learned my addr 172.21.15.88:0/895041046 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-09-30T01:34:31.860 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:31.836+0000 7fed4a7fc700 10 monclient(hunting): get_auth_request con 0x7fed4c0a4b00 auth_method 0 2023-09-30T01:34:31.860 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:31.836+0000 7fed4a7fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:34:31.860 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:31.836+0000 7fed4a7fc700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:34:31.860 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:31.836+0000 7fed4a7fc700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:34:31.860 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:31.836+0000 7fed4b7fe700 1 --2- 172.21.15.88:0/895041046 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7fed4c1333c0 0x7fed4c09ee10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:34:31.860 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:31.836+0000 7fed4affd700 1 --2- 172.21.15.88:0/895041046 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7fed4c0a3270 0x7fed4c09e390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:34:31.860 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:31.836+0000 7fed4a7fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:34:31.860 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:31.836+0000 7fed4a7fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:34:31.860 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:31.836+0000 7fed4a7fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:34:31.860 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:31.836+0000 7fed4b7fe700 10 monclient(hunting): get_auth_request con 0x7fed4c1333c0 auth_method 0 2023-09-30T01:34:31.860 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:31.836+0000 7fed4b7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:34:31.860 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:31.836+0000 7fed4b7fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:34:31.860 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:31.836+0000 7fed4b7fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:34:31.861 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:31.836+0000 7fed4affd700 10 monclient(hunting): get_auth_request con 0x7fed4c0a3270 auth_method 0 2023-09-30T01:34:31.861 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:31.836+0000 7fed4affd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:34:31.861 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:31.836+0000 7fed4affd700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:34:31.861 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:31.836+0000 7fed4affd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:34:31.861 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:31.836+0000 7fed4affd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:34:31.861 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:31.836+0000 7fed4affd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:34:31.861 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:31.836+0000 7fed4affd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:34:31.861 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:31.836+0000 7fed4b7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:34:31.861 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:31.836+0000 7fed4b7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:34:31.861 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:31.836+0000 7fed4b7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:34:31.861 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:31.836+0000 7fed4a7fc700 10 monclient(hunting): handle_auth_done global_id 4466 payload 824 2023-09-30T01:34:31.861 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:31.836+0000 7fed4a7fc700 1 -- 172.21.15.88:0/895041046 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7fed4c1333c0 msgr2=0x7fed4c09ee10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:34:31.862 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:31.836+0000 7fed4a7fc700 1 --2- 172.21.15.88:0/895041046 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7fed4c1333c0 0x7fed4c09ee10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:34:31.862 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:31.836+0000 7fed4a7fc700 1 -- 172.21.15.88:0/895041046 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7fed4c0a3270 msgr2=0x7fed4c09e390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:34:31.862 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:31.836+0000 7fed4a7fc700 1 --2- 172.21.15.88:0/895041046 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7fed4c0a3270 0x7fed4c09e390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:34:31.862 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:31.836+0000 7fed4a7fc700 10 monclient: _finish_hunting 0 2023-09-30T01:34:31.862 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:31.836+0000 7fed4a7fc700 1 monclient: found mon.noname-b 2023-09-30T01:34:31.862 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:31.836+0000 7fed4a7fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:34:31.862 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:31.836+0000 7fed4a7fc700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.88:3301/0 2023-09-30T01:34:31.862 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:31.836+0000 7fed4a7fc700 1 -- 172.21.15.88:0/895041046 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed40009420 con 0x7fed4c0a4b00 2023-09-30T01:34:31.862 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:31.836+0000 7fed4a7fc700 1 --2- 172.21.15.88:0/895041046 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fed4c0a4b00 0x7fed4c09e8d0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fed3c00ec80 tx=0x7fed3c0091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:34:31.862 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:31.836+0000 7fed32ffd700 1 -- 172.21.15.88:0/895041046 <== mon.2 v2:172.21.15.88:3301/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7fed3c00ccd0 con 0x7fed4c0a4b00 2023-09-30T01:34:31.862 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:31.836+0000 7fed32ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:34:31.862 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:31.836+0000 7fed32ffd700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T01:34:31.863 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:31.836+0000 7fed32ffd700 10 monclient: dump: 2023-09-30T01:34:31.863 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:34:31.863 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:34:31.863 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:34:31.863 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:34:31.863 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:34:31.863 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:34:31.863 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:34:31.863 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:34:31.863 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:34:31.863 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:34:31.863 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:34:31.863 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:34:31.864 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:34:31.864 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:34:31.864 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:34:31.864 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:34:31.864 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:31.836+0000 7fed32ffd700 10 monclient: _finish_auth 0 2023-09-30T01:34:31.864 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:31.836+0000 7fed32ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:34:31.864 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:31.836+0000 7fed5184d700 5 monclient: authenticate success, global_id 4466 2023-09-30T01:34:31.864 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:31.836+0000 7fed32ffd700 1 -- 172.21.15.88:0/895041046 <== mon.2 v2:172.21.15.88:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed3c00ce30 con 0x7fed4c0a4b00 2023-09-30T01:34:31.864 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:31.836+0000 7fed5184d700 10 monclient: _renew_subs 2023-09-30T01:34:31.864 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:31.836+0000 7fed5184d700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.88:3301/0 2023-09-30T01:34:31.864 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:31.836+0000 7fed5184d700 1 -- 172.21.15.88:0/895041046 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed3c009420 con 0x7fed4c0a4b00 2023-09-30T01:34:31.864 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:31.836+0000 7fed32ffd700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:34:31.864 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:31.836+0000 7fed32ffd700 1 -- 172.21.15.88:0/895041046 <== mon.2 v2:172.21.15.88:3301/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7fed3c0138f0 con 0x7fed4c0a4b00 2023-09-30T01:34:31.865 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:31.836+0000 7fed32ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:34:31.865 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:31.836+0000 7fed32ffd700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T01:34:31.865 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:31.836+0000 7fed497fa700 4 set_mon_vals no callback set 2023-09-30T01:34:31.865 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:31.836+0000 7fed32ffd700 10 monclient: dump: 2023-09-30T01:34:31.865 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:34:31.865 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:34:31.865 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:34:31.865 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:34:31.865 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:34:31.865 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:34:31.865 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:34:31.865 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:34:31.866 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:34:31.866 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:34:31.866 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:34:31.866 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:34:31.866 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:34:31.866 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:34:31.866 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:34:31.866 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:34:31.866 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:31.836+0000 7fed5184d700 10 monclient: _renew_subs 2023-09-30T01:34:31.866 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:31.836+0000 7fed5184d700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.88:3301/0 2023-09-30T01:34:31.866 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:31.836+0000 7fed5184d700 1 -- 172.21.15.88:0/895041046 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed44009ae0 con 0x7fed4c0a4b00 2023-09-30T01:34:31.866 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:31.840+0000 7fed32ffd700 1 -- 172.21.15.88:0/895041046 <== mon.2 v2:172.21.15.88:3301/0 4 ==== mgrmap(e 5) v1 ==== 40144+0+0 (secure 0 0 0) 0x7fed3c013470 con 0x7fed4c0a4b00 2023-09-30T01:34:31.867 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:31.840+0000 7fed5184d700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:34:31.867 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:31.840+0000 7fed5184d700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:34:31.867 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:31.840+0000 7fed5184d700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.88:3301/0 2023-09-30T01:34:31.867 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:31.840+0000 7fed5184d700 1 -- 172.21.15.88:0/895041046 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed4c071960 con 0x7fed4c0a4b00 2023-09-30T01:34:31.867 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:31.840+0000 7fed32ffd700 1 --2- 172.21.15.88:0/895041046 >> v2:172.21.15.107:6812/29926 conn(0x7fed38033360 0x7fed38035810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:34:31.867 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:31.840+0000 7fed32ffd700 1 -- 172.21.15.88:0/895041046 <== mon.2 v2:172.21.15.88:3301/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 3502+0+0 (secure 0 0 0) 0x7fed3c0437f0 con 0x7fed4c0a4b00 2023-09-30T01:34:31.867 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:31.840+0000 7fed4affd700 1 --2- 172.21.15.88:0/895041046 >> v2:172.21.15.107:6812/29926 conn(0x7fed38033360 0x7fed38035810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:34:31.867 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:31.840+0000 7fed4affd700 10 monclient: get_auth_request con 0x7fed38033360 auth_method 0 2023-09-30T01:34:31.868 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:31.840+0000 7fed4affd700 1 --2- 172.21.15.88:0/895041046 >> v2:172.21.15.107:6812/29926 conn(0x7fed38033360 0x7fed38035810 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fed40009f40 tx=0x7fed400091a0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:34:31.868 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:31.848+0000 7fed32ffd700 1 -- 172.21.15.88:0/895041046 <== mon.2 v2:172.21.15.88:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fed3c013720 con 0x7fed4c0a4b00 2023-09-30T01:34:31.868 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:31.848+0000 7fed32ffd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:34:31.868 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:31.848+0000 7fed32ffd700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:34:32.045 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:32.040+0000 7fed5184d700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T01:34:32.045 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:32.040+0000 7fed5184d700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:34:32.045 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:32.040+0000 7fed5184d700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.88:3301/0 2023-09-30T01:34:32.045 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:32.040+0000 7fed5184d700 1 -- 172.21.15.88:0/895041046 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fed4c12dc00 con 0x7fed4c0a4b00 2023-09-30T01:34:32.046 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:32.040+0000 7fed32ffd700 1 -- 172.21.15.88:0/895041046 <== mon.2 v2:172.21.15.88:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7fed3c01bba0 con 0x7fed4c0a4b00 2023-09-30T01:34:32.046 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:32.040+0000 7fed32ffd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:34:32.046 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:32.040+0000 7fed32ffd700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:34:32.046 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-09-30T01:34:32.046 INFO:teuthology.orchestra.run.smithi088.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:34:32.048 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:32.044+0000 7fed5184d700 1 -- 172.21.15.88:0/895041046 >> v2:172.21.15.107:6812/29926 conn(0x7fed38033360 msgr2=0x7fed38035810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:34:32.048 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:32.044+0000 7fed5184d700 1 --2- 172.21.15.88:0/895041046 >> v2:172.21.15.107:6812/29926 conn(0x7fed38033360 0x7fed38035810 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fed40009f40 tx=0x7fed400091a0 comp rx=0 tx=0).stop 2023-09-30T01:34:32.048 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:32.044+0000 7fed5184d700 10 monclient: shutdown 2023-09-30T01:34:32.048 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:32.044+0000 7fed5184d700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:34:32.048 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:32.044+0000 7fed5184d700 1 -- 172.21.15.88:0/895041046 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fed4c0a4b00 msgr2=0x7fed4c09e8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:34:32.048 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:32.044+0000 7fed5184d700 1 --2- 172.21.15.88:0/895041046 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fed4c0a4b00 0x7fed4c09e8d0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fed3c00ec80 tx=0x7fed3c0091a0 comp rx=0 tx=0).stop 2023-09-30T01:34:32.049 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:32.044+0000 7fed5184d700 1 -- 172.21.15.88:0/895041046 shutdown_connections 2023-09-30T01:34:32.049 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:32.044+0000 7fed5184d700 1 --2- 172.21.15.88:0/895041046 >> v2:172.21.15.107:6812/29926 conn(0x7fed38033360 0x7fed38035810 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:34:32.049 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:32.044+0000 7fed5184d700 1 --2- 172.21.15.88:0/895041046 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7fed4c1333c0 0x7fed4c09ee10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:34:32.049 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:32.044+0000 7fed5184d700 1 --2- 172.21.15.88:0/895041046 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fed4c0a4b00 0x7fed4c09e8d0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:34:32.049 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:32.044+0000 7fed5184d700 1 --2- 172.21.15.88:0/895041046 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7fed4c0a3270 0x7fed4c09e390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:34:32.049 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:32.044+0000 7fed5184d700 1 -- 172.21.15.88:0/895041046 >> 172.21.15.88:0/895041046 conn(0x7fed4c0a67f0 msgr2=0x7fed4c131410 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:34:32.049 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:32.044+0000 7fed5184d700 1 -- 172.21.15.88:0/895041046 shutdown_connections 2023-09-30T01:34:32.049 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:32.044+0000 7fed5184d700 1 -- 172.21.15.88:0/895041046 wait complete. 2023-09-30T01:34:35.060 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:34:35.174 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:35.167+0000 7fbcad8a2700 10 monclient: get_monmap_and_config 2023-09-30T01:34:35.175 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:35.167+0000 7fbcad8a2700 10 monclient: build_initial_monmap 2023-09-30T01:34:35.175 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:35.167+0000 7fbcad8a2700 10 monclient: monmap: 2023-09-30T01:34:35.175 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2023-09-30T01:34:35.175 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:34:35.175 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:34:35.175158+0000 2023-09-30T01:34:35.175 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:34:35.175158+0000 2023-09-30T01:34:35.175 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2023-09-30T01:34:35.175 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:34:35.175 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a-legacy 2023-09-30T01:34:35.176 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.107:6789/0 mon.noname-f-legacy 2023-09-30T01:34:35.176 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.noname-b 2023-09-30T01:34:35.176 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.noname-g 2023-09-30T01:34:35.176 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.noname-c 2023-09-30T01:34:35.176 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.noname-h 2023-09-30T01:34:35.176 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.noname-d 2023-09-30T01:34:35.176 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.noname-i 2023-09-30T01:34:35.176 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.noname-e 2023-09-30T01:34:35.176 INFO:teuthology.orchestra.run.smithi088.stderr:9: v2:172.21.15.88:3300/0 mon.noname-a 2023-09-30T01:34:35.176 INFO:teuthology.orchestra.run.smithi088.stderr:10: v2:172.21.15.107:3300/0 mon.noname-f 2023-09-30T01:34:35.176 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:34:35.180 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:35.171+0000 7fbcad8a2700 1 Processor -- start 2023-09-30T01:34:35.180 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:35.171+0000 7fbcad8a2700 1 -- start start 2023-09-30T01:34:35.180 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:35.171+0000 7fbcad8a2700 10 monclient: init 2023-09-30T01:34:35.180 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:35.171+0000 7fbcad8a2700 10 monclient: _reopen_session rank -1 2023-09-30T01:34:35.180 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:35.171+0000 7fbcad8a2700 10 monclient: _add_conns ranks=[1,8,9,6,0,4,5,3,10,2,7] 2023-09-30T01:34:35.180 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:35.171+0000 7fbcad8a2700 10 monclient(hunting): picked mon.noname-f-legacy con 0x7fbca80a3380 addr v1:172.21.15.107:6789/0 2023-09-30T01:34:35.180 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:35.171+0000 7fbcad8a2700 1 --2- >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7fbca809ffd0 0x7fbca80a03a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:34:35.180 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:35.171+0000 7fbcad8a2700 10 monclient(hunting): picked mon.noname-e con 0x7fbca809ffd0 addr [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] 2023-09-30T01:34:35.180 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:35.171+0000 7fbcad8a2700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fbca81386e0 0x7fbca813aac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:34:35.180 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:35.171+0000 7fbcad8a2700 10 monclient(hunting): picked mon.noname-a con 0x7fbca81386e0 addr v2:172.21.15.88:3300/0 2023-09-30T01:34:35.181 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:35.171+0000 7fbcad8a2700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbca804bb20 con 0x7fbca80a3380 2023-09-30T01:34:35.181 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:35.171+0000 7fbcad8a2700 10 monclient(hunting): start opening mon connection 2023-09-30T01:34:35.181 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:35.171+0000 7fbcad8a2700 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbca8097ab0 con 0x7fbca81386e0 2023-09-30T01:34:35.181 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:35.171+0000 7fbcad8a2700 10 monclient(hunting): start opening mon connection 2023-09-30T01:34:35.181 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:35.171+0000 7fbcad8a2700 1 -- --> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] -- mon_getmap magic: 0 v1 -- 0x7fbca80a08e0 con 0x7fbca809ffd0 2023-09-30T01:34:35.181 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:35.171+0000 7fbcad8a2700 10 monclient(hunting): _renew_subs 2023-09-30T01:34:35.181 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:35.171+0000 7fbcad8a2700 10 monclient(hunting): authenticate will time out at 947.212676s 2023-09-30T01:34:35.181 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:35.171+0000 7fbca7fff700 1 --2- >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7fbca809ffd0 0x7fbca80a03a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:34:35.181 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:35.171+0000 7fbcad0a1700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fbca81386e0 0x7fbca813aac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:34:35.181 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:35.171+0000 7fbca7fff700 1 --2- >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7fbca809ffd0 0x7fbca80a03a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3304/0 says I am v2:172.21.15.88:57922/0 (socket says 172.21.15.88:57922) 2023-09-30T01:34:35.181 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:35.171+0000 7fbca7fff700 1 -- 172.21.15.88:0/1942006863 learned_addr learned my addr 172.21.15.88:0/1942006863 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-09-30T01:34:35.181 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:35.171+0000 7fbcad0a1700 10 monclient(hunting): get_auth_request con 0x7fbca81386e0 auth_method 0 2023-09-30T01:34:35.182 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:35.171+0000 7fbcad0a1700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:34:35.182 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:35.171+0000 7fbcad0a1700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:34:35.182 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:35.171+0000 7fbcad0a1700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:34:35.182 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:35.171+0000 7fbca7fff700 10 monclient(hunting): get_auth_request con 0x7fbca809ffd0 auth_method 0 2023-09-30T01:34:35.182 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:35.171+0000 7fbca7fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:34:35.182 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:35.171+0000 7fbca7fff700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:34:35.182 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:35.171+0000 7fbca7fff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:34:35.182 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:35.171+0000 7fbca7fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:34:35.182 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:35.171+0000 7fbca7fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:34:35.182 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:35.171+0000 7fbca7fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:34:35.182 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:35.171+0000 7fbcad0a1700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:34:35.183 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:35.171+0000 7fbcad0a1700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:34:35.183 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:35.171+0000 7fbcad0a1700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:34:35.183 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:35.171+0000 7fbca7fff700 10 monclient(hunting): handle_auth_done global_id 4553 payload 274 2023-09-30T01:34:35.183 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:35.171+0000 7fbca7fff700 1 -- 172.21.15.88:0/1942006863 >> v2:172.21.15.88:3300/0 conn(0x7fbca81386e0 msgr2=0x7fbca813aac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:34:35.183 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:35.171+0000 7fbca7fff700 1 --2- 172.21.15.88:0/1942006863 >> v2:172.21.15.88:3300/0 conn(0x7fbca81386e0 0x7fbca813aac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:34:35.183 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:35.171+0000 7fbca7fff700 1 -- 172.21.15.88:0/1942006863 >> v1:172.21.15.107:6789/0 conn(0x7fbca80a3380 legacy=0x7fbca80a48b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:34:35.183 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:35.171+0000 7fbca7fff700 10 monclient: _finish_hunting 0 2023-09-30T01:34:35.183 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:35.171+0000 7fbca7fff700 1 monclient: found mon.noname-e 2023-09-30T01:34:35.183 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:35.171+0000 7fbca7fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:34:35.183 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:35.171+0000 7fbca7fff700 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.88:3304/0 2023-09-30T01:34:35.183 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:35.171+0000 7fbca7fff700 1 -- 172.21.15.88:0/1942006863 --> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbca8098480 con 0x7fbca809ffd0 2023-09-30T01:34:35.183 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:35.171+0000 7fbca7fff700 1 --2- 172.21.15.88:0/1942006863 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7fbca809ffd0 0x7fbca80a03a0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fbc98009f20 tx=0x7fbc98009840 comp rx=0 tx=0).ready entity=mon.8 client_cookie=365fa422a8a6237 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:34:35.184 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:35.171+0000 7fbca77fe700 1 -- 172.21.15.88:0/1942006863 <== mon.8 v2:172.21.15.88:3304/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7fbc9800e070 con 0x7fbca809ffd0 2023-09-30T01:34:35.184 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:35.171+0000 7fbca77fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:34:35.184 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:35.171+0000 7fbca77fe700 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2023-09-30T01:34:35.184 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:35.171+0000 7fbca77fe700 10 monclient: dump: 2023-09-30T01:34:35.184 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:34:35.184 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:34:35.184 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:34:35.184 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:34:35.184 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:34:35.184 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:34:35.184 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:34:35.184 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:34:35.184 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:34:35.185 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:34:35.185 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:34:35.185 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:34:35.185 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:34:35.185 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:34:35.185 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:34:35.185 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:34:35.185 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:35.171+0000 7fbca77fe700 10 monclient: _finish_auth 0 2023-09-30T01:34:35.185 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:35.171+0000 7fbca77fe700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:34:35.185 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:35.171+0000 7fbcad8a2700 5 monclient: authenticate success, global_id 4553 2023-09-30T01:34:35.185 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:35.171+0000 7fbcad8a2700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:34:35.185 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:35.171+0000 7fbca77fe700 1 -- 172.21.15.88:0/1942006863 <== mon.8 v2:172.21.15.88:3304/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc9800be50 con 0x7fbca809ffd0 2023-09-30T01:34:35.185 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:35.171+0000 7fbca77fe700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:34:35.186 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:35.171+0000 7fbcad8a2700 10 monclient: get_monmap_and_config success 2023-09-30T01:34:35.186 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:35.171+0000 7fbcad8a2700 4 set_mon_vals no callback set 2023-09-30T01:34:35.186 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:35.171+0000 7fbca77fe700 1 -- 172.21.15.88:0/1942006863 <== mon.8 v2:172.21.15.88:3304/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7fbc980092e0 con 0x7fbca809ffd0 2023-09-30T01:34:35.186 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:35.171+0000 7fbca77fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:34:35.186 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:35.171+0000 7fbca77fe700 10 monclient: got monmap 1 from mon.e (according to old e1) 2023-09-30T01:34:35.186 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:35.171+0000 7fbca77fe700 10 monclient: dump: 2023-09-30T01:34:35.186 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:34:35.186 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:34:35.186 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:34:35.186 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:34:35.186 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:34:35.186 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:34:35.187 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:34:35.187 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:34:35.187 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:34:35.187 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:34:35.187 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:34:35.187 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:34:35.187 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:34:35.187 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:34:35.187 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:34:35.187 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:34:35.187 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:35.171+0000 7fbcad8a2700 10 monclient: shutdown 2023-09-30T01:34:35.187 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:35.171+0000 7fbcad8a2700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:34:35.187 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:35.171+0000 7fbcad8a2700 1 -- 172.21.15.88:0/1942006863 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7fbca809ffd0 msgr2=0x7fbca80a03a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:34:35.188 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:35.171+0000 7fbcad8a2700 1 --2- 172.21.15.88:0/1942006863 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7fbca809ffd0 0x7fbca80a03a0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fbc98009f20 tx=0x7fbc98009840 comp rx=0 tx=0).stop 2023-09-30T01:34:35.188 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:35.171+0000 7fbcad8a2700 1 -- 172.21.15.88:0/1942006863 shutdown_connections 2023-09-30T01:34:35.188 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:35.171+0000 7fbcad8a2700 1 --2- 172.21.15.88:0/1942006863 >> v2:172.21.15.88:3300/0 conn(0x7fbca81386e0 0x7fbca813aac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:34:35.188 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:35.171+0000 7fbcad8a2700 1 --2- 172.21.15.88:0/1942006863 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7fbca809ffd0 0x7fbca80a03a0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:34:35.188 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:35.171+0000 7fbcad8a2700 1 -- 172.21.15.88:0/1942006863 >> 172.21.15.88:0/1942006863 conn(0x7fbca806e5e0 msgr2=0x7fbca806e9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:34:35.188 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:35.171+0000 7fbcad8a2700 1 -- 172.21.15.88:0/1942006863 shutdown_connections 2023-09-30T01:34:35.188 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:35.171+0000 7fbcad8a2700 1 -- 172.21.15.88:0/1942006863 wait complete. 2023-09-30T01:34:35.188 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:35.175+0000 7fbcad8a2700 10 monclient: build_initial_monmap 2023-09-30T01:34:35.188 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:35.175+0000 7fbcad8a2700 10 monclient: monmap: 2023-09-30T01:34:35.188 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2023-09-30T01:34:35.188 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:34:35.188 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 0.000000 2023-09-30T01:34:35.189 INFO:teuthology.orchestra.run.smithi088.stderr:created 0.000000 2023-09-30T01:34:35.189 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2023-09-30T01:34:35.189 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 1 2023-09-30T01:34:35.189 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.noname-a 2023-09-30T01:34:35.189 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.noname-f 2023-09-30T01:34:35.189 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.noname-b 2023-09-30T01:34:35.189 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.noname-g 2023-09-30T01:34:35.189 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.noname-c 2023-09-30T01:34:35.189 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.noname-h 2023-09-30T01:34:35.189 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.noname-d 2023-09-30T01:34:35.189 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.noname-i 2023-09-30T01:34:35.189 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.noname-e 2023-09-30T01:34:35.189 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:34:35.193 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:35.175+0000 7fbcad8a2700 1 Processor -- start 2023-09-30T01:34:35.194 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:35.175+0000 7fbcad8a2700 1 -- start start 2023-09-30T01:34:35.194 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:35.175+0000 7fbcad8a2700 10 monclient: init 2023-09-30T01:34:35.194 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:35.175+0000 7fbcad8a2700 10 monclient: _reopen_session rank -1 2023-09-30T01:34:35.194 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:35.175+0000 7fbcad8a2700 10 monclient: _add_conns ranks=[0,4,1,6,2,3,8,7,5] 2023-09-30T01:34:35.194 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:35.175+0000 7fbcad8a2700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbca809ffd0 0x7fbca811fad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:34:35.194 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:35.175+0000 7fbcad8a2700 10 monclient(hunting): picked mon.noname-a con 0x7fbca809ffd0 addr [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] 2023-09-30T01:34:35.194 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:35.175+0000 7fbcad8a2700 1 --2- >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7fbca80a3380 0x7fbca8120010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:34:35.194 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:35.175+0000 7fbcad8a2700 10 monclient(hunting): picked mon.noname-c con 0x7fbca80a3380 addr [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] 2023-09-30T01:34:35.194 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:35.175+0000 7fbcad8a2700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbca81386e0 0x7fbca8120550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:34:35.194 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:35.175+0000 7fbcad8a2700 10 monclient(hunting): picked mon.noname-f con 0x7fbca81386e0 addr [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] 2023-09-30T01:34:35.194 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:35.175+0000 7fbcad8a2700 10 monclient(hunting): start opening mon connection 2023-09-30T01:34:35.195 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:35.175+0000 7fbcad8a2700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbca8097ce0 con 0x7fbca809ffd0 2023-09-30T01:34:35.195 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:35.175+0000 7fbcad8a2700 10 monclient(hunting): start opening mon connection 2023-09-30T01:34:35.195 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:35.175+0000 7fbcad8a2700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbca81233f0 con 0x7fbca81386e0 2023-09-30T01:34:35.195 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:35.175+0000 7fbcad8a2700 10 monclient(hunting): start opening mon connection 2023-09-30T01:34:35.195 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:35.175+0000 7fbcad8a2700 1 -- --> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] -- mon_getmap magic: 0 v1 -- 0x7fbca8123110 con 0x7fbca80a3380 2023-09-30T01:34:35.195 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:35.175+0000 7fbcad8a2700 10 monclient(hunting): _renew_subs 2023-09-30T01:34:35.195 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:35.175+0000 7fbcad8a2700 10 monclient(hunting): authenticate will time out at 947.218784s 2023-09-30T01:34:35.195 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:35.175+0000 7fbcac8a0700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbca809ffd0 0x7fbca811fad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:34:35.195 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:35.175+0000 7fbcac8a0700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbca809ffd0 0x7fbca811fad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:52866/0 (socket says 172.21.15.88:52866) 2023-09-30T01:34:35.195 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:35.175+0000 7fbcac8a0700 1 -- 172.21.15.88:0/285279241 learned_addr learned my addr 172.21.15.88:0/285279241 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-09-30T01:34:35.195 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:35.175+0000 7fbcac8a0700 10 monclient(hunting): get_auth_request con 0x7fbca809ffd0 auth_method 0 2023-09-30T01:34:35.195 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:35.175+0000 7fbcac8a0700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:34:35.196 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:35.175+0000 7fbcac8a0700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:34:35.196 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:35.175+0000 7fbcac8a0700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:34:35.196 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:35.175+0000 7fbca7fff700 1 --2- 172.21.15.88:0/285279241 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7fbca80a3380 0x7fbca8120010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:34:35.196 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:35.175+0000 7fbca7fff700 10 monclient(hunting): get_auth_request con 0x7fbca80a3380 auth_method 0 2023-09-30T01:34:35.196 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:35.175+0000 7fbca7fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:34:35.196 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:35.175+0000 7fbca7fff700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:34:35.196 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:35.175+0000 7fbca7fff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:34:35.196 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:35.175+0000 7fbcac8a0700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:34:35.196 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:35.175+0000 7fbcac8a0700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:34:35.196 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:35.175+0000 7fbcac8a0700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:34:35.196 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:35.175+0000 7fbcad0a1700 1 --2- 172.21.15.88:0/285279241 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbca81386e0 0x7fbca8120550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:34:35.196 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:35.175+0000 7fbcad0a1700 10 monclient(hunting): get_auth_request con 0x7fbca81386e0 auth_method 0 2023-09-30T01:34:35.197 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:35.175+0000 7fbcad0a1700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:34:35.197 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:35.175+0000 7fbcad0a1700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:34:35.197 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:35.175+0000 7fbcad0a1700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:34:35.197 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:35.175+0000 7fbca7fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:34:35.197 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:35.175+0000 7fbca7fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:34:35.197 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:35.175+0000 7fbca7fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:34:35.197 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:35.175+0000 7fbcac8a0700 10 monclient(hunting): handle_auth_done global_id 4662 payload 824 2023-09-30T01:34:35.197 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:35.175+0000 7fbcac8a0700 1 -- 172.21.15.88:0/285279241 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7fbca80a3380 msgr2=0x7fbca8120010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:34:35.197 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:35.175+0000 7fbcac8a0700 1 --2- 172.21.15.88:0/285279241 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7fbca80a3380 0x7fbca8120010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:34:35.197 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:35.175+0000 7fbcac8a0700 1 -- 172.21.15.88:0/285279241 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbca81386e0 msgr2=0x7fbca8120550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:34:35.197 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:35.175+0000 7fbcac8a0700 1 --2- 172.21.15.88:0/285279241 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbca81386e0 0x7fbca8120550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:34:35.197 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:35.175+0000 7fbcac8a0700 10 monclient: _finish_hunting 0 2023-09-30T01:34:35.198 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:35.175+0000 7fbcac8a0700 1 monclient: found mon.noname-a 2023-09-30T01:34:35.198 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:35.175+0000 7fbcac8a0700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:34:35.198 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:35.175+0000 7fbcac8a0700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.88:3300/0 2023-09-30T01:34:35.198 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:35.175+0000 7fbcac8a0700 1 -- 172.21.15.88:0/285279241 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc98009ae0 con 0x7fbca809ffd0 2023-09-30T01:34:35.198 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:35.175+0000 7fbcac8a0700 1 --2- 172.21.15.88:0/285279241 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbca809ffd0 0x7fbca811fad0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fbc9c00c790 tx=0x7fbc9c007f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:34:35.198 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:35.179+0000 7fbca4ff9700 1 -- 172.21.15.88:0/285279241 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7fbc9c011a10 con 0x7fbca809ffd0 2023-09-30T01:34:35.198 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:35.179+0000 7fbca4ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:34:35.198 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:35.179+0000 7fbca4ff9700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T01:34:35.198 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:35.179+0000 7fbca4ff9700 10 monclient: dump: 2023-09-30T01:34:35.198 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:34:35.198 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:34:35.198 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:34:35.199 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:34:35.199 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:34:35.199 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:34:35.199 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:34:35.199 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:34:35.199 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:34:35.199 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:34:35.199 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:34:35.199 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:34:35.199 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:34:35.199 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:34:35.199 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:34:35.199 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:34:35.200 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:35.179+0000 7fbca4ff9700 10 monclient: _finish_auth 0 2023-09-30T01:34:35.200 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:35.179+0000 7fbca4ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:34:35.200 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:35.179+0000 7fbcad8a2700 5 monclient: authenticate success, global_id 4662 2023-09-30T01:34:35.200 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:35.179+0000 7fbcad8a2700 10 monclient: _renew_subs 2023-09-30T01:34:35.200 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:35.179+0000 7fbcad8a2700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.88:3300/0 2023-09-30T01:34:35.200 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:35.179+0000 7fbcad8a2700 1 -- 172.21.15.88:0/285279241 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbca81381f0 con 0x7fbca809ffd0 2023-09-30T01:34:35.200 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:35.179+0000 7fbca4ff9700 1 -- 172.21.15.88:0/285279241 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc9c011590 con 0x7fbca809ffd0 2023-09-30T01:34:35.200 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:35.179+0000 7fbca4ff9700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:34:35.200 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:35.179+0000 7fbca4ff9700 1 -- 172.21.15.88:0/285279241 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7fbc9c001120 con 0x7fbca809ffd0 2023-09-30T01:34:35.200 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:35.179+0000 7fbca4ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:34:35.200 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:35.179+0000 7fbca6ffd700 4 set_mon_vals no callback set 2023-09-30T01:34:35.200 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:35.179+0000 7fbca4ff9700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:34:35.201 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:35.179+0000 7fbca4ff9700 10 monclient: dump: 2023-09-30T01:34:35.201 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:34:35.201 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:34:35.201 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:34:35.201 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:34:35.201 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:34:35.201 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:34:35.201 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:34:35.201 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:34:35.201 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:34:35.201 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:34:35.201 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:34:35.202 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:34:35.202 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:34:35.202 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:34:35.202 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:34:35.202 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:34:35.202 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:35.179+0000 7fbcad8a2700 10 monclient: _renew_subs 2023-09-30T01:34:35.202 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:35.179+0000 7fbcad8a2700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.88:3300/0 2023-09-30T01:34:35.202 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:35.179+0000 7fbcad8a2700 1 -- 172.21.15.88:0/285279241 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbca8122220 con 0x7fbca809ffd0 2023-09-30T01:34:35.202 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:35.179+0000 7fbca4ff9700 1 -- 172.21.15.88:0/285279241 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 40144+0+0 (secure 0 0 0) 0x7fbc9c011700 con 0x7fbca809ffd0 2023-09-30T01:34:35.202 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:35.179+0000 7fbcad8a2700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:34:35.202 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:35.179+0000 7fbcad8a2700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:34:35.203 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:35.179+0000 7fbcad8a2700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.88:3300/0 2023-09-30T01:34:35.203 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:35.179+0000 7fbcad8a2700 1 -- 172.21.15.88:0/285279241 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbca80a5b60 con 0x7fbca809ffd0 2023-09-30T01:34:35.203 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:35.179+0000 7fbca4ff9700 1 --2- 172.21.15.88:0/285279241 >> v2:172.21.15.107:6812/29926 conn(0x7fbc94033360 0x7fbc94035810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:34:35.203 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:35.179+0000 7fbca4ff9700 1 -- 172.21.15.88:0/285279241 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 3502+0+0 (secure 0 0 0) 0x7fbc9c0424d0 con 0x7fbca809ffd0 2023-09-30T01:34:35.203 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:35.179+0000 7fbca7fff700 1 --2- 172.21.15.88:0/285279241 >> v2:172.21.15.107:6812/29926 conn(0x7fbc94033360 0x7fbc94035810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:34:35.203 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:35.179+0000 7fbca7fff700 10 monclient: get_auth_request con 0x7fbc94033360 auth_method 0 2023-09-30T01:34:35.203 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:35.179+0000 7fbca7fff700 1 --2- 172.21.15.88:0/285279241 >> v2:172.21.15.107:6812/29926 conn(0x7fbc94033360 0x7fbc94035810 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fbc98009f20 tx=0x7fbc98018000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:34:35.203 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:35.187+0000 7fbca4ff9700 1 -- 172.21.15.88:0/285279241 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fbc9c0175e0 con 0x7fbca809ffd0 2023-09-30T01:34:35.203 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:35.187+0000 7fbca4ff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:34:35.203 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:35.187+0000 7fbca4ff9700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:34:35.389 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:35.383+0000 7fbcad8a2700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T01:34:35.389 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:35.383+0000 7fbcad8a2700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:34:35.389 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:35.383+0000 7fbcad8a2700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.88:3300/0 2023-09-30T01:34:35.389 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:35.383+0000 7fbcad8a2700 1 -- 172.21.15.88:0/285279241 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fbca80710d0 con 0x7fbca809ffd0 2023-09-30T01:34:35.390 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:35.383+0000 7fbca4ff9700 1 -- 172.21.15.88:0/285279241 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7fbc9c017030 con 0x7fbca809ffd0 2023-09-30T01:34:35.390 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:35.383+0000 7fbca4ff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:34:35.390 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:35.383+0000 7fbca4ff9700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:34:35.390 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-09-30T01:34:35.390 INFO:teuthology.orchestra.run.smithi088.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:34:35.393 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:35.387+0000 7fbcad8a2700 1 -- 172.21.15.88:0/285279241 >> v2:172.21.15.107:6812/29926 conn(0x7fbc94033360 msgr2=0x7fbc94035810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:34:35.393 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:35.387+0000 7fbcad8a2700 1 --2- 172.21.15.88:0/285279241 >> v2:172.21.15.107:6812/29926 conn(0x7fbc94033360 0x7fbc94035810 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fbc98009f20 tx=0x7fbc98018000 comp rx=0 tx=0).stop 2023-09-30T01:34:35.393 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:35.387+0000 7fbcad8a2700 10 monclient: shutdown 2023-09-30T01:34:35.393 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:35.387+0000 7fbcad8a2700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:34:35.393 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:35.387+0000 7fbcad8a2700 1 -- 172.21.15.88:0/285279241 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbca809ffd0 msgr2=0x7fbca811fad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:34:35.394 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:35.387+0000 7fbcad8a2700 1 --2- 172.21.15.88:0/285279241 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbca809ffd0 0x7fbca811fad0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fbc9c00c790 tx=0x7fbc9c007f40 comp rx=0 tx=0).stop 2023-09-30T01:34:35.394 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:35.387+0000 7fbcad8a2700 1 -- 172.21.15.88:0/285279241 shutdown_connections 2023-09-30T01:34:35.394 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:35.387+0000 7fbcad8a2700 1 --2- 172.21.15.88:0/285279241 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbca81386e0 0x7fbca8120550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:34:35.394 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:35.387+0000 7fbcad8a2700 1 --2- 172.21.15.88:0/285279241 >> v2:172.21.15.107:6812/29926 conn(0x7fbc94033360 0x7fbc94035810 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:34:35.394 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:35.387+0000 7fbcad8a2700 1 --2- 172.21.15.88:0/285279241 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7fbca80a3380 0x7fbca8120010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:34:35.394 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:35.387+0000 7fbcad8a2700 1 --2- 172.21.15.88:0/285279241 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbca809ffd0 0x7fbca811fad0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:34:35.394 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:35.387+0000 7fbcad8a2700 1 -- 172.21.15.88:0/285279241 >> 172.21.15.88:0/285279241 conn(0x7fbca806e5e0 msgr2=0x7fbca8098ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:34:35.394 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:35.387+0000 7fbcad8a2700 1 -- 172.21.15.88:0/285279241 shutdown_connections 2023-09-30T01:34:35.394 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:35.387+0000 7fbcad8a2700 1 -- 172.21.15.88:0/285279241 wait complete. 2023-09-30T01:34:38.409 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:34:38.521 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:38.515+0000 7f1d80ac9700 10 monclient: get_monmap_and_config 2023-09-30T01:34:38.521 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:38.515+0000 7f1d80ac9700 10 monclient: build_initial_monmap 2023-09-30T01:34:38.521 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:38.515+0000 7f1d80ac9700 10 monclient: monmap: 2023-09-30T01:34:38.521 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2023-09-30T01:34:38.522 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:34:38.522 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:34:38.521628+0000 2023-09-30T01:34:38.522 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:34:38.521628+0000 2023-09-30T01:34:38.522 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2023-09-30T01:34:38.522 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:34:38.522 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a-legacy 2023-09-30T01:34:38.522 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.107:6789/0 mon.noname-f-legacy 2023-09-30T01:34:38.522 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.noname-b 2023-09-30T01:34:38.522 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.noname-g 2023-09-30T01:34:38.522 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.noname-c 2023-09-30T01:34:38.522 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.noname-h 2023-09-30T01:34:38.522 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.noname-d 2023-09-30T01:34:38.523 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.noname-i 2023-09-30T01:34:38.523 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.noname-e 2023-09-30T01:34:38.523 INFO:teuthology.orchestra.run.smithi088.stderr:9: v2:172.21.15.88:3300/0 mon.noname-a 2023-09-30T01:34:38.523 INFO:teuthology.orchestra.run.smithi088.stderr:10: v2:172.21.15.107:3300/0 mon.noname-f 2023-09-30T01:34:38.523 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:34:38.531 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:38.515+0000 7f1d80ac9700 1 Processor -- start 2023-09-30T01:34:38.531 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:38.515+0000 7f1d80ac9700 1 -- start start 2023-09-30T01:34:38.531 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:38.515+0000 7f1d80ac9700 10 monclient: init 2023-09-30T01:34:38.531 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:38.515+0000 7f1d80ac9700 10 monclient: _reopen_session rank -1 2023-09-30T01:34:38.531 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:38.515+0000 7f1d80ac9700 10 monclient: _add_conns ranks=[8,7,6,3,2,1,4,5,10,9,0] 2023-09-30T01:34:38.531 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:38.515+0000 7f1d80ac9700 1 --2- >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f1d7c0966a0 0x7f1d7c096a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:34:38.531 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:38.515+0000 7f1d80ac9700 10 monclient(hunting): picked mon.noname-e con 0x7f1d7c0966a0 addr [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] 2023-09-30T01:34:38.531 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:38.515+0000 7f1d80ac9700 1 --2- >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7f1d7c097e40 0x7f1d7c09ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:34:38.532 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:38.515+0000 7f1d80ac9700 10 monclient(hunting): picked mon.noname-i con 0x7f1d7c097e40 addr [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] 2023-09-30T01:34:38.532 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:38.515+0000 7f1d80ac9700 1 --2- >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7f1d7c098210 0x7f1d7c09e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:34:38.532 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:38.515+0000 7f1d80ac9700 10 monclient(hunting): picked mon.noname-d con 0x7f1d7c098210 addr [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] 2023-09-30T01:34:38.532 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:38.515+0000 7f1d80ac9700 10 monclient(hunting): start opening mon connection 2023-09-30T01:34:38.532 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:38.515+0000 7f1d80ac9700 1 -- --> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] -- mon_getmap magic: 0 v1 -- 0x7f1d7c097740 con 0x7f1d7c098210 2023-09-30T01:34:38.532 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:38.515+0000 7f1d80ac9700 10 monclient(hunting): start opening mon connection 2023-09-30T01:34:38.532 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:38.515+0000 7f1d80ac9700 1 -- --> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] -- mon_getmap magic: 0 v1 -- 0x7f1d7c063510 con 0x7f1d7c097e40 2023-09-30T01:34:38.532 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:38.515+0000 7f1d80ac9700 10 monclient(hunting): start opening mon connection 2023-09-30T01:34:38.532 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:38.515+0000 7f1d80ac9700 1 -- --> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] -- mon_getmap magic: 0 v1 -- 0x7f1d7c063750 con 0x7f1d7c0966a0 2023-09-30T01:34:38.532 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:38.515+0000 7f1d80ac9700 10 monclient(hunting): _renew_subs 2023-09-30T01:34:38.532 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:38.515+0000 7f1d80ac9700 10 monclient(hunting): authenticate will time out at 950.559851s 2023-09-30T01:34:38.532 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:38.519+0000 7f1d7a59c700 1 --2- >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f1d7c0966a0 0x7f1d7c096a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:34:38.533 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:38.519+0000 7f1d7ad9d700 1 --2- >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7f1d7c098210 0x7f1d7c09e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:34:38.533 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:38.519+0000 7f1d7ad9d700 1 --2- >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7f1d7c098210 0x7f1d7c09e2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3303/0 says I am v2:172.21.15.88:36426/0 (socket says 172.21.15.88:36426) 2023-09-30T01:34:38.533 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:38.519+0000 7f1d7ad9d700 1 -- 172.21.15.88:0/2390345684 learned_addr learned my addr 172.21.15.88:0/2390345684 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-09-30T01:34:38.533 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:38.519+0000 7f1d7ad9d700 10 monclient(hunting): get_auth_request con 0x7f1d7c098210 auth_method 0 2023-09-30T01:34:38.533 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:38.519+0000 7f1d7ad9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:34:38.533 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:38.519+0000 7f1d7ad9d700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:34:38.533 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:38.519+0000 7f1d7ad9d700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:34:38.533 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:38.519+0000 7f1d7a59c700 1 --2- >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f1d7c0966a0 0x7f1d7c096a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3304/0 says I am v2:172.21.15.88:57932/0 (socket says 172.21.15.88:57932) 2023-09-30T01:34:38.533 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:38.519+0000 7f1d7a59c700 10 monclient(hunting): get_auth_request con 0x7f1d7c0966a0 auth_method 0 2023-09-30T01:34:38.533 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:38.519+0000 7f1d7a59c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:34:38.533 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:38.519+0000 7f1d7a59c700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:34:38.533 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:38.519+0000 7f1d7a59c700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:34:38.534 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:38.519+0000 7f1d79d9b700 1 --2- 172.21.15.88:0/2390345684 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7f1d7c097e40 0x7f1d7c09ddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:34:38.534 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:38.519+0000 7f1d79d9b700 10 monclient(hunting): get_auth_request con 0x7f1d7c097e40 auth_method 0 2023-09-30T01:34:38.534 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:38.519+0000 7f1d79d9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:34:38.534 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:38.519+0000 7f1d79d9b700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:34:38.534 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:38.519+0000 7f1d79d9b700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:34:38.534 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:38.519+0000 7f1d7a59c700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:34:38.534 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:38.519+0000 7f1d7a59c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:34:38.534 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:38.519+0000 7f1d7a59c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:34:38.534 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:38.519+0000 7f1d7ad9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:34:38.534 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:38.519+0000 7f1d7ad9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:34:38.534 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:38.519+0000 7f1d7ad9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:34:38.535 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:38.519+0000 7f1d79d9b700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:34:38.535 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:38.519+0000 7f1d79d9b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:34:38.535 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:38.519+0000 7f1d79d9b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:34:38.535 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:38.519+0000 7f1d7ad9d700 10 monclient(hunting): handle_auth_done global_id 4425 payload 274 2023-09-30T01:34:38.535 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:38.519+0000 7f1d7ad9d700 1 -- 172.21.15.88:0/2390345684 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f1d7c0966a0 msgr2=0x7f1d7c096a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:34:38.535 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:38.519+0000 7f1d7ad9d700 1 --2- 172.21.15.88:0/2390345684 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f1d7c0966a0 0x7f1d7c096a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:34:38.535 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:38.519+0000 7f1d7ad9d700 1 -- 172.21.15.88:0/2390345684 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7f1d7c097e40 msgr2=0x7f1d7c09ddb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:34:38.535 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:38.519+0000 7f1d7ad9d700 1 --2- 172.21.15.88:0/2390345684 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7f1d7c097e40 0x7f1d7c09ddb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:34:38.535 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:38.519+0000 7f1d7ad9d700 10 monclient: _finish_hunting 0 2023-09-30T01:34:38.535 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:38.519+0000 7f1d7ad9d700 1 monclient: found mon.noname-d 2023-09-30T01:34:38.535 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:38.519+0000 7f1d7ad9d700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:34:38.535 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:38.519+0000 7f1d7ad9d700 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.88:3303/0 2023-09-30T01:34:38.536 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:38.519+0000 7f1d7ad9d700 1 -- 172.21.15.88:0/2390345684 --> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d7c05ecb0 con 0x7f1d7c098210 2023-09-30T01:34:38.536 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:38.519+0000 7f1d7a59c700 1 --2- 172.21.15.88:0/2390345684 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f1d7c0966a0 0x7f1d7c096a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:34:38.536 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:38.519+0000 7f1d7ad9d700 1 --2- 172.21.15.88:0/2390345684 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7f1d7c098210 0x7f1d7c09e2f0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f1d74009f50 tx=0x7f1d74009840 comp rx=0 tx=0).ready entity=mon.6 client_cookie=a9ca8e5fbb6ac175 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:34:38.536 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:38.519+0000 7f1d7959a700 1 -- 172.21.15.88:0/2390345684 <== mon.6 v2:172.21.15.88:3303/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f1d7400e070 con 0x7f1d7c098210 2023-09-30T01:34:38.536 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:38.519+0000 7f1d7959a700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:34:38.536 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:38.519+0000 7f1d7959a700 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2023-09-30T01:34:38.536 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:38.519+0000 7f1d7959a700 10 monclient: dump: 2023-09-30T01:34:38.536 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:34:38.536 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:34:38.536 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:34:38.536 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:34:38.537 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:34:38.537 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:34:38.537 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:34:38.537 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:34:38.537 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:34:38.537 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:34:38.537 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:34:38.537 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:34:38.537 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:34:38.537 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:34:38.537 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:34:38.538 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:34:38.538 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:38.519+0000 7f1d7959a700 10 monclient: _finish_auth 0 2023-09-30T01:34:38.538 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:38.519+0000 7f1d7959a700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:34:38.538 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:38.519+0000 7f1d80ac9700 5 monclient: authenticate success, global_id 4425 2023-09-30T01:34:38.538 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:38.519+0000 7f1d80ac9700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:34:38.538 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:38.519+0000 7f1d7959a700 1 -- 172.21.15.88:0/2390345684 <== mon.6 v2:172.21.15.88:3303/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d7400bdd0 con 0x7f1d7c098210 2023-09-30T01:34:38.538 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:38.519+0000 7f1d7959a700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:34:38.538 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:38.519+0000 7f1d80ac9700 10 monclient: get_monmap_and_config success 2023-09-30T01:34:38.538 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:38.519+0000 7f1d80ac9700 4 set_mon_vals no callback set 2023-09-30T01:34:38.538 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:38.519+0000 7f1d7959a700 1 -- 172.21.15.88:0/2390345684 <== mon.6 v2:172.21.15.88:3303/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f1d740092e0 con 0x7f1d7c098210 2023-09-30T01:34:38.538 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:38.519+0000 7f1d7959a700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:34:38.539 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:38.519+0000 7f1d7959a700 10 monclient: got monmap 1 from mon.d (according to old e1) 2023-09-30T01:34:38.539 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:38.519+0000 7f1d7959a700 10 monclient: dump: 2023-09-30T01:34:38.539 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:34:38.539 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:34:38.539 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:34:38.539 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:34:38.539 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:34:38.539 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:34:38.539 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:34:38.539 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:34:38.539 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:34:38.539 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:34:38.540 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:34:38.540 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:34:38.540 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:34:38.540 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:34:38.540 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:34:38.540 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:34:38.540 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:38.519+0000 7f1d80ac9700 10 monclient: shutdown 2023-09-30T01:34:38.540 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:38.519+0000 7f1d80ac9700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:34:38.540 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:38.519+0000 7f1d80ac9700 1 -- 172.21.15.88:0/2390345684 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7f1d7c098210 msgr2=0x7f1d7c09e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:34:38.540 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:38.519+0000 7f1d80ac9700 1 --2- 172.21.15.88:0/2390345684 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7f1d7c098210 0x7f1d7c09e2f0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f1d74009f50 tx=0x7f1d74009840 comp rx=0 tx=0).stop 2023-09-30T01:34:38.540 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:38.519+0000 7f1d80ac9700 1 -- 172.21.15.88:0/2390345684 shutdown_connections 2023-09-30T01:34:38.540 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:38.519+0000 7f1d80ac9700 1 --2- 172.21.15.88:0/2390345684 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7f1d7c098210 0x7f1d7c09e2f0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:34:38.541 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:38.519+0000 7f1d80ac9700 1 --2- 172.21.15.88:0/2390345684 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7f1d7c097e40 0x7f1d7c09ddb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:34:38.541 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:38.519+0000 7f1d80ac9700 1 --2- 172.21.15.88:0/2390345684 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f1d7c0966a0 0x7f1d7c096a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:34:38.541 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:38.519+0000 7f1d80ac9700 1 -- 172.21.15.88:0/2390345684 >> 172.21.15.88:0/2390345684 conn(0x7f1d7c0a3440 msgr2=0x7f1d7c0a3840 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:34:38.541 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:38.519+0000 7f1d80ac9700 1 -- 172.21.15.88:0/2390345684 shutdown_connections 2023-09-30T01:34:38.541 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:38.519+0000 7f1d80ac9700 1 -- 172.21.15.88:0/2390345684 wait complete. 2023-09-30T01:34:38.541 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:38.519+0000 7f1d80ac9700 10 monclient: build_initial_monmap 2023-09-30T01:34:38.541 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:38.519+0000 7f1d80ac9700 10 monclient: monmap: 2023-09-30T01:34:38.541 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2023-09-30T01:34:38.541 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:34:38.541 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 0.000000 2023-09-30T01:34:38.541 INFO:teuthology.orchestra.run.smithi088.stderr:created 0.000000 2023-09-30T01:34:38.542 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2023-09-30T01:34:38.542 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 1 2023-09-30T01:34:38.542 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.noname-a 2023-09-30T01:34:38.542 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.noname-f 2023-09-30T01:34:38.542 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.noname-b 2023-09-30T01:34:38.542 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.noname-g 2023-09-30T01:34:38.542 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.noname-c 2023-09-30T01:34:38.542 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.noname-h 2023-09-30T01:34:38.542 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.noname-d 2023-09-30T01:34:38.542 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.noname-i 2023-09-30T01:34:38.542 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.noname-e 2023-09-30T01:34:38.542 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:34:38.543 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:38.519+0000 7f1d80ac9700 1 Processor -- start 2023-09-30T01:34:38.543 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:38.519+0000 7f1d80ac9700 1 -- start start 2023-09-30T01:34:38.543 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:38.519+0000 7f1d80ac9700 10 monclient: init 2023-09-30T01:34:38.543 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:38.519+0000 7f1d80ac9700 10 monclient: _reopen_session rank -1 2023-09-30T01:34:38.543 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:38.519+0000 7f1d80ac9700 10 monclient: _add_conns ranks=[6,3,7,2,8,4,5,1,0] 2023-09-30T01:34:38.543 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:38.519+0000 7f1d80ac9700 1 --2- >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7f1d7c0966a0 0x7f1d7c142940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:34:38.543 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:38.519+0000 7f1d80ac9700 10 monclient(hunting): picked mon.noname-d con 0x7f1d7c0966a0 addr [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] 2023-09-30T01:34:38.543 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:38.519+0000 7f1d80ac9700 1 --2- >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7f1d7c097e40 0x7f1d7c142e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:34:38.543 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:38.519+0000 7f1d80ac9700 10 monclient(hunting): picked mon.noname-g con 0x7f1d7c097e40 addr [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] 2023-09-30T01:34:38.543 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:38.519+0000 7f1d80ac9700 1 --2- >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7f1d7c098210 0x7f1d7c08e060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:34:38.543 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:38.519+0000 7f1d80ac9700 10 monclient(hunting): picked mon.noname-i con 0x7f1d7c098210 addr [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] 2023-09-30T01:34:38.544 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:38.519+0000 7f1d80ac9700 10 monclient(hunting): start opening mon connection 2023-09-30T01:34:38.544 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:38.519+0000 7f1d80ac9700 1 -- --> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1d7c09f780 con 0x7f1d7c097e40 2023-09-30T01:34:38.544 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:38.519+0000 7f1d80ac9700 10 monclient(hunting): start opening mon connection 2023-09-30T01:34:38.544 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:38.519+0000 7f1d80ac9700 1 -- --> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] -- mon_getmap magic: 0 v1 -- 0x7f1d7c0917c0 con 0x7f1d7c0966a0 2023-09-30T01:34:38.544 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:38.523+0000 7f1d80ac9700 10 monclient(hunting): start opening mon connection 2023-09-30T01:34:38.544 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:38.523+0000 7f1d80ac9700 1 -- --> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] -- mon_getmap magic: 0 v1 -- 0x7f1d7c091aa0 con 0x7f1d7c098210 2023-09-30T01:34:38.544 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:38.523+0000 7f1d80ac9700 10 monclient(hunting): _renew_subs 2023-09-30T01:34:38.544 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:38.523+0000 7f1d7a59c700 1 --2- >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7f1d7c0966a0 0x7f1d7c142940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:34:38.544 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:38.523+0000 7f1d80ac9700 10 monclient(hunting): authenticate will time out at 950.564032s 2023-09-30T01:34:38.544 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:38.523+0000 7f1d7a59c700 1 --2- >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7f1d7c0966a0 0x7f1d7c142940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3303/0 says I am v2:172.21.15.88:36442/0 (socket says 172.21.15.88:36442) 2023-09-30T01:34:38.544 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:38.523+0000 7f1d7a59c700 1 -- 172.21.15.88:0/1442002004 learned_addr learned my addr 172.21.15.88:0/1442002004 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-09-30T01:34:38.545 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:38.523+0000 7f1d7a59c700 10 monclient(hunting): get_auth_request con 0x7f1d7c0966a0 auth_method 0 2023-09-30T01:34:38.545 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:38.523+0000 7f1d7a59c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:34:38.545 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:38.523+0000 7f1d7a59c700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:34:38.545 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:38.523+0000 7f1d7a59c700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:34:38.545 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:38.523+0000 7f1d7a59c700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:34:38.545 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:38.523+0000 7f1d7a59c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:34:38.545 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:38.523+0000 7f1d7a59c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:34:38.545 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:38.523+0000 7f1d79d9b700 1 --2- 172.21.15.88:0/1442002004 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7f1d7c097e40 0x7f1d7c142e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:34:38.545 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:38.523+0000 7f1d79d9b700 10 monclient(hunting): get_auth_request con 0x7f1d7c097e40 auth_method 0 2023-09-30T01:34:38.545 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:38.523+0000 7f1d79d9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:34:38.545 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:38.523+0000 7f1d79d9b700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:34:38.546 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:38.523+0000 7f1d79d9b700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:34:38.546 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:38.523+0000 7f1d7ad9d700 1 --2- 172.21.15.88:0/1442002004 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7f1d7c098210 0x7f1d7c08e060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:34:38.546 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:38.523+0000 7f1d7ad9d700 10 monclient(hunting): get_auth_request con 0x7f1d7c098210 auth_method 0 2023-09-30T01:34:38.546 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:38.523+0000 7f1d7ad9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:34:38.546 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:38.523+0000 7f1d7ad9d700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:34:38.546 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:38.523+0000 7f1d7ad9d700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:34:38.546 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:38.523+0000 7f1d7a59c700 10 monclient(hunting): handle_auth_done global_id 4434 payload 824 2023-09-30T01:34:38.546 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:38.523+0000 7f1d7a59c700 1 -- 172.21.15.88:0/1442002004 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7f1d7c098210 msgr2=0x7f1d7c08e060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:34:38.546 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:38.523+0000 7f1d7a59c700 1 --2- 172.21.15.88:0/1442002004 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7f1d7c098210 0x7f1d7c08e060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:34:38.546 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:38.523+0000 7f1d7a59c700 1 -- 172.21.15.88:0/1442002004 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7f1d7c097e40 msgr2=0x7f1d7c142e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:34:38.546 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:38.523+0000 7f1d7a59c700 1 --2- 172.21.15.88:0/1442002004 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7f1d7c097e40 0x7f1d7c142e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:34:38.547 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:38.523+0000 7f1d7a59c700 10 monclient: _finish_hunting 0 2023-09-30T01:34:38.547 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:38.523+0000 7f1d7a59c700 1 monclient: found mon.noname-d 2023-09-30T01:34:38.547 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:38.523+0000 7f1d7a59c700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:34:38.547 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:38.523+0000 7f1d7a59c700 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.88:3303/0 2023-09-30T01:34:38.547 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:38.523+0000 7f1d7a59c700 1 -- 172.21.15.88:0/1442002004 --> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d6c009420 con 0x7f1d7c0966a0 2023-09-30T01:34:38.547 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:38.523+0000 7f1d79d9b700 1 --2- 172.21.15.88:0/1442002004 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7f1d7c097e40 0x7f1d7c142e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-30T01:34:38.547 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:38.523+0000 7f1d7a59c700 1 --2- 172.21.15.88:0/1442002004 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7f1d7c0966a0 0x7f1d7c142940 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f1d6c00ec80 tx=0x7f1d6c0091a0 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:34:38.547 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:38.523+0000 7f1d6a7fc700 1 -- 172.21.15.88:0/1442002004 <== mon.6 v2:172.21.15.88:3303/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f1d6c00ce50 con 0x7f1d7c0966a0 2023-09-30T01:34:38.547 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:38.523+0000 7f1d6a7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:34:38.547 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:38.523+0000 7f1d6a7fc700 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2023-09-30T01:34:38.547 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:38.523+0000 7f1d6a7fc700 10 monclient: dump: 2023-09-30T01:34:38.547 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:34:38.548 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:34:38.548 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:34:38.548 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:34:38.548 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:34:38.548 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:34:38.548 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:34:38.548 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:34:38.548 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:34:38.548 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:34:38.548 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:34:38.548 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:34:38.549 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:34:38.549 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:34:38.549 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:34:38.549 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:34:38.549 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:38.523+0000 7f1d6a7fc700 10 monclient: _finish_auth 0 2023-09-30T01:34:38.549 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:38.523+0000 7f1d6a7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:34:38.549 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:38.523+0000 7f1d80ac9700 5 monclient: authenticate success, global_id 4434 2023-09-30T01:34:38.549 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:38.523+0000 7f1d6a7fc700 1 -- 172.21.15.88:0/1442002004 <== mon.6 v2:172.21.15.88:3303/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d6c013470 con 0x7f1d7c0966a0 2023-09-30T01:34:38.549 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:38.523+0000 7f1d6a7fc700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:34:38.549 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:38.523+0000 7f1d6a7fc700 1 -- 172.21.15.88:0/1442002004 <== mon.6 v2:172.21.15.88:3303/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f1d6c013a60 con 0x7f1d7c0966a0 2023-09-30T01:34:38.549 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:38.523+0000 7f1d6a7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:34:38.549 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:38.523+0000 7f1d78d99700 4 set_mon_vals no callback set 2023-09-30T01:34:38.550 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:38.523+0000 7f1d6a7fc700 10 monclient: got monmap 1 from mon.d (according to old e1) 2023-09-30T01:34:38.550 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:38.523+0000 7f1d6a7fc700 10 monclient: dump: 2023-09-30T01:34:38.550 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:34:38.550 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:34:38.550 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:34:38.550 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:34:38.550 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:34:38.550 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:34:38.550 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:34:38.550 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:34:38.550 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:34:38.551 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:34:38.551 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:34:38.551 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:34:38.551 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:34:38.551 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:34:38.551 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:34:38.551 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:34:38.551 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:38.523+0000 7f1d80ac9700 10 monclient: _renew_subs 2023-09-30T01:34:38.551 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:38.523+0000 7f1d80ac9700 10 monclient: _send_mon_message to mon.d at v2:172.21.15.88:3303/0 2023-09-30T01:34:38.551 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:38.523+0000 7f1d80ac9700 1 -- 172.21.15.88:0/1442002004 --> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1d74009ae0 con 0x7f1d7c0966a0 2023-09-30T01:34:38.551 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:38.523+0000 7f1d80ac9700 10 monclient: _renew_subs 2023-09-30T01:34:38.551 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:38.523+0000 7f1d80ac9700 10 monclient: _send_mon_message to mon.d at v2:172.21.15.88:3303/0 2023-09-30T01:34:38.552 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:38.523+0000 7f1d80ac9700 1 -- 172.21.15.88:0/1442002004 --> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1d7c08fc90 con 0x7f1d7c0966a0 2023-09-30T01:34:38.552 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:38.523+0000 7f1d6a7fc700 1 -- 172.21.15.88:0/1442002004 <== mon.6 v2:172.21.15.88:3303/0 4 ==== mgrmap(e 5) v1 ==== 40144+0+0 (secure 0 0 0) 0x7f1d6c013bc0 con 0x7f1d7c0966a0 2023-09-30T01:34:38.552 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:38.523+0000 7f1d80ac9700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:34:38.552 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:38.523+0000 7f1d80ac9700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:34:38.552 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:38.523+0000 7f1d80ac9700 10 monclient: _send_mon_message to mon.d at v2:172.21.15.88:3303/0 2023-09-30T01:34:38.552 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:38.523+0000 7f1d80ac9700 1 -- 172.21.15.88:0/1442002004 --> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1d7c071960 con 0x7f1d7c0966a0 2023-09-30T01:34:38.552 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:38.523+0000 7f1d6a7fc700 1 --2- 172.21.15.88:0/1442002004 >> v2:172.21.15.107:6812/29926 conn(0x7f1d5c033360 0x7f1d5c035810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:34:38.552 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:38.523+0000 7f1d6a7fc700 1 -- 172.21.15.88:0/1442002004 <== mon.6 v2:172.21.15.88:3303/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 3502+0+0 (secure 0 0 0) 0x7f1d6c043b90 con 0x7f1d7c0966a0 2023-09-30T01:34:38.552 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:38.523+0000 7f1d79d9b700 1 --2- 172.21.15.88:0/1442002004 >> v2:172.21.15.107:6812/29926 conn(0x7f1d5c033360 0x7f1d5c035810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:34:38.552 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:38.523+0000 7f1d79d9b700 10 monclient: get_auth_request con 0x7f1d5c033360 auth_method 0 2023-09-30T01:34:38.552 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:38.523+0000 7f1d79d9b700 1 --2- 172.21.15.88:0/1442002004 >> v2:172.21.15.107:6812/29926 conn(0x7f1d5c033360 0x7f1d5c035810 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f1d70009d80 tx=0x7f1d70008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:34:38.553 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:38.531+0000 7f1d6a7fc700 1 -- 172.21.15.88:0/1442002004 <== mon.6 v2:172.21.15.88:3303/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f1d6c0138c0 con 0x7f1d7c0966a0 2023-09-30T01:34:38.553 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:38.531+0000 7f1d6a7fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:34:38.553 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:38.531+0000 7f1d6a7fc700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:34:38.732 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:38.727+0000 7f1d80ac9700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T01:34:38.759 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:38.727+0000 7f1d80ac9700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:34:38.759 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:38.727+0000 7f1d80ac9700 10 monclient: _send_mon_message to mon.d at v2:172.21.15.88:3303/0 2023-09-30T01:34:38.759 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:38.727+0000 7f1d80ac9700 1 -- 172.21.15.88:0/1442002004 --> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f1d7c099590 con 0x7f1d7c0966a0 2023-09-30T01:34:38.759 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:38.727+0000 7f1d6a7fc700 1 -- 172.21.15.88:0/1442002004 <== mon.6 v2:172.21.15.88:3303/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f1d6c01b3b0 con 0x7f1d7c0966a0 2023-09-30T01:34:38.759 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:38.727+0000 7f1d6a7fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:34:38.760 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:38.727+0000 7f1d6a7fc700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:34:38.760 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-09-30T01:34:38.760 INFO:teuthology.orchestra.run.smithi088.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:34:38.762 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:38.727+0000 7f1d80ac9700 1 -- 172.21.15.88:0/1442002004 >> v2:172.21.15.107:6812/29926 conn(0x7f1d5c033360 msgr2=0x7f1d5c035810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:34:38.762 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:38.727+0000 7f1d80ac9700 1 --2- 172.21.15.88:0/1442002004 >> v2:172.21.15.107:6812/29926 conn(0x7f1d5c033360 0x7f1d5c035810 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f1d70009d80 tx=0x7f1d70008040 comp rx=0 tx=0).stop 2023-09-30T01:34:38.762 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:38.727+0000 7f1d80ac9700 10 monclient: shutdown 2023-09-30T01:34:38.762 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:38.727+0000 7f1d80ac9700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:34:38.762 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:38.727+0000 7f1d80ac9700 1 -- 172.21.15.88:0/1442002004 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7f1d7c0966a0 msgr2=0x7f1d7c142940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:34:38.762 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:38.727+0000 7f1d80ac9700 1 --2- 172.21.15.88:0/1442002004 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7f1d7c0966a0 0x7f1d7c142940 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f1d6c00ec80 tx=0x7f1d6c0091a0 comp rx=0 tx=0).stop 2023-09-30T01:34:38.762 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:38.727+0000 7f1d80ac9700 1 -- 172.21.15.88:0/1442002004 shutdown_connections 2023-09-30T01:34:38.762 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:38.727+0000 7f1d80ac9700 1 --2- 172.21.15.88:0/1442002004 >> v2:172.21.15.107:6812/29926 conn(0x7f1d5c033360 0x7f1d5c035810 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:34:38.762 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:38.727+0000 7f1d80ac9700 1 --2- 172.21.15.88:0/1442002004 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7f1d7c098210 0x7f1d7c08e060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:34:38.762 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:38.727+0000 7f1d80ac9700 1 --2- 172.21.15.88:0/1442002004 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7f1d7c097e40 0x7f1d7c142e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:34:38.763 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:38.727+0000 7f1d80ac9700 1 --2- 172.21.15.88:0/1442002004 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7f1d7c0966a0 0x7f1d7c142940 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:34:38.763 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:38.727+0000 7f1d80ac9700 1 -- 172.21.15.88:0/1442002004 >> 172.21.15.88:0/1442002004 conn(0x7f1d7c0a3440 msgr2=0x7f1d7c0a1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:34:38.763 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:38.727+0000 7f1d80ac9700 1 -- 172.21.15.88:0/1442002004 shutdown_connections 2023-09-30T01:34:38.763 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:38.727+0000 7f1d80ac9700 1 -- 172.21.15.88:0/1442002004 wait complete. 2023-09-30T01:34:41.764 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:34:41.876 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:41.871+0000 7f45e4125700 10 monclient: get_monmap_and_config 2023-09-30T01:34:41.876 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:41.871+0000 7f45e4125700 10 monclient: build_initial_monmap 2023-09-30T01:34:41.877 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:41.871+0000 7f45e4125700 10 monclient: monmap: 2023-09-30T01:34:41.877 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2023-09-30T01:34:41.877 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:34:41.877 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:34:41.876757+0000 2023-09-30T01:34:41.877 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:34:41.876757+0000 2023-09-30T01:34:41.877 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2023-09-30T01:34:41.877 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:34:41.878 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a-legacy 2023-09-30T01:34:41.878 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.107:6789/0 mon.noname-f-legacy 2023-09-30T01:34:41.878 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.noname-b 2023-09-30T01:34:41.878 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.noname-g 2023-09-30T01:34:41.878 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.noname-c 2023-09-30T01:34:41.878 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.noname-h 2023-09-30T01:34:41.878 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.noname-d 2023-09-30T01:34:41.879 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.noname-i 2023-09-30T01:34:41.879 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.noname-e 2023-09-30T01:34:41.879 INFO:teuthology.orchestra.run.smithi088.stderr:9: v2:172.21.15.88:3300/0 mon.noname-a 2023-09-30T01:34:41.879 INFO:teuthology.orchestra.run.smithi088.stderr:10: v2:172.21.15.107:3300/0 mon.noname-f 2023-09-30T01:34:41.879 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:34:41.888 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:41.871+0000 7f45e4125700 1 Processor -- start 2023-09-30T01:34:41.888 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:41.871+0000 7f45e4125700 1 -- start start 2023-09-30T01:34:41.888 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:41.871+0000 7f45e4125700 10 monclient: init 2023-09-30T01:34:41.888 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:41.871+0000 7f45e4125700 10 monclient: _reopen_session rank -1 2023-09-30T01:34:41.888 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:41.871+0000 7f45e4125700 10 monclient: _add_conns ranks=[2,6,0,1,3,7,9,4,10,8,5] 2023-09-30T01:34:41.889 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:41.871+0000 7f45e4125700 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f45dc0a35f0 0x7f45dc133330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:34:41.889 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:41.871+0000 7f45e4125700 10 monclient(hunting): picked mon.noname-b con 0x7f45dc0a35f0 addr [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] 2023-09-30T01:34:41.889 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:41.871+0000 7f45e4125700 1 --2- >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7f45dc133870 0x7f45dc133c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:34:41.889 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:41.871+0000 7f45e4125700 10 monclient(hunting): picked mon.noname-d con 0x7f45dc133870 addr [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] 2023-09-30T01:34:41.889 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:41.871+0000 7f45e4125700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f45dc0a46f0 addr v1:172.21.15.88:6789/0 2023-09-30T01:34:41.889 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:41.871+0000 7f45e4125700 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45dc05ecb0 con 0x7f45dc0a46f0 2023-09-30T01:34:41.889 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:41.871+0000 7f45e4125700 10 monclient(hunting): start opening mon connection 2023-09-30T01:34:41.889 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:41.871+0000 7f45e4125700 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45dc097ad0 con 0x7f45dc0a35f0 2023-09-30T01:34:41.889 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:41.871+0000 7f45e4125700 10 monclient(hunting): start opening mon connection 2023-09-30T01:34:41.889 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:41.871+0000 7f45e4125700 1 -- --> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] -- mon_getmap magic: 0 v1 -- 0x7f45dc1341c0 con 0x7f45dc133870 2023-09-30T01:34:41.889 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:41.871+0000 7f45e4125700 10 monclient(hunting): _renew_subs 2023-09-30T01:34:41.889 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:41.871+0000 7f45e4125700 10 monclient(hunting): authenticate will time out at 953.914288s 2023-09-30T01:34:41.890 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:41.871+0000 7f45e26c2700 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f45dc0a46f0 0x7f45dc0a4b60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:57952/0 (socket says 172.21.15.88:57952) 2023-09-30T01:34:41.890 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:41.871+0000 7f45e1ec1700 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f45dc0a35f0 0x7f45dc133330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:34:41.890 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:41.871+0000 7f45e26c2700 1 -- 172.21.15.88:0/1241314900 learned_addr learned my addr 172.21.15.88:0/1241314900 (peer_addr_for_me v1:172.21.15.88:0/0) 2023-09-30T01:34:41.890 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:41.871+0000 7f45e16c0700 1 --2- >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7f45dc133870 0x7f45dc133c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:34:41.890 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:41.871+0000 7f45e16c0700 10 monclient(hunting): get_auth_request con 0x7f45dc133870 auth_method 0 2023-09-30T01:34:41.890 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:41.871+0000 7f45e16c0700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:34:41.890 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:41.871+0000 7f45e16c0700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:34:41.890 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:41.871+0000 7f45e16c0700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:34:41.890 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:41.871+0000 7f45e1ec1700 10 monclient(hunting): get_auth_request con 0x7f45dc0a35f0 auth_method 0 2023-09-30T01:34:41.890 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:41.871+0000 7f45e1ec1700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:34:41.890 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:41.871+0000 7f45e1ec1700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:34:41.891 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:41.871+0000 7f45e1ec1700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:34:41.891 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:41.871+0000 7f45e16c0700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:34:41.891 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:41.871+0000 7f45e16c0700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:34:41.891 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:41.871+0000 7f45e16c0700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:34:41.891 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:41.871+0000 7f45e1ec1700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:34:41.891 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:41.871+0000 7f45e1ec1700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:34:41.891 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:41.871+0000 7f45e0ebf700 1 -- 172.21.15.88:0/1241314900 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 158817504 0 0) 0x7f45dc05ecb0 con 0x7f45dc0a46f0 2023-09-30T01:34:41.891 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:41.871+0000 7f45e1ec1700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:34:41.891 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:41.871+0000 7f45e0ebf700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:34:41.891 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:41.871+0000 7f45e0ebf700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:34:41.891 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:41.871+0000 7f45e0ebf700 10 monclient(hunting): my global_id is 4671 2023-09-30T01:34:41.892 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:41.871+0000 7f45e0ebf700 1 -- 172.21.15.88:0/1241314900 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f45c8003570 con 0x7f45dc0a46f0 2023-09-30T01:34:41.892 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:41.871+0000 7f45e16c0700 10 monclient(hunting): handle_auth_done global_id 4443 payload 274 2023-09-30T01:34:41.892 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:41.871+0000 7f45e16c0700 1 -- 172.21.15.88:0/1241314900 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f45dc0a35f0 msgr2=0x7f45dc133330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:34:41.892 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:41.871+0000 7f45e16c0700 1 --2- 172.21.15.88:0/1241314900 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f45dc0a35f0 0x7f45dc133330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:34:41.892 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:41.871+0000 7f45e16c0700 1 -- 172.21.15.88:0/1241314900 >> v1:172.21.15.88:6789/0 conn(0x7f45dc0a46f0 legacy=0x7f45dc0a4b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:34:41.892 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:41.871+0000 7f45e16c0700 10 monclient: _finish_hunting 0 2023-09-30T01:34:41.892 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:41.871+0000 7f45e16c0700 1 monclient: found mon.noname-d 2023-09-30T01:34:41.892 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:41.871+0000 7f45e16c0700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:34:41.892 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:41.871+0000 7f45e16c0700 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.88:3303/0 2023-09-30T01:34:41.892 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:41.871+0000 7f45e16c0700 1 -- 172.21.15.88:0/1241314900 --> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45dc0a0940 con 0x7f45dc133870 2023-09-30T01:34:41.892 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:41.871+0000 7f45e16c0700 1 --2- 172.21.15.88:0/1241314900 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7f45dc133870 0x7f45dc133c80 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f45cc009910 tx=0x7f45cc0091a0 comp rx=0 tx=0).ready entity=mon.6 client_cookie=eb015e68a1b886e6 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:34:41.892 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:41.875+0000 7f45e0ebf700 1 -- 172.21.15.88:0/1241314900 <== mon.6 v2:172.21.15.88:3303/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f45cc011ab0 con 0x7f45dc133870 2023-09-30T01:34:41.893 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:41.875+0000 7f45e0ebf700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:34:41.893 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:41.875+0000 7f45e0ebf700 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2023-09-30T01:34:41.893 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:41.875+0000 7f45e0ebf700 10 monclient: dump: 2023-09-30T01:34:41.893 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:34:41.893 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:34:41.893 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:34:41.893 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:34:41.893 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:34:41.893 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:34:41.893 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:34:41.894 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:34:41.894 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:34:41.894 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:34:41.894 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:34:41.894 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:34:41.894 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:34:41.894 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:34:41.894 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:34:41.894 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:34:41.894 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:41.875+0000 7f45e0ebf700 10 monclient: _finish_auth 0 2023-09-30T01:34:41.894 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:41.875+0000 7f45e0ebf700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:34:41.895 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:41.875+0000 7f45e0ebf700 1 -- 172.21.15.88:0/1241314900 <== mon.6 v2:172.21.15.88:3303/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f45cc011c30 con 0x7f45dc133870 2023-09-30T01:34:41.895 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:41.875+0000 7f45e4125700 5 monclient: authenticate success, global_id 4443 2023-09-30T01:34:41.895 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:41.875+0000 7f45e4125700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:34:41.895 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:41.875+0000 7f45e0ebf700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:34:41.895 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:41.875+0000 7f45e0ebf700 1 -- 172.21.15.88:0/1241314900 <== mon.6 v2:172.21.15.88:3303/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f45cc011da0 con 0x7f45dc133870 2023-09-30T01:34:41.895 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:41.875+0000 7f45e4125700 10 monclient: get_monmap_and_config success 2023-09-30T01:34:41.895 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:41.875+0000 7f45e4125700 4 set_mon_vals no callback set 2023-09-30T01:34:41.895 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:41.875+0000 7f45e0ebf700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:34:41.895 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:41.875+0000 7f45e0ebf700 10 monclient: got monmap 1 from mon.d (according to old e1) 2023-09-30T01:34:41.895 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:41.875+0000 7f45e0ebf700 10 monclient: dump: 2023-09-30T01:34:41.895 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:34:41.896 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:34:41.896 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:34:41.896 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:34:41.896 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:34:41.896 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:34:41.896 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:34:41.896 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:34:41.896 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:34:41.896 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:34:41.896 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:34:41.896 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:34:41.897 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:34:41.897 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:34:41.897 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:34:41.897 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:34:41.897 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:41.875+0000 7f45e4125700 10 monclient: shutdown 2023-09-30T01:34:41.897 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:41.875+0000 7f45e4125700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:34:41.897 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:41.875+0000 7f45e4125700 1 -- 172.21.15.88:0/1241314900 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7f45dc133870 msgr2=0x7f45dc133c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:34:41.897 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:41.875+0000 7f45e4125700 1 --2- 172.21.15.88:0/1241314900 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7f45dc133870 0x7f45dc133c80 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f45cc009910 tx=0x7f45cc0091a0 comp rx=0 tx=0).stop 2023-09-30T01:34:41.897 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:41.875+0000 7f45e4125700 1 -- 172.21.15.88:0/1241314900 shutdown_connections 2023-09-30T01:34:41.897 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:41.875+0000 7f45e4125700 1 --2- 172.21.15.88:0/1241314900 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7f45dc133870 0x7f45dc133c80 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:34:41.897 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:41.875+0000 7f45e4125700 1 --2- 172.21.15.88:0/1241314900 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f45dc0a35f0 0x7f45dc133330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:34:41.898 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:41.875+0000 7f45e4125700 1 -- 172.21.15.88:0/1241314900 >> 172.21.15.88:0/1241314900 conn(0x7f45dc06e5e0 msgr2=0x7f45dc06e9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:34:41.898 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:41.875+0000 7f45e4125700 1 -- 172.21.15.88:0/1241314900 shutdown_connections 2023-09-30T01:34:41.898 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:41.875+0000 7f45e4125700 1 -- 172.21.15.88:0/1241314900 wait complete. 2023-09-30T01:34:41.898 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:41.875+0000 7f45e4125700 10 monclient: build_initial_monmap 2023-09-30T01:34:41.898 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:41.875+0000 7f45e4125700 10 monclient: monmap: 2023-09-30T01:34:41.898 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2023-09-30T01:34:41.898 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:34:41.898 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 0.000000 2023-09-30T01:34:41.898 INFO:teuthology.orchestra.run.smithi088.stderr:created 0.000000 2023-09-30T01:34:41.898 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2023-09-30T01:34:41.898 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 1 2023-09-30T01:34:41.899 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.noname-a 2023-09-30T01:34:41.899 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.noname-f 2023-09-30T01:34:41.899 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.noname-b 2023-09-30T01:34:41.899 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.noname-g 2023-09-30T01:34:41.899 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.noname-c 2023-09-30T01:34:41.899 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.noname-h 2023-09-30T01:34:41.899 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.noname-d 2023-09-30T01:34:41.899 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.noname-i 2023-09-30T01:34:41.899 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.noname-e 2023-09-30T01:34:41.899 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:34:41.899 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:41.875+0000 7f45e4125700 1 Processor -- start 2023-09-30T01:34:41.900 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:41.875+0000 7f45e4125700 1 -- start start 2023-09-30T01:34:41.900 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:41.875+0000 7f45e4125700 10 monclient: init 2023-09-30T01:34:41.900 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:41.879+0000 7f45e4125700 10 monclient: _reopen_session rank -1 2023-09-30T01:34:41.900 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:41.879+0000 7f45e4125700 10 monclient: _add_conns ranks=[0,4,2,5,7,6,1,3,8] 2023-09-30T01:34:41.900 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:41.879+0000 7f45e4125700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f45dc0a35f0 0x7f45dc142550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:34:41.900 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:41.879+0000 7f45e4125700 10 monclient(hunting): picked mon.noname-a con 0x7f45dc0a35f0 addr [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] 2023-09-30T01:34:41.900 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:41.879+0000 7f45e4125700 1 --2- >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f45dc0a46f0 0x7f45dc142a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:34:41.900 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:41.879+0000 7f45e4125700 10 monclient(hunting): picked mon.noname-c con 0x7f45dc0a46f0 addr [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] 2023-09-30T01:34:41.900 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:41.879+0000 7f45e4125700 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f45dc133870 0x7f45dc142fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:34:41.901 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:41.879+0000 7f45e4125700 10 monclient(hunting): picked mon.noname-b con 0x7f45dc133870 addr [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] 2023-09-30T01:34:41.901 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:41.879+0000 7f45e4125700 10 monclient(hunting): start opening mon connection 2023-09-30T01:34:41.901 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:41.879+0000 7f45e4125700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45dc121e10 con 0x7f45dc0a35f0 2023-09-30T01:34:41.901 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:41.879+0000 7f45e4125700 10 monclient(hunting): start opening mon connection 2023-09-30T01:34:41.901 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:41.879+0000 7f45e4125700 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45dc122a90 con 0x7f45dc133870 2023-09-30T01:34:41.901 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:41.879+0000 7f45e4125700 10 monclient(hunting): start opening mon connection 2023-09-30T01:34:41.901 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:41.879+0000 7f45e4125700 1 -- --> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] -- mon_getmap magic: 0 v1 -- 0x7f45dc0a0b70 con 0x7f45dc0a46f0 2023-09-30T01:34:41.901 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:41.879+0000 7f45e4125700 10 monclient(hunting): _renew_subs 2023-09-30T01:34:41.901 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:41.879+0000 7f45e4125700 10 monclient(hunting): authenticate will time out at 953.921255s 2023-09-30T01:34:41.901 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:41.879+0000 7f45e1ec1700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f45dc0a35f0 0x7f45dc142550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:34:41.902 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:41.879+0000 7f45e16c0700 1 --2- >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f45dc0a46f0 0x7f45dc142a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:34:41.902 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:41.879+0000 7f45e1ec1700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f45dc0a35f0 0x7f45dc142550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:42836/0 (socket says 172.21.15.88:42836) 2023-09-30T01:34:41.902 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:41.879+0000 7f45e1ec1700 1 -- 172.21.15.88:0/367484896 learned_addr learned my addr 172.21.15.88:0/367484896 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-09-30T01:34:41.902 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:41.879+0000 7f45e1ec1700 10 monclient(hunting): get_auth_request con 0x7f45dc0a35f0 auth_method 0 2023-09-30T01:34:41.902 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:41.879+0000 7f45e1ec1700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:34:41.902 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:41.879+0000 7f45e1ec1700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:34:41.902 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:41.879+0000 7f45e1ec1700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:34:41.902 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:41.879+0000 7f45e26c2700 1 --2- 172.21.15.88:0/367484896 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f45dc133870 0x7f45dc142fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:34:41.902 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:41.879+0000 7f45e16c0700 10 monclient(hunting): get_auth_request con 0x7f45dc0a46f0 auth_method 0 2023-09-30T01:34:41.902 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:41.879+0000 7f45e16c0700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:34:41.903 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:41.879+0000 7f45e16c0700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:34:41.903 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:41.879+0000 7f45e16c0700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:34:41.903 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:41.879+0000 7f45e26c2700 10 monclient(hunting): get_auth_request con 0x7f45dc133870 auth_method 0 2023-09-30T01:34:41.903 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:41.879+0000 7f45e26c2700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:34:41.903 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:41.879+0000 7f45e26c2700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:34:41.903 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:41.879+0000 7f45e26c2700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:34:41.903 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:41.879+0000 7f45e1ec1700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:34:41.903 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:41.879+0000 7f45e1ec1700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:34:41.903 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:41.879+0000 7f45e1ec1700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:34:41.903 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:41.879+0000 7f45e16c0700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:34:41.904 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:41.879+0000 7f45e16c0700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:34:41.904 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:41.879+0000 7f45e16c0700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:34:41.904 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:41.879+0000 7f45e26c2700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:34:41.904 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:41.879+0000 7f45e26c2700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:34:41.904 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:41.879+0000 7f45e26c2700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:34:41.904 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:41.879+0000 7f45e1ec1700 10 monclient(hunting): handle_auth_done global_id 4680 payload 824 2023-09-30T01:34:41.904 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:41.879+0000 7f45e1ec1700 1 -- 172.21.15.88:0/367484896 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f45dc0a46f0 msgr2=0x7f45dc142a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:34:41.904 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:41.879+0000 7f45e1ec1700 1 --2- 172.21.15.88:0/367484896 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f45dc0a46f0 0x7f45dc142a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:34:41.904 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:41.879+0000 7f45e1ec1700 1 -- 172.21.15.88:0/367484896 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f45dc133870 msgr2=0x7f45dc142fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:34:41.904 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:41.879+0000 7f45e1ec1700 1 --2- 172.21.15.88:0/367484896 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f45dc133870 0x7f45dc142fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:34:41.904 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:41.879+0000 7f45e1ec1700 10 monclient: _finish_hunting 0 2023-09-30T01:34:41.905 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:41.879+0000 7f45e1ec1700 1 monclient: found mon.noname-a 2023-09-30T01:34:41.905 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:41.879+0000 7f45e1ec1700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:34:41.905 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:41.879+0000 7f45e1ec1700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.88:3300/0 2023-09-30T01:34:41.905 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:41.879+0000 7f45e1ec1700 1 -- 172.21.15.88:0/367484896 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45cc009420 con 0x7f45dc0a35f0 2023-09-30T01:34:41.905 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:41.879+0000 7f45e1ec1700 1 --2- 172.21.15.88:0/367484896 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f45dc0a35f0 0x7f45dc142550 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f45d800ec80 tx=0x7f45d80091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:34:41.905 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:41.879+0000 7f45d1ffb700 1 -- 172.21.15.88:0/367484896 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f45d8009690 con 0x7f45dc0a35f0 2023-09-30T01:34:41.905 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:41.879+0000 7f45d1ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:34:41.905 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:41.879+0000 7f45d1ffb700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T01:34:41.905 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:41.879+0000 7f45d1ffb700 10 monclient: dump: 2023-09-30T01:34:41.905 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:34:41.906 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:34:41.906 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:34:41.906 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:34:41.906 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:34:41.906 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:34:41.906 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:34:41.906 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:34:41.906 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:34:41.906 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:34:41.906 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:34:41.906 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:34:41.907 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:34:41.907 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:34:41.907 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:34:41.907 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:34:41.907 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:41.879+0000 7f45d1ffb700 10 monclient: _finish_auth 0 2023-09-30T01:34:41.907 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:41.879+0000 7f45d1ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:34:41.907 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:41.879+0000 7f45e4125700 5 monclient: authenticate success, global_id 4680 2023-09-30T01:34:41.907 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:41.879+0000 7f45d1ffb700 1 -- 172.21.15.88:0/367484896 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f45d800ccd0 con 0x7f45dc0a35f0 2023-09-30T01:34:41.907 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:41.879+0000 7f45e4125700 10 monclient: _renew_subs 2023-09-30T01:34:41.907 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:41.879+0000 7f45e4125700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.88:3300/0 2023-09-30T01:34:41.907 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:41.879+0000 7f45e4125700 1 -- 172.21.15.88:0/367484896 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f45dc121520 con 0x7f45dc0a35f0 2023-09-30T01:34:41.907 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:41.879+0000 7f45d1ffb700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:34:41.908 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:41.879+0000 7f45d1ffb700 1 -- 172.21.15.88:0/367484896 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f45d800ce40 con 0x7f45dc0a35f0 2023-09-30T01:34:41.908 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:41.879+0000 7f45d1ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:34:41.908 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:41.879+0000 7f45d1ffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:34:41.908 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:41.879+0000 7f45d3fff700 4 set_mon_vals no callback set 2023-09-30T01:34:41.908 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:41.879+0000 7f45d1ffb700 10 monclient: dump: 2023-09-30T01:34:41.908 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:34:41.908 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:34:41.908 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:34:41.908 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:34:41.908 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:34:41.908 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:34:41.908 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:34:41.909 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:34:41.909 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:34:41.909 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:34:41.909 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:34:41.909 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:34:41.909 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:34:41.909 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:34:41.909 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:34:41.909 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:34:41.909 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:41.879+0000 7f45e4125700 10 monclient: _renew_subs 2023-09-30T01:34:41.909 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:41.879+0000 7f45e4125700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.88:3300/0 2023-09-30T01:34:41.909 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:41.879+0000 7f45e4125700 1 -- 172.21.15.88:0/367484896 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f45dc064be0 con 0x7f45dc0a35f0 2023-09-30T01:34:41.910 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:41.879+0000 7f45d1ffb700 1 -- 172.21.15.88:0/367484896 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 40144+0+0 (secure 0 0 0) 0x7f45d800c850 con 0x7f45dc0a35f0 2023-09-30T01:34:41.910 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:41.883+0000 7f45d1ffb700 1 --2- 172.21.15.88:0/367484896 >> v2:172.21.15.107:6812/29926 conn(0x7f45c0033390 0x7f45c0035840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:34:41.910 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:41.883+0000 7f45d1ffb700 1 -- 172.21.15.88:0/367484896 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 3502+0+0 (secure 0 0 0) 0x7f45d8043990 con 0x7f45dc0a35f0 2023-09-30T01:34:41.910 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:41.883+0000 7f45e16c0700 1 --2- 172.21.15.88:0/367484896 >> v2:172.21.15.107:6812/29926 conn(0x7f45c0033390 0x7f45c0035840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:34:41.910 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:41.883+0000 7f45e16c0700 10 monclient: get_auth_request con 0x7f45c0033390 auth_method 0 2023-09-30T01:34:41.910 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:41.883+0000 7f45e4125700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:34:41.910 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:41.883+0000 7f45e4125700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:34:41.910 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:41.883+0000 7f45e4125700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.88:3300/0 2023-09-30T01:34:41.910 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:41.883+0000 7f45e4125700 1 -- 172.21.15.88:0/367484896 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f45dc11fd50 con 0x7f45dc0a35f0 2023-09-30T01:34:41.910 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:41.883+0000 7f45e16c0700 1 --2- 172.21.15.88:0/367484896 >> v2:172.21.15.107:6812/29926 conn(0x7f45c0033390 0x7f45c0035840 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f45dc0a35c0 tx=0x7f45cc017000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:34:41.911 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:41.887+0000 7f45d1ffb700 1 -- 172.21.15.88:0/367484896 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f45d800cb00 con 0x7f45dc0a35f0 2023-09-30T01:34:41.911 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:41.887+0000 7f45d1ffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:34:41.911 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:41.887+0000 7f45d1ffb700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:34:42.092 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:42.087+0000 7f45e4125700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T01:34:42.092 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:42.087+0000 7f45e4125700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:34:42.092 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:42.087+0000 7f45e4125700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.88:3300/0 2023-09-30T01:34:42.092 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:42.087+0000 7f45e4125700 1 -- 172.21.15.88:0/367484896 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f45dc0710d0 con 0x7f45dc0a35f0 2023-09-30T01:34:42.093 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:42.087+0000 7f45d1ffb700 1 -- 172.21.15.88:0/367484896 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f45d8029080 con 0x7f45dc0a35f0 2023-09-30T01:34:42.093 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:42.087+0000 7f45d1ffb700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:34:42.093 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:42.087+0000 7f45d1ffb700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:34:42.094 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-09-30T01:34:42.094 INFO:teuthology.orchestra.run.smithi088.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:34:42.098 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:42.091+0000 7f45e4125700 1 -- 172.21.15.88:0/367484896 >> v2:172.21.15.107:6812/29926 conn(0x7f45c0033390 msgr2=0x7f45c0035840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:34:42.098 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:42.091+0000 7f45e4125700 1 --2- 172.21.15.88:0/367484896 >> v2:172.21.15.107:6812/29926 conn(0x7f45c0033390 0x7f45c0035840 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f45dc0a35c0 tx=0x7f45cc017000 comp rx=0 tx=0).stop 2023-09-30T01:34:42.098 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:42.091+0000 7f45e4125700 10 monclient: shutdown 2023-09-30T01:34:42.098 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:42.091+0000 7f45e4125700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:34:42.099 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:42.091+0000 7f45e4125700 1 -- 172.21.15.88:0/367484896 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f45dc0a35f0 msgr2=0x7f45dc142550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:34:42.099 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:42.091+0000 7f45e4125700 1 --2- 172.21.15.88:0/367484896 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f45dc0a35f0 0x7f45dc142550 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f45d800ec80 tx=0x7f45d80091a0 comp rx=0 tx=0).stop 2023-09-30T01:34:42.099 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:42.091+0000 7f45e4125700 1 -- 172.21.15.88:0/367484896 shutdown_connections 2023-09-30T01:34:42.100 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:42.091+0000 7f45e4125700 1 --2- 172.21.15.88:0/367484896 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f45dc133870 0x7f45dc142fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:34:42.100 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:42.091+0000 7f45e4125700 1 --2- 172.21.15.88:0/367484896 >> v2:172.21.15.107:6812/29926 conn(0x7f45c0033390 0x7f45c0035840 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:34:42.100 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:42.091+0000 7f45e4125700 1 --2- 172.21.15.88:0/367484896 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f45dc0a46f0 0x7f45dc142a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:34:42.100 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:42.091+0000 7f45e4125700 1 --2- 172.21.15.88:0/367484896 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f45dc0a35f0 0x7f45dc142550 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:34:42.100 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:42.091+0000 7f45e4125700 1 -- 172.21.15.88:0/367484896 >> 172.21.15.88:0/367484896 conn(0x7f45dc06e5e0 msgr2=0x7f45dc0a1f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:34:42.101 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:42.091+0000 7f45e4125700 1 -- 172.21.15.88:0/367484896 shutdown_connections 2023-09-30T01:34:42.101 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:42.091+0000 7f45e4125700 1 -- 172.21.15.88:0/367484896 wait complete. 2023-09-30T01:34:45.113 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:34:45.225 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:45.223+0000 7f2873fff700 10 monclient: get_monmap_and_config 2023-09-30T01:34:45.225 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:45.223+0000 7f2873fff700 10 monclient: build_initial_monmap 2023-09-30T01:34:45.225 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:45.223+0000 7f2873fff700 10 monclient: monmap: 2023-09-30T01:34:45.225 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2023-09-30T01:34:45.225 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:34:45.225 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:34:45.225400+0000 2023-09-30T01:34:45.225 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:34:45.225400+0000 2023-09-30T01:34:45.226 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2023-09-30T01:34:45.226 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:34:45.226 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a-legacy 2023-09-30T01:34:45.226 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.107:6789/0 mon.noname-f-legacy 2023-09-30T01:34:45.226 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.noname-b 2023-09-30T01:34:45.226 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.noname-g 2023-09-30T01:34:45.226 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.noname-c 2023-09-30T01:34:45.226 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.noname-h 2023-09-30T01:34:45.226 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.noname-d 2023-09-30T01:34:45.226 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.noname-i 2023-09-30T01:34:45.226 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.noname-e 2023-09-30T01:34:45.227 INFO:teuthology.orchestra.run.smithi088.stderr:9: v2:172.21.15.88:3300/0 mon.noname-a 2023-09-30T01:34:45.227 INFO:teuthology.orchestra.run.smithi088.stderr:10: v2:172.21.15.107:3300/0 mon.noname-f 2023-09-30T01:34:45.227 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:34:45.236 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:45.223+0000 7f2873fff700 1 Processor -- start 2023-09-30T01:34:45.236 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:45.223+0000 7f2873fff700 1 -- start start 2023-09-30T01:34:45.236 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:45.223+0000 7f2873fff700 10 monclient: init 2023-09-30T01:34:45.236 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:45.223+0000 7f2873fff700 10 monclient: _reopen_session rank -1 2023-09-30T01:34:45.236 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:45.223+0000 7f2873fff700 10 monclient: _add_conns ranks=[8,2,10,1,7,4,6,3,5,0,9] 2023-09-30T01:34:45.236 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:45.223+0000 7f2873fff700 1 --2- >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f2874133480 0x7f2874133850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:34:45.237 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:45.223+0000 7f2873fff700 10 monclient(hunting): picked mon.noname-e con 0x7f2874133480 addr [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] 2023-09-30T01:34:45.237 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:45.223+0000 7f2873fff700 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2874099590 0x7f2874099960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:34:45.237 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:45.223+0000 7f2873fff700 10 monclient(hunting): picked mon.noname-b con 0x7f2874099590 addr [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] 2023-09-30T01:34:45.237 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:45.223+0000 7f2873fff700 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f2874099ea0 0x7f28740686c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:34:45.237 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:45.223+0000 7f2873fff700 10 monclient(hunting): picked mon.noname-f con 0x7f2874099ea0 addr v2:172.21.15.107:3300/0 2023-09-30T01:34:45.237 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:45.223+0000 7f2873fff700 10 monclient(hunting): start opening mon connection 2023-09-30T01:34:45.237 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:45.223+0000 7f2873fff700 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f28740a20c0 con 0x7f2874099ea0 2023-09-30T01:34:45.237 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:45.223+0000 7f2873fff700 10 monclient(hunting): start opening mon connection 2023-09-30T01:34:45.237 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:45.223+0000 7f2873fff700 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2874097b50 con 0x7f2874099590 2023-09-30T01:34:45.237 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:45.223+0000 7f2873fff700 10 monclient(hunting): start opening mon connection 2023-09-30T01:34:45.237 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:45.223+0000 7f2873fff700 1 -- --> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] -- mon_getmap magic: 0 v1 -- 0x7f2874097d80 con 0x7f2874133480 2023-09-30T01:34:45.237 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:45.223+0000 7f2873fff700 10 monclient(hunting): _renew_subs 2023-09-30T01:34:45.237 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:45.223+0000 7f2873fff700 10 monclient(hunting): authenticate will time out at 957.263370s 2023-09-30T01:34:45.238 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:45.223+0000 7f28727fc700 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2874099590 0x7f2874099960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:34:45.238 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:45.223+0000 7f2872ffd700 1 --2- >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f2874133480 0x7f2874133850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:34:45.238 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:45.223+0000 7f28727fc700 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2874099590 0x7f2874099960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3301/0 says I am v2:172.21.15.88:32982/0 (socket says 172.21.15.88:32982) 2023-09-30T01:34:45.238 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:45.223+0000 7f28727fc700 1 -- 172.21.15.88:0/792059628 learned_addr learned my addr 172.21.15.88:0/792059628 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-09-30T01:34:45.238 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:45.223+0000 7f28727fc700 10 monclient(hunting): get_auth_request con 0x7f2874099590 auth_method 0 2023-09-30T01:34:45.238 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:45.223+0000 7f28727fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:34:45.238 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:45.223+0000 7f28727fc700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:34:45.238 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:45.223+0000 7f28727fc700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:34:45.238 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:45.223+0000 7f2872ffd700 10 monclient(hunting): get_auth_request con 0x7f2874133480 auth_method 0 2023-09-30T01:34:45.238 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:45.223+0000 7f2872ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:34:45.238 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:45.223+0000 7f2872ffd700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:34:45.239 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:45.223+0000 7f2872ffd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:34:45.239 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:45.223+0000 7f28727fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:34:45.239 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:45.223+0000 7f28727fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:34:45.239 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:45.223+0000 7f28737fe700 1 --2- 172.21.15.88:0/792059628 >> v2:172.21.15.107:3300/0 conn(0x7f2874099ea0 0x7f28740686c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:34:45.239 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:45.223+0000 7f28727fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:34:45.239 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:45.223+0000 7f28737fe700 10 monclient(hunting): get_auth_request con 0x7f2874099ea0 auth_method 0 2023-09-30T01:34:45.239 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:45.223+0000 7f28737fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:34:45.239 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:45.223+0000 7f28737fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:34:45.239 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:45.223+0000 7f28737fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:34:45.239 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:45.223+0000 7f2872ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:34:45.239 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:45.223+0000 7f2872ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:34:45.239 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:45.223+0000 7f2872ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:34:45.239 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:45.223+0000 7f28737fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:34:45.240 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:45.223+0000 7f28737fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:34:45.240 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:45.223+0000 7f28737fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:34:45.240 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:45.223+0000 7f28727fc700 10 monclient(hunting): handle_auth_done global_id 4493 payload 274 2023-09-30T01:34:45.240 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:45.223+0000 7f28727fc700 1 -- 172.21.15.88:0/792059628 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f2874133480 msgr2=0x7f2874133850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:34:45.240 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:45.223+0000 7f28727fc700 1 --2- 172.21.15.88:0/792059628 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f2874133480 0x7f2874133850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:34:45.240 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:45.223+0000 7f28727fc700 1 -- 172.21.15.88:0/792059628 >> v2:172.21.15.107:3300/0 conn(0x7f2874099ea0 msgr2=0x7f28740686c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:34:45.240 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:45.223+0000 7f28727fc700 1 --2- 172.21.15.88:0/792059628 >> v2:172.21.15.107:3300/0 conn(0x7f2874099ea0 0x7f28740686c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:34:45.240 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:45.223+0000 7f28727fc700 10 monclient: _finish_hunting 0 2023-09-30T01:34:45.240 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:45.223+0000 7f28727fc700 1 monclient: found mon.noname-b 2023-09-30T01:34:45.240 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:45.223+0000 7f28727fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:34:45.240 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:45.223+0000 7f28727fc700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.88:3301/0 2023-09-30T01:34:45.240 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:45.223+0000 7f28727fc700 1 -- 172.21.15.88:0/792059628 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f287404bb20 con 0x7f2874099590 2023-09-30T01:34:45.241 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:45.223+0000 7f28727fc700 1 --2- 172.21.15.88:0/792059628 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2874099590 0x7f2874099960 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f28640099b0 tx=0x7f2864009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=68ba05a2f6a9bdc9 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:34:45.241 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:45.223+0000 7f2871ffb700 1 -- 172.21.15.88:0/792059628 <== mon.2 v2:172.21.15.88:3301/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f286400bb00 con 0x7f2874099590 2023-09-30T01:34:45.241 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:45.223+0000 7f2871ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:34:45.241 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:45.223+0000 7f2871ffb700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T01:34:45.241 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:45.223+0000 7f2871ffb700 10 monclient: dump: 2023-09-30T01:34:45.241 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:34:45.241 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:34:45.241 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:34:45.241 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:34:45.241 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:34:45.241 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:34:45.242 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:34:45.242 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:34:45.242 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:34:45.242 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:34:45.242 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:34:45.242 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:34:45.242 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:34:45.242 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:34:45.242 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:34:45.242 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:34:45.242 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:45.223+0000 7f2871ffb700 10 monclient: _finish_auth 0 2023-09-30T01:34:45.242 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:45.223+0000 7f2871ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:34:45.243 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:45.223+0000 7f2873fff700 5 monclient: authenticate success, global_id 4493 2023-09-30T01:34:45.243 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:45.223+0000 7f2873fff700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:34:45.243 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:45.223+0000 7f2871ffb700 1 -- 172.21.15.88:0/792059628 <== mon.2 v2:172.21.15.88:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f286400bc80 con 0x7f2874099590 2023-09-30T01:34:45.243 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:45.223+0000 7f2871ffb700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:34:45.243 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:45.223+0000 7f2873fff700 10 monclient: get_monmap_and_config success 2023-09-30T01:34:45.243 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:45.223+0000 7f2873fff700 4 set_mon_vals no callback set 2023-09-30T01:34:45.243 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:45.223+0000 7f2871ffb700 1 -- 172.21.15.88:0/792059628 <== mon.2 v2:172.21.15.88:3301/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f286400bdf0 con 0x7f2874099590 2023-09-30T01:34:45.243 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:45.223+0000 7f2871ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:34:45.243 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:45.223+0000 7f2871ffb700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T01:34:45.243 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:45.223+0000 7f2871ffb700 10 monclient: dump: 2023-09-30T01:34:45.243 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:34:45.243 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:34:45.243 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:34:45.244 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:34:45.244 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:34:45.244 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:34:45.244 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:34:45.244 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:34:45.244 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:34:45.244 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:34:45.244 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:34:45.244 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:34:45.244 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:34:45.244 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:34:45.244 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:34:45.245 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:34:45.245 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:45.223+0000 7f2873fff700 10 monclient: shutdown 2023-09-30T01:34:45.245 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:45.223+0000 7f2873fff700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:34:45.245 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:45.223+0000 7f2873fff700 1 -- 172.21.15.88:0/792059628 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2874099590 msgr2=0x7f2874099960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:34:45.245 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:45.223+0000 7f2873fff700 1 --2- 172.21.15.88:0/792059628 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2874099590 0x7f2874099960 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f28640099b0 tx=0x7f2864009250 comp rx=0 tx=0).stop 2023-09-30T01:34:45.245 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:45.223+0000 7f2873fff700 1 -- 172.21.15.88:0/792059628 shutdown_connections 2023-09-30T01:34:45.245 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:45.223+0000 7f2873fff700 1 --2- 172.21.15.88:0/792059628 >> v2:172.21.15.107:3300/0 conn(0x7f2874099ea0 0x7f28740686c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:34:45.245 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:45.223+0000 7f2873fff700 1 --2- 172.21.15.88:0/792059628 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2874099590 0x7f2874099960 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:34:45.245 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:45.223+0000 7f2873fff700 1 --2- 172.21.15.88:0/792059628 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f2874133480 0x7f2874133850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:34:45.245 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:45.223+0000 7f2873fff700 1 -- 172.21.15.88:0/792059628 >> 172.21.15.88:0/792059628 conn(0x7f28740a6600 msgr2=0x7f28740a6a00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:34:45.245 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:45.223+0000 7f2873fff700 1 -- 172.21.15.88:0/792059628 shutdown_connections 2023-09-30T01:34:45.245 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:45.223+0000 7f2873fff700 1 -- 172.21.15.88:0/792059628 wait complete. 2023-09-30T01:34:45.246 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:45.223+0000 7f2873fff700 10 monclient: build_initial_monmap 2023-09-30T01:34:45.246 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:45.223+0000 7f2873fff700 10 monclient: monmap: 2023-09-30T01:34:45.246 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2023-09-30T01:34:45.246 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:34:45.246 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 0.000000 2023-09-30T01:34:45.246 INFO:teuthology.orchestra.run.smithi088.stderr:created 0.000000 2023-09-30T01:34:45.246 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2023-09-30T01:34:45.246 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 1 2023-09-30T01:34:45.246 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.noname-a 2023-09-30T01:34:45.246 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.noname-f 2023-09-30T01:34:45.246 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.noname-b 2023-09-30T01:34:45.246 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.noname-g 2023-09-30T01:34:45.246 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.noname-c 2023-09-30T01:34:45.247 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.noname-h 2023-09-30T01:34:45.247 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.noname-d 2023-09-30T01:34:45.247 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.noname-i 2023-09-30T01:34:45.247 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.noname-e 2023-09-30T01:34:45.247 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:34:45.247 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:45.223+0000 7f2873fff700 1 Processor -- start 2023-09-30T01:34:45.247 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:45.223+0000 7f2873fff700 1 -- start start 2023-09-30T01:34:45.247 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:45.223+0000 7f2873fff700 10 monclient: init 2023-09-30T01:34:45.247 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:45.227+0000 7f2873fff700 10 monclient: _reopen_session rank -1 2023-09-30T01:34:45.247 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:45.227+0000 7f2873fff700 10 monclient: _add_conns ranks=[7,2,6,3,8,5,0,1,4] 2023-09-30T01:34:45.247 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:45.227+0000 7f2873fff700 1 --2- >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7f2874099590 0x7f287411d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:34:45.247 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:45.227+0000 7f2873fff700 10 monclient(hunting): picked mon.noname-i con 0x7f2874099590 addr [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] 2023-09-30T01:34:45.247 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:45.227+0000 7f2873fff700 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2874099ea0 0x7f287411ded0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:34:45.248 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:45.227+0000 7f2873fff700 10 monclient(hunting): picked mon.noname-b con 0x7f2874099ea0 addr [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] 2023-09-30T01:34:45.248 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:45.227+0000 7f2873fff700 1 --2- >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7f2874133480 0x7f287411e410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:34:45.248 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:45.227+0000 7f2873fff700 10 monclient(hunting): picked mon.noname-d con 0x7f2874133480 addr [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] 2023-09-30T01:34:45.248 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:45.227+0000 7f2873fff700 10 monclient(hunting): start opening mon connection 2023-09-30T01:34:45.248 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:45.227+0000 7f2873fff700 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28740a2390 con 0x7f2874099ea0 2023-09-30T01:34:45.248 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:45.227+0000 7f2873fff700 10 monclient(hunting): start opening mon connection 2023-09-30T01:34:45.248 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:45.227+0000 7f2873fff700 1 -- --> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] -- mon_getmap magic: 0 v1 -- 0x7f2874121340 con 0x7f2874133480 2023-09-30T01:34:45.248 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:45.227+0000 7f2873fff700 10 monclient(hunting): start opening mon connection 2023-09-30T01:34:45.248 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:45.227+0000 7f2873fff700 1 -- --> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] -- mon_getmap magic: 0 v1 -- 0x7f28741305f0 con 0x7f2874099590 2023-09-30T01:34:45.248 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:45.227+0000 7f2873fff700 10 monclient(hunting): _renew_subs 2023-09-30T01:34:45.248 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:45.227+0000 7f2873fff700 10 monclient(hunting): authenticate will time out at 957.268710s 2023-09-30T01:34:45.248 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:45.227+0000 7f28727fc700 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2874099ea0 0x7f287411ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:34:45.249 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:45.227+0000 7f28727fc700 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2874099ea0 0x7f287411ded0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3301/0 says I am v2:172.21.15.88:32994/0 (socket says 172.21.15.88:32994) 2023-09-30T01:34:45.249 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:45.227+0000 7f28727fc700 1 -- 172.21.15.88:0/1839459202 learned_addr learned my addr 172.21.15.88:0/1839459202 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-09-30T01:34:45.249 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:45.227+0000 7f28727fc700 10 monclient(hunting): get_auth_request con 0x7f2874099ea0 auth_method 0 2023-09-30T01:34:45.249 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:45.227+0000 7f28727fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:34:45.249 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:45.227+0000 7f28727fc700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:34:45.249 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:45.227+0000 7f28727fc700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:34:45.249 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:45.227+0000 7f28737fe700 1 --2- 172.21.15.88:0/1839459202 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7f2874133480 0x7f287411e410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:34:45.249 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:45.227+0000 7f28737fe700 10 monclient(hunting): get_auth_request con 0x7f2874133480 auth_method 0 2023-09-30T01:34:45.249 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:45.227+0000 7f28737fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:34:45.249 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:45.227+0000 7f28737fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:34:45.249 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:45.227+0000 7f28737fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:34:45.249 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:45.227+0000 7f2872ffd700 1 --2- 172.21.15.88:0/1839459202 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7f2874099590 0x7f287411d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:34:45.250 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:45.227+0000 7f28727fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:34:45.250 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:45.227+0000 7f28727fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:34:45.250 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:45.227+0000 7f28727fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:34:45.250 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:45.227+0000 7f2872ffd700 10 monclient(hunting): get_auth_request con 0x7f2874099590 auth_method 0 2023-09-30T01:34:45.250 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:45.227+0000 7f2872ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:34:45.250 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:45.227+0000 7f2872ffd700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:34:45.250 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:45.227+0000 7f2872ffd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:34:45.250 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:45.227+0000 7f28737fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:34:45.250 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:45.227+0000 7f28737fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:34:45.250 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:45.227+0000 7f28737fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:34:45.250 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:45.227+0000 7f2872ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:34:45.250 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:45.227+0000 7f2872ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:34:45.251 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:45.227+0000 7f2872ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:34:45.251 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:45.227+0000 7f28727fc700 10 monclient(hunting): handle_auth_done global_id 4502 payload 824 2023-09-30T01:34:45.251 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:45.227+0000 7f28727fc700 1 -- 172.21.15.88:0/1839459202 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7f2874099590 msgr2=0x7f287411d990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:34:45.251 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:45.227+0000 7f28727fc700 1 --2- 172.21.15.88:0/1839459202 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7f2874099590 0x7f287411d990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:34:45.251 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:45.227+0000 7f28727fc700 1 -- 172.21.15.88:0/1839459202 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7f2874133480 msgr2=0x7f287411e410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:34:45.251 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:45.227+0000 7f28727fc700 1 --2- 172.21.15.88:0/1839459202 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7f2874133480 0x7f287411e410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:34:45.251 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:45.227+0000 7f28727fc700 10 monclient: _finish_hunting 0 2023-09-30T01:34:45.251 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:45.227+0000 7f28727fc700 1 monclient: found mon.noname-b 2023-09-30T01:34:45.251 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:45.227+0000 7f28727fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:34:45.251 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:45.227+0000 7f28727fc700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.88:3301/0 2023-09-30T01:34:45.251 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:45.227+0000 7f28727fc700 1 -- 172.21.15.88:0/1839459202 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28640094f0 con 0x7f2874099ea0 2023-09-30T01:34:45.251 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:45.227+0000 7f28727fc700 1 --2- 172.21.15.88:0/1839459202 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2874099ea0 0x7f287411ded0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f2864013000 tx=0x7f2864017000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:34:45.251 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:45.227+0000 7f286affd700 1 -- 172.21.15.88:0/1839459202 <== mon.2 v2:172.21.15.88:3301/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f2864006b60 con 0x7f2874099ea0 2023-09-30T01:34:45.252 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:45.227+0000 7f286affd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:34:45.252 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:45.227+0000 7f286affd700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T01:34:45.252 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:45.227+0000 7f286affd700 10 monclient: dump: 2023-09-30T01:34:45.252 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:34:45.252 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:34:45.252 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:34:45.252 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:34:45.252 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:34:45.252 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:34:45.252 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:34:45.252 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:34:45.252 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:34:45.252 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:34:45.253 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:34:45.253 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:34:45.253 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:34:45.253 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:34:45.253 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:34:45.253 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:34:45.253 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:45.227+0000 7f286affd700 10 monclient: _finish_auth 0 2023-09-30T01:34:45.253 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:45.227+0000 7f286affd700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:34:45.253 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:45.227+0000 7f286affd700 1 -- 172.21.15.88:0/1839459202 <== mon.2 v2:172.21.15.88:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f286401a450 con 0x7f2874099ea0 2023-09-30T01:34:45.253 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:45.227+0000 7f2873fff700 5 monclient: authenticate success, global_id 4502 2023-09-30T01:34:45.253 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:45.227+0000 7f2873fff700 10 monclient: _renew_subs 2023-09-30T01:34:45.253 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:45.227+0000 7f2873fff700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.88:3301/0 2023-09-30T01:34:45.253 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:45.227+0000 7f2873fff700 1 -- 172.21.15.88:0/1839459202 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2874121060 con 0x7f2874099ea0 2023-09-30T01:34:45.254 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:45.227+0000 7f286affd700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:34:45.254 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:45.227+0000 7f286affd700 1 -- 172.21.15.88:0/1839459202 <== mon.2 v2:172.21.15.88:3301/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f286401aa40 con 0x7f2874099ea0 2023-09-30T01:34:45.254 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:45.227+0000 7f286affd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:34:45.254 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:45.227+0000 7f286affd700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T01:34:45.254 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:45.227+0000 7f28717fa700 4 set_mon_vals no callback set 2023-09-30T01:34:45.254 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:45.227+0000 7f286affd700 10 monclient: dump: 2023-09-30T01:34:45.254 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:34:45.254 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:34:45.254 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:34:45.254 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:34:45.254 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:34:45.254 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:34:45.254 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:34:45.255 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:34:45.255 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:34:45.255 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:34:45.255 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:34:45.255 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:34:45.255 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:34:45.255 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:34:45.255 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:34:45.255 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:34:45.255 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:45.227+0000 7f2873fff700 10 monclient: _renew_subs 2023-09-30T01:34:45.255 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:45.227+0000 7f2873fff700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.88:3301/0 2023-09-30T01:34:45.256 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:45.227+0000 7f2873fff700 1 -- 172.21.15.88:0/1839459202 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f287411f530 con 0x7f2874099ea0 2023-09-30T01:34:45.256 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:45.227+0000 7f286affd700 1 -- 172.21.15.88:0/1839459202 <== mon.2 v2:172.21.15.88:3301/0 4 ==== mgrmap(e 5) v1 ==== 40144+0+0 (secure 0 0 0) 0x7f286401aba0 con 0x7f2874099ea0 2023-09-30T01:34:45.256 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:45.227+0000 7f2873fff700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:34:45.256 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:45.227+0000 7f2873fff700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:34:45.256 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:45.227+0000 7f2873fff700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.88:3301/0 2023-09-30T01:34:45.256 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:45.227+0000 7f2873fff700 1 -- 172.21.15.88:0/1839459202 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f28741214b0 con 0x7f2874099ea0 2023-09-30T01:34:45.256 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:45.231+0000 7f286affd700 1 --2- 172.21.15.88:0/1839459202 >> v2:172.21.15.107:6812/29926 conn(0x7f2854033290 0x7f2854035740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:34:45.256 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:45.231+0000 7f286affd700 1 -- 172.21.15.88:0/1839459202 <== mon.2 v2:172.21.15.88:3301/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 3502+0+0 (secure 0 0 0) 0x7f2864044fa0 con 0x7f2874099ea0 2023-09-30T01:34:45.256 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:45.231+0000 7f2872ffd700 1 --2- 172.21.15.88:0/1839459202 >> v2:172.21.15.107:6812/29926 conn(0x7f2854033290 0x7f2854035740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:34:45.256 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:45.231+0000 7f2872ffd700 10 monclient: get_auth_request con 0x7f2854033290 auth_method 0 2023-09-30T01:34:45.257 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:45.231+0000 7f2872ffd700 1 --2- 172.21.15.88:0/1839459202 >> v2:172.21.15.107:6812/29926 conn(0x7f2854033290 0x7f2854035740 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f285c009d10 tx=0x7f285c0091a0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:34:45.257 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:45.235+0000 7f286affd700 1 -- 172.21.15.88:0/1839459202 <== mon.2 v2:172.21.15.88:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f2864074be0 con 0x7f2874099ea0 2023-09-30T01:34:45.257 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:45.235+0000 7f286affd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:34:45.257 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:45.235+0000 7f286affd700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:34:45.435 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:45.431+0000 7f2873fff700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T01:34:45.435 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:45.431+0000 7f2873fff700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:34:45.435 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:45.431+0000 7f2873fff700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.88:3301/0 2023-09-30T01:34:45.435 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:45.431+0000 7f2873fff700 1 -- 172.21.15.88:0/1839459202 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f2874071960 con 0x7f2874099ea0 2023-09-30T01:34:45.435 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:45.435+0000 7f286affd700 1 -- 172.21.15.88:0/1839459202 <== mon.2 v2:172.21.15.88:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f2864015070 con 0x7f2874099ea0 2023-09-30T01:34:45.436 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:45.435+0000 7f286affd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:34:45.436 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:45.435+0000 7f286affd700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:34:45.436 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-09-30T01:34:45.436 INFO:teuthology.orchestra.run.smithi088.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:34:45.438 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:45.435+0000 7f2873fff700 1 -- 172.21.15.88:0/1839459202 >> v2:172.21.15.107:6812/29926 conn(0x7f2854033290 msgr2=0x7f2854035740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:34:45.438 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:45.435+0000 7f2873fff700 1 --2- 172.21.15.88:0/1839459202 >> v2:172.21.15.107:6812/29926 conn(0x7f2854033290 0x7f2854035740 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f285c009d10 tx=0x7f285c0091a0 comp rx=0 tx=0).stop 2023-09-30T01:34:45.438 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:45.435+0000 7f2873fff700 10 monclient: shutdown 2023-09-30T01:34:45.438 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:45.435+0000 7f2873fff700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:34:45.438 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:45.435+0000 7f2873fff700 1 -- 172.21.15.88:0/1839459202 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2874099ea0 msgr2=0x7f287411ded0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:34:45.438 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:45.435+0000 7f2873fff700 1 --2- 172.21.15.88:0/1839459202 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2874099ea0 0x7f287411ded0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f2864013000 tx=0x7f2864017000 comp rx=0 tx=0).stop 2023-09-30T01:34:45.438 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:45.435+0000 7f2873fff700 1 -- 172.21.15.88:0/1839459202 shutdown_connections 2023-09-30T01:34:45.439 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:45.435+0000 7f2873fff700 1 --2- 172.21.15.88:0/1839459202 >> v2:172.21.15.107:6812/29926 conn(0x7f2854033290 0x7f2854035740 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:34:45.439 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:45.435+0000 7f2873fff700 1 --2- 172.21.15.88:0/1839459202 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7f2874133480 0x7f287411e410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:34:45.439 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:45.435+0000 7f2873fff700 1 --2- 172.21.15.88:0/1839459202 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2874099ea0 0x7f287411ded0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:34:45.439 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:45.435+0000 7f2873fff700 1 --2- 172.21.15.88:0/1839459202 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7f2874099590 0x7f287411d990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:34:45.439 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:45.435+0000 7f2873fff700 1 -- 172.21.15.88:0/1839459202 >> 172.21.15.88:0/1839459202 conn(0x7f28740a6600 msgr2=0x7f2874132830 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:34:45.439 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:45.435+0000 7f2873fff700 1 -- 172.21.15.88:0/1839459202 shutdown_connections 2023-09-30T01:34:45.439 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:45.435+0000 7f2873fff700 1 -- 172.21.15.88:0/1839459202 wait complete. 2023-09-30T01:34:48.450 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:34:48.562 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:48.559+0000 7f4183fff700 10 monclient: get_monmap_and_config 2023-09-30T01:34:48.563 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:48.559+0000 7f4183fff700 10 monclient: build_initial_monmap 2023-09-30T01:34:48.563 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:48.559+0000 7f4183fff700 10 monclient: monmap: 2023-09-30T01:34:48.563 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2023-09-30T01:34:48.563 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:34:48.563 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:34:48.562881+0000 2023-09-30T01:34:48.563 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:34:48.562881+0000 2023-09-30T01:34:48.563 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2023-09-30T01:34:48.563 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:34:48.563 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a-legacy 2023-09-30T01:34:48.563 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.107:6789/0 mon.noname-f-legacy 2023-09-30T01:34:48.563 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.noname-b 2023-09-30T01:34:48.564 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.noname-g 2023-09-30T01:34:48.564 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.noname-c 2023-09-30T01:34:48.564 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.noname-h 2023-09-30T01:34:48.564 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.noname-d 2023-09-30T01:34:48.564 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.noname-i 2023-09-30T01:34:48.564 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.noname-e 2023-09-30T01:34:48.564 INFO:teuthology.orchestra.run.smithi088.stderr:9: v2:172.21.15.88:3300/0 mon.noname-a 2023-09-30T01:34:48.564 INFO:teuthology.orchestra.run.smithi088.stderr:10: v2:172.21.15.107:3300/0 mon.noname-f 2023-09-30T01:34:48.564 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:34:48.568 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:48.563+0000 7f4183fff700 1 Processor -- start 2023-09-30T01:34:48.569 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:48.563+0000 7f4183fff700 1 -- start start 2023-09-30T01:34:48.569 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:48.563+0000 7f4183fff700 10 monclient: init 2023-09-30T01:34:48.569 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:48.563+0000 7f4183fff700 10 monclient: _reopen_session rank -1 2023-09-30T01:34:48.569 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:48.563+0000 7f4183fff700 10 monclient: _add_conns ranks=[2,9,7,3,1,8,0,5,10,4,6] 2023-09-30T01:34:48.569 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:48.563+0000 7f4183fff700 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f41840976f0 0x7f4184098de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:34:48.569 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:48.563+0000 7f4183fff700 10 monclient(hunting): picked mon.noname-b con 0x7f41840976f0 addr [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] 2023-09-30T01:34:48.569 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:48.563+0000 7f4183fff700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f418409ed80 0x7f418409f150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:34:48.569 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:48.563+0000 7f4183fff700 10 monclient(hunting): picked mon.noname-a con 0x7f418409ed80 addr v2:172.21.15.88:3300/0 2023-09-30T01:34:48.569 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:48.563+0000 7f4183fff700 1 --2- >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7f418409f690 0x7f4184130570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:34:48.570 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:48.563+0000 7f4183fff700 10 monclient(hunting): picked mon.noname-i con 0x7f418409f690 addr [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] 2023-09-30T01:34:48.570 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:48.563+0000 7f4183fff700 10 monclient(hunting): start opening mon connection 2023-09-30T01:34:48.570 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:48.563+0000 7f4183fff700 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4184099320 con 0x7f418409ed80 2023-09-30T01:34:48.570 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:48.563+0000 7f4183fff700 10 monclient(hunting): start opening mon connection 2023-09-30T01:34:48.570 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:48.563+0000 7f4183fff700 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4184098400 con 0x7f41840976f0 2023-09-30T01:34:48.570 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:48.563+0000 7f4183fff700 10 monclient(hunting): start opening mon connection 2023-09-30T01:34:48.570 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:48.563+0000 7f4183fff700 1 -- --> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] -- mon_getmap magic: 0 v1 -- 0x7f4184098a30 con 0x7f418409f690 2023-09-30T01:34:48.570 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:48.563+0000 7f4183fff700 10 monclient(hunting): _renew_subs 2023-09-30T01:34:48.570 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:48.563+0000 7f4183fff700 10 monclient(hunting): authenticate will time out at 960.601395s 2023-09-30T01:34:48.570 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:48.563+0000 7f41827fc700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f418409ed80 0x7f418409f150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:34:48.570 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:48.563+0000 7f4182ffd700 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f41840976f0 0x7f4184098de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:34:48.571 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:48.563+0000 7f41827fc700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f418409ed80 0x7f418409f150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:42846/0 (socket says 172.21.15.88:42846) 2023-09-30T01:34:48.571 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:48.563+0000 7f4182ffd700 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f41840976f0 0x7f4184098de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3301/0 says I am v2:172.21.15.88:33000/0 (socket says 172.21.15.88:33000) 2023-09-30T01:34:48.571 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:48.563+0000 7f41827fc700 1 -- 172.21.15.88:0/3591036056 learned_addr learned my addr 172.21.15.88:0/3591036056 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-09-30T01:34:48.571 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:48.563+0000 7f41827fc700 10 monclient(hunting): get_auth_request con 0x7f418409ed80 auth_method 0 2023-09-30T01:34:48.571 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:48.563+0000 7f41827fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:34:48.571 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:48.563+0000 7f41827fc700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:34:48.571 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:48.563+0000 7f41827fc700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:34:48.571 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:48.563+0000 7f4182ffd700 10 monclient(hunting): get_auth_request con 0x7f41840976f0 auth_method 0 2023-09-30T01:34:48.571 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:48.563+0000 7f4182ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:34:48.571 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:48.563+0000 7f4182ffd700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:34:48.571 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:48.563+0000 7f4182ffd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:34:48.571 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:48.563+0000 7f41827fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:34:48.572 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:48.563+0000 7f41827fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:34:48.572 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:48.563+0000 7f41827fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:34:48.572 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:48.563+0000 7f4182ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:34:48.572 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:48.563+0000 7f4182ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:34:48.572 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:48.563+0000 7f4182ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:34:48.572 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:48.563+0000 7f41837fe700 1 --2- 172.21.15.88:0/3591036056 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7f418409f690 0x7f4184130570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:34:48.572 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:48.563+0000 7f41837fe700 10 monclient(hunting): get_auth_request con 0x7f418409f690 auth_method 0 2023-09-30T01:34:48.572 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:48.563+0000 7f41837fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:34:48.572 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:48.563+0000 7f41837fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:34:48.572 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:48.563+0000 7f41837fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:34:48.572 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:48.563+0000 7f41827fc700 10 monclient(hunting): handle_auth_done global_id 4689 payload 274 2023-09-30T01:34:48.572 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:48.563+0000 7f41827fc700 1 -- 172.21.15.88:0/3591036056 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7f418409f690 msgr2=0x7f4184130570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:34:48.572 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:48.563+0000 7f41827fc700 1 --2- 172.21.15.88:0/3591036056 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7f418409f690 0x7f4184130570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:34:48.573 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:48.563+0000 7f41827fc700 1 -- 172.21.15.88:0/3591036056 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f41840976f0 msgr2=0x7f4184098de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:34:48.573 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:48.563+0000 7f41827fc700 1 --2- 172.21.15.88:0/3591036056 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f41840976f0 0x7f4184098de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:34:48.573 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:48.563+0000 7f41827fc700 10 monclient: _finish_hunting 0 2023-09-30T01:34:48.573 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:48.563+0000 7f41827fc700 1 monclient: found mon.noname-a 2023-09-30T01:34:48.573 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:48.563+0000 7f41827fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:34:48.573 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:48.563+0000 7f41827fc700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.88:3300/0 2023-09-30T01:34:48.573 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:48.563+0000 7f41827fc700 1 -- 172.21.15.88:0/3591036056 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f418404bb20 con 0x7f418409ed80 2023-09-30T01:34:48.573 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:48.563+0000 7f4182ffd700 1 --2- 172.21.15.88:0/3591036056 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f41840976f0 0x7f4184098de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:34:48.573 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:48.563+0000 7f41827fc700 1 --2- 172.21.15.88:0/3591036056 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f418409ed80 0x7f418409f150 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f417800bc50 tx=0x7f417800b4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=926b8201569cbaab server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:34:48.573 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:48.563+0000 7f4181ffb700 1 -- 172.21.15.88:0/3591036056 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f417800dbf0 con 0x7f418409ed80 2023-09-30T01:34:48.573 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:48.563+0000 7f4181ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:34:48.573 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:48.563+0000 7f4181ffb700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T01:34:48.574 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:48.563+0000 7f4181ffb700 10 monclient: dump: 2023-09-30T01:34:48.574 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:34:48.574 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:34:48.574 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:34:48.574 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:34:48.574 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:34:48.574 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:34:48.574 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:34:48.574 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:34:48.574 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:34:48.574 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:34:48.574 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:34:48.575 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:34:48.575 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:34:48.575 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:34:48.575 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:34:48.575 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:34:48.575 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:48.563+0000 7f4181ffb700 10 monclient: _finish_auth 0 2023-09-30T01:34:48.575 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:48.563+0000 7f4181ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:34:48.575 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:48.563+0000 7f4181ffb700 1 -- 172.21.15.88:0/3591036056 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f417800dd50 con 0x7f418409ed80 2023-09-30T01:34:48.575 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:48.563+0000 7f4183fff700 5 monclient: authenticate success, global_id 4689 2023-09-30T01:34:48.575 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:48.563+0000 7f4183fff700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:34:48.575 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:48.563+0000 7f4181ffb700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:34:48.575 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:48.563+0000 7f4181ffb700 1 -- 172.21.15.88:0/3591036056 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f4178019d80 con 0x7f418409ed80 2023-09-30T01:34:48.575 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:48.563+0000 7f4183fff700 10 monclient: get_monmap_and_config success 2023-09-30T01:34:48.576 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:48.563+0000 7f4183fff700 4 set_mon_vals no callback set 2023-09-30T01:34:48.576 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:48.563+0000 7f4181ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:34:48.576 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:48.563+0000 7f4181ffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:34:48.576 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:48.563+0000 7f4181ffb700 10 monclient: dump: 2023-09-30T01:34:48.576 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:34:48.576 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:34:48.576 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:34:48.576 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:34:48.576 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:34:48.576 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:34:48.576 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:34:48.576 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:34:48.577 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:34:48.577 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:34:48.577 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:34:48.577 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:34:48.577 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:34:48.577 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:34:48.577 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:34:48.577 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:34:48.577 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:48.563+0000 7f4183fff700 10 monclient: shutdown 2023-09-30T01:34:48.577 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:48.563+0000 7f4183fff700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:34:48.577 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:48.563+0000 7f4183fff700 1 -- 172.21.15.88:0/3591036056 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f418409ed80 msgr2=0x7f418409f150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:34:48.577 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:48.563+0000 7f4183fff700 1 --2- 172.21.15.88:0/3591036056 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f418409ed80 0x7f418409f150 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f417800bc50 tx=0x7f417800b4f0 comp rx=0 tx=0).stop 2023-09-30T01:34:48.578 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:48.563+0000 7f4183fff700 1 -- 172.21.15.88:0/3591036056 shutdown_connections 2023-09-30T01:34:48.578 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:48.563+0000 7f4183fff700 1 --2- 172.21.15.88:0/3591036056 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7f418409f690 0x7f4184130570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:34:48.578 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:48.563+0000 7f4183fff700 1 --2- 172.21.15.88:0/3591036056 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f418409ed80 0x7f418409f150 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:34:48.578 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:48.563+0000 7f4183fff700 1 --2- 172.21.15.88:0/3591036056 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f41840976f0 0x7f4184098de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:34:48.578 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:48.563+0000 7f4183fff700 1 -- 172.21.15.88:0/3591036056 >> 172.21.15.88:0/3591036056 conn(0x7f41840a0910 msgr2=0x7f41840a33c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:34:48.578 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:48.563+0000 7f4183fff700 1 -- 172.21.15.88:0/3591036056 shutdown_connections 2023-09-30T01:34:48.578 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:48.563+0000 7f4183fff700 1 -- 172.21.15.88:0/3591036056 wait complete. 2023-09-30T01:34:48.578 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:48.567+0000 7f4183fff700 10 monclient: build_initial_monmap 2023-09-30T01:34:48.578 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:48.567+0000 7f4183fff700 10 monclient: monmap: 2023-09-30T01:34:48.578 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2023-09-30T01:34:48.578 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:34:48.578 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 0.000000 2023-09-30T01:34:48.579 INFO:teuthology.orchestra.run.smithi088.stderr:created 0.000000 2023-09-30T01:34:48.579 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2023-09-30T01:34:48.579 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 1 2023-09-30T01:34:48.579 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.noname-a 2023-09-30T01:34:48.579 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.noname-f 2023-09-30T01:34:48.579 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.noname-b 2023-09-30T01:34:48.579 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.noname-g 2023-09-30T01:34:48.579 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.noname-c 2023-09-30T01:34:48.579 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.noname-h 2023-09-30T01:34:48.579 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.noname-d 2023-09-30T01:34:48.579 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.noname-i 2023-09-30T01:34:48.579 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.noname-e 2023-09-30T01:34:48.579 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:34:48.580 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:48.567+0000 7f4183fff700 1 Processor -- start 2023-09-30T01:34:48.580 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:48.567+0000 7f4183fff700 1 -- start start 2023-09-30T01:34:48.580 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:48.567+0000 7f4183fff700 10 monclient: init 2023-09-30T01:34:48.583 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:48.567+0000 7f4183fff700 10 monclient: _reopen_session rank -1 2023-09-30T01:34:48.583 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:48.567+0000 7f4183fff700 10 monclient: _add_conns ranks=[6,0,1,7,4,8,2,3,5] 2023-09-30T01:34:48.584 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:48.567+0000 7f4183fff700 1 --2- >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7f41840976f0 0x7f418412e840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:34:48.584 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:48.567+0000 7f4183fff700 10 monclient(hunting): picked mon.noname-d con 0x7f41840976f0 addr [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] 2023-09-30T01:34:48.584 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:48.567+0000 7f4183fff700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f418409ed80 0x7f418413de20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:34:48.584 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:48.567+0000 7f4183fff700 10 monclient(hunting): picked mon.noname-a con 0x7f418409ed80 addr [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] 2023-09-30T01:34:48.584 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:48.567+0000 7f4183fff700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f418409f690 0x7f418413e360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:34:48.584 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:48.567+0000 7f4183fff700 10 monclient(hunting): picked mon.noname-f con 0x7f418409f690 addr [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] 2023-09-30T01:34:48.584 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:48.567+0000 7f4183fff700 10 monclient(hunting): start opening mon connection 2023-09-30T01:34:48.584 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:48.567+0000 7f4183fff700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f418406ed00 con 0x7f418409ed80 2023-09-30T01:34:48.584 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:48.567+0000 7f4183fff700 10 monclient(hunting): start opening mon connection 2023-09-30T01:34:48.584 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:48.567+0000 7f4183fff700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4184097d80 con 0x7f418409f690 2023-09-30T01:34:48.584 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:48.567+0000 7f4183fff700 10 monclient(hunting): start opening mon connection 2023-09-30T01:34:48.585 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:48.567+0000 7f4183fff700 1 -- --> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] -- mon_getmap magic: 0 v1 -- 0x7f4184129a60 con 0x7f41840976f0 2023-09-30T01:34:48.585 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:48.567+0000 7f4183fff700 10 monclient(hunting): _renew_subs 2023-09-30T01:34:48.585 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:48.567+0000 7f4183fff700 10 monclient(hunting): authenticate will time out at 960.606944s 2023-09-30T01:34:48.585 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:48.567+0000 7f4182ffd700 1 --2- >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7f41840976f0 0x7f418412e840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:34:48.585 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:48.567+0000 7f4182ffd700 1 --2- >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7f41840976f0 0x7f418412e840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3303/0 says I am v2:172.21.15.88:60176/0 (socket says 172.21.15.88:60176) 2023-09-30T01:34:48.585 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:48.567+0000 7f4182ffd700 1 -- 172.21.15.88:0/39033161 learned_addr learned my addr 172.21.15.88:0/39033161 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-09-30T01:34:48.585 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:48.567+0000 7f4182ffd700 10 monclient(hunting): get_auth_request con 0x7f41840976f0 auth_method 0 2023-09-30T01:34:48.585 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:48.567+0000 7f4182ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:34:48.585 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:48.567+0000 7f4182ffd700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:34:48.585 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:48.567+0000 7f4182ffd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:34:48.585 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:48.567+0000 7f41827fc700 1 --2- 172.21.15.88:0/39033161 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f418409ed80 0x7f418413de20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:34:48.585 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:48.567+0000 7f4182ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:34:48.586 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:48.567+0000 7f4182ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:34:48.586 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:48.567+0000 7f4182ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:34:48.586 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:48.567+0000 7f41827fc700 10 monclient(hunting): get_auth_request con 0x7f418409ed80 auth_method 0 2023-09-30T01:34:48.586 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:48.567+0000 7f41827fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:34:48.586 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:48.567+0000 7f41827fc700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:34:48.586 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:48.567+0000 7f41827fc700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:34:48.586 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:48.567+0000 7f41827fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:34:48.586 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:48.567+0000 7f41827fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:34:48.586 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:48.567+0000 7f41827fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:34:48.586 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:48.567+0000 7f4182ffd700 10 monclient(hunting): handle_auth_done global_id 4461 payload 824 2023-09-30T01:34:48.586 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:48.567+0000 7f41837fe700 1 --2- 172.21.15.88:0/39033161 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f418409f690 0x7f418413e360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:34:48.586 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:48.567+0000 7f4182ffd700 1 -- 172.21.15.88:0/39033161 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f418409f690 msgr2=0x7f418413e360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:34:48.587 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:48.567+0000 7f4182ffd700 1 --2- 172.21.15.88:0/39033161 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f418409f690 0x7f418413e360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:34:48.587 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:48.567+0000 7f4182ffd700 1 -- 172.21.15.88:0/39033161 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f418409ed80 msgr2=0x7f418413de20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:34:48.587 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:48.567+0000 7f4182ffd700 1 --2- 172.21.15.88:0/39033161 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f418409ed80 0x7f418413de20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:34:48.587 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:48.567+0000 7f4182ffd700 10 monclient: _finish_hunting 0 2023-09-30T01:34:48.587 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:48.567+0000 7f4182ffd700 1 monclient: found mon.noname-d 2023-09-30T01:34:48.587 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:48.567+0000 7f4182ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:34:48.587 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:48.567+0000 7f4182ffd700 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.88:3303/0 2023-09-30T01:34:48.587 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:48.567+0000 7f4182ffd700 1 -- 172.21.15.88:0/39033161 --> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41740093c0 con 0x7f41840976f0 2023-09-30T01:34:48.587 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:48.567+0000 7f41837fe700 10 monclient: get_auth_request con 0x7f418409f690 auth_method 0 2023-09-30T01:34:48.587 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:48.567+0000 7f41837fe700 1 --2- 172.21.15.88:0/39033161 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f418409f690 0x7f418413e360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-09-30T01:34:48.587 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:48.567+0000 7f4182ffd700 1 --2- 172.21.15.88:0/39033161 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7f41840976f0 0x7f418412e840 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f417400ec80 tx=0x7f4174009120 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:34:48.587 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:48.571+0000 7f416affd700 1 -- 172.21.15.88:0/39033161 <== mon.6 v2:172.21.15.88:3303/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f4174005e80 con 0x7f41840976f0 2023-09-30T01:34:48.588 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:48.571+0000 7f416affd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:34:48.588 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:48.571+0000 7f416affd700 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2023-09-30T01:34:48.588 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:48.571+0000 7f416affd700 10 monclient: dump: 2023-09-30T01:34:48.588 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:34:48.588 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:34:48.588 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:34:48.588 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:34:48.588 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:34:48.588 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:34:48.588 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:34:48.588 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:34:48.588 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:34:48.589 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:34:48.589 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:34:48.589 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:34:48.589 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:34:48.589 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:34:48.589 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:34:48.589 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:34:48.589 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:48.571+0000 7f416affd700 10 monclient: _finish_auth 0 2023-09-30T01:34:48.589 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:48.571+0000 7f416affd700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:34:48.589 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:48.571+0000 7f4183fff700 5 monclient: authenticate success, global_id 4461 2023-09-30T01:34:48.589 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:48.571+0000 7f416affd700 1 -- 172.21.15.88:0/39033161 <== mon.6 v2:172.21.15.88:3303/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f417400c920 con 0x7f41840976f0 2023-09-30T01:34:48.589 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:48.571+0000 7f416affd700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:34:48.590 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:48.571+0000 7f416affd700 1 -- 172.21.15.88:0/39033161 <== mon.6 v2:172.21.15.88:3303/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f4174006030 con 0x7f41840976f0 2023-09-30T01:34:48.590 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:48.571+0000 7f416affd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:34:48.590 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:48.571+0000 7f416affd700 10 monclient: got monmap 1 from mon.d (according to old e1) 2023-09-30T01:34:48.590 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:48.571+0000 7f41817fa700 4 set_mon_vals no callback set 2023-09-30T01:34:48.590 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:48.571+0000 7f416affd700 10 monclient: dump: 2023-09-30T01:34:48.590 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:34:48.590 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:34:48.590 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:34:48.590 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:34:48.590 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:34:48.590 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:34:48.590 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:34:48.591 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:34:48.591 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:34:48.591 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:34:48.591 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:34:48.591 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:34:48.591 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:34:48.591 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:34:48.591 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:34:48.591 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:34:48.591 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:48.571+0000 7f4183fff700 10 monclient: _renew_subs 2023-09-30T01:34:48.591 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:48.571+0000 7f4183fff700 10 monclient: _send_mon_message to mon.d at v2:172.21.15.88:3303/0 2023-09-30T01:34:48.591 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:48.571+0000 7f4183fff700 1 -- 172.21.15.88:0/39033161 --> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f417800b790 con 0x7f41840976f0 2023-09-30T01:34:48.591 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:48.571+0000 7f4183fff700 10 monclient: _renew_subs 2023-09-30T01:34:48.592 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:48.571+0000 7f4183fff700 10 monclient: _send_mon_message to mon.d at v2:172.21.15.88:3303/0 2023-09-30T01:34:48.592 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:48.571+0000 7f4183fff700 1 -- 172.21.15.88:0/39033161 --> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f418406efd0 con 0x7f41840976f0 2023-09-30T01:34:48.592 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:48.571+0000 7f416affd700 1 -- 172.21.15.88:0/39033161 <== mon.6 v2:172.21.15.88:3303/0 4 ==== mgrmap(e 5) v1 ==== 40144+0+0 (secure 0 0 0) 0x7f4174005a00 con 0x7f41840976f0 2023-09-30T01:34:48.592 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:48.571+0000 7f416affd700 1 --2- 172.21.15.88:0/39033161 >> v2:172.21.15.107:6812/29926 conn(0x7f4170033360 0x7f4170035810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:34:48.592 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:48.571+0000 7f4168ff9700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:34:48.592 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:48.571+0000 7f4168ff9700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:34:48.592 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:48.571+0000 7f4168ff9700 10 monclient: _send_mon_message to mon.d at v2:172.21.15.88:3303/0 2023-09-30T01:34:48.592 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:48.571+0000 7f4168ff9700 1 -- 172.21.15.88:0/39033161 --> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f415c0050b0 con 0x7f41840976f0 2023-09-30T01:34:48.592 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:48.571+0000 7f416affd700 1 -- 172.21.15.88:0/39033161 <== mon.6 v2:172.21.15.88:3303/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 3502+0+0 (secure 0 0 0) 0x7f41740438b0 con 0x7f41840976f0 2023-09-30T01:34:48.592 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:48.571+0000 7f41827fc700 1 --2- 172.21.15.88:0/39033161 >> v2:172.21.15.107:6812/29926 conn(0x7f4170033360 0x7f4170035810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:34:48.592 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:48.571+0000 7f41827fc700 10 monclient: get_auth_request con 0x7f4170033360 auth_method 0 2023-09-30T01:34:48.592 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:48.571+0000 7f41827fc700 1 --2- 172.21.15.88:0/39033161 >> v2:172.21.15.107:6812/29926 conn(0x7f4170033360 0x7f4170035810 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f417800bb20 tx=0x7f417801b000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:34:48.593 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:48.579+0000 7f416affd700 1 -- 172.21.15.88:0/39033161 <== mon.6 v2:172.21.15.88:3303/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f4174005cb0 con 0x7f41840976f0 2023-09-30T01:34:48.593 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:48.579+0000 7f416affd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:34:48.593 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:48.579+0000 7f416affd700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:34:48.777 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:48.775+0000 7f4168ff9700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T01:34:48.777 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:48.775+0000 7f4168ff9700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:34:48.777 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:48.775+0000 7f4168ff9700 10 monclient: _send_mon_message to mon.d at v2:172.21.15.88:3303/0 2023-09-30T01:34:48.777 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:48.775+0000 7f4168ff9700 1 -- 172.21.15.88:0/39033161 --> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f415c005630 con 0x7f41840976f0 2023-09-30T01:34:48.778 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:48.775+0000 7f416affd700 1 -- 172.21.15.88:0/39033161 <== mon.6 v2:172.21.15.88:3303/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f417402a080 con 0x7f41840976f0 2023-09-30T01:34:48.778 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:48.775+0000 7f416affd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:34:48.778 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:48.775+0000 7f416affd700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:34:48.778 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-09-30T01:34:48.778 INFO:teuthology.orchestra.run.smithi088.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:34:48.780 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:48.779+0000 7f4168ff9700 1 -- 172.21.15.88:0/39033161 >> v2:172.21.15.107:6812/29926 conn(0x7f4170033360 msgr2=0x7f4170035810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:34:48.780 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:48.779+0000 7f4168ff9700 1 --2- 172.21.15.88:0/39033161 >> v2:172.21.15.107:6812/29926 conn(0x7f4170033360 0x7f4170035810 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f417800bb20 tx=0x7f417801b000 comp rx=0 tx=0).stop 2023-09-30T01:34:48.780 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:48.779+0000 7f4168ff9700 10 monclient: shutdown 2023-09-30T01:34:48.780 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:48.779+0000 7f4168ff9700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:34:48.780 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:48.779+0000 7f4168ff9700 1 -- 172.21.15.88:0/39033161 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7f41840976f0 msgr2=0x7f418412e840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:34:48.780 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:48.779+0000 7f4168ff9700 1 --2- 172.21.15.88:0/39033161 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7f41840976f0 0x7f418412e840 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f417400ec80 tx=0x7f4174009120 comp rx=0 tx=0).stop 2023-09-30T01:34:48.780 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:48.779+0000 7f4168ff9700 1 -- 172.21.15.88:0/39033161 shutdown_connections 2023-09-30T01:34:48.780 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:48.779+0000 7f4168ff9700 1 --2- 172.21.15.88:0/39033161 >> v2:172.21.15.107:6812/29926 conn(0x7f4170033360 0x7f4170035810 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:34:48.780 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:48.779+0000 7f4168ff9700 1 --2- 172.21.15.88:0/39033161 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f418409f690 0x7f418413e360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:34:48.780 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:48.779+0000 7f4168ff9700 1 --2- 172.21.15.88:0/39033161 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f418409ed80 0x7f418413de20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:34:48.781 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:48.779+0000 7f4168ff9700 1 --2- 172.21.15.88:0/39033161 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7f41840976f0 0x7f418412e840 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:34:48.781 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:48.779+0000 7f4168ff9700 1 -- 172.21.15.88:0/39033161 >> 172.21.15.88:0/39033161 conn(0x7f41840a0910 msgr2=0x7f41841319f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:34:48.781 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:48.779+0000 7f4168ff9700 1 -- 172.21.15.88:0/39033161 shutdown_connections 2023-09-30T01:34:48.781 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:48.779+0000 7f4168ff9700 1 -- 172.21.15.88:0/39033161 wait complete. 2023-09-30T01:34:51.791 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:34:51.906 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:51.903+0000 7f74e0ad1700 10 monclient: get_monmap_and_config 2023-09-30T01:34:51.907 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:51.903+0000 7f74e0ad1700 10 monclient: build_initial_monmap 2023-09-30T01:34:51.907 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:51.903+0000 7f74e0ad1700 10 monclient: monmap: 2023-09-30T01:34:51.907 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2023-09-30T01:34:51.907 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:34:51.907 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:34:51.906986+0000 2023-09-30T01:34:51.907 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:34:51.906986+0000 2023-09-30T01:34:51.907 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2023-09-30T01:34:51.907 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:34:51.907 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a-legacy 2023-09-30T01:34:51.907 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.107:6789/0 mon.noname-f-legacy 2023-09-30T01:34:51.907 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.noname-b 2023-09-30T01:34:51.908 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.noname-g 2023-09-30T01:34:51.908 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.noname-c 2023-09-30T01:34:51.908 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.noname-h 2023-09-30T01:34:51.908 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.noname-d 2023-09-30T01:34:51.908 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.noname-i 2023-09-30T01:34:51.908 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.noname-e 2023-09-30T01:34:51.908 INFO:teuthology.orchestra.run.smithi088.stderr:9: v2:172.21.15.88:3300/0 mon.noname-a 2023-09-30T01:34:51.908 INFO:teuthology.orchestra.run.smithi088.stderr:10: v2:172.21.15.107:3300/0 mon.noname-f 2023-09-30T01:34:51.908 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:34:51.912 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:51.903+0000 7f74e0ad1700 1 Processor -- start 2023-09-30T01:34:51.913 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:51.903+0000 7f74e0ad1700 1 -- start start 2023-09-30T01:34:51.913 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:51.903+0000 7f74e0ad1700 10 monclient: init 2023-09-30T01:34:51.913 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:51.907+0000 7f74e0ad1700 10 monclient: _reopen_session rank -1 2023-09-30T01:34:51.913 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:51.907+0000 7f74e0ad1700 10 monclient: _add_conns ranks=[10,8,9,7,5,1,6,2,0,4,3] 2023-09-30T01:34:51.913 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:51.907+0000 7f74e0ad1700 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f74dc0a3380 0x7f74dc0a48b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:34:51.913 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:51.907+0000 7f74e0ad1700 10 monclient(hunting): picked mon.noname-f con 0x7f74dc0a3380 addr v2:172.21.15.107:3300/0 2023-09-30T01:34:51.913 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:51.907+0000 7f74e0ad1700 1 --2- >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f74dc09ffd0 0x7f74dc0a03a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:34:51.913 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:51.907+0000 7f74e0ad1700 10 monclient(hunting): picked mon.noname-e con 0x7f74dc09ffd0 addr [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] 2023-09-30T01:34:51.913 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:51.907+0000 7f74e0ad1700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f74dc1333b0 0x7f74dc133820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:34:51.914 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:51.907+0000 7f74e0ad1700 10 monclient(hunting): picked mon.noname-a con 0x7f74dc1333b0 addr v2:172.21.15.88:3300/0 2023-09-30T01:34:51.914 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:51.907+0000 7f74e0ad1700 10 monclient(hunting): start opening mon connection 2023-09-30T01:34:51.914 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:51.907+0000 7f74e0ad1700 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f74dc097ab0 con 0x7f74dc1333b0 2023-09-30T01:34:51.914 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:51.907+0000 7f74e0ad1700 10 monclient(hunting): start opening mon connection 2023-09-30T01:34:51.914 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:51.907+0000 7f74e0ad1700 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f74dc0a08e0 con 0x7f74dc0a3380 2023-09-30T01:34:51.914 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:51.907+0000 7f74e0ad1700 10 monclient(hunting): start opening mon connection 2023-09-30T01:34:51.914 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:51.907+0000 7f74e0ad1700 1 -- --> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] -- mon_getmap magic: 0 v1 -- 0x7f74dc098480 con 0x7f74dc09ffd0 2023-09-30T01:34:51.914 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:51.907+0000 7f74e0ad1700 10 monclient(hunting): _renew_subs 2023-09-30T01:34:51.914 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:51.907+0000 7f74e0ad1700 10 monclient(hunting): authenticate will time out at 963.944551s 2023-09-30T01:34:51.914 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:51.907+0000 7f74dbfff700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f74dc1333b0 0x7f74dc133820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:34:51.914 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:51.907+0000 7f74d3fff700 1 --2- >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f74dc09ffd0 0x7f74dc0a03a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:34:51.915 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:51.907+0000 7f74dbfff700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f74dc1333b0 0x7f74dc133820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:33222/0 (socket says 172.21.15.88:33222) 2023-09-30T01:34:51.915 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:51.907+0000 7f74dbfff700 1 -- 172.21.15.88:0/3759309567 learned_addr learned my addr 172.21.15.88:0/3759309567 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-09-30T01:34:51.915 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:51.907+0000 7f74dbfff700 10 monclient(hunting): get_auth_request con 0x7f74dc1333b0 auth_method 0 2023-09-30T01:34:51.915 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:51.907+0000 7f74dbfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:34:51.915 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:51.907+0000 7f74dbfff700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:34:51.915 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:51.907+0000 7f74dbfff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:34:51.915 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:51.907+0000 7f74d3fff700 10 monclient(hunting): get_auth_request con 0x7f74dc09ffd0 auth_method 0 2023-09-30T01:34:51.915 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:51.907+0000 7f74d3fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:34:51.915 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:51.907+0000 7f74d3fff700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:34:51.915 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:51.907+0000 7f74d3fff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:34:51.915 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:51.907+0000 7f74db7fe700 1 --2- 172.21.15.88:0/3759309567 >> v2:172.21.15.107:3300/0 conn(0x7f74dc0a3380 0x7f74dc0a48b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:34:51.915 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:51.907+0000 7f74dbfff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:34:51.916 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:51.907+0000 7f74dbfff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:34:51.916 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:51.907+0000 7f74dbfff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:34:51.916 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:51.907+0000 7f74db7fe700 10 monclient(hunting): get_auth_request con 0x7f74dc0a3380 auth_method 0 2023-09-30T01:34:51.916 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:51.907+0000 7f74db7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:34:51.916 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:51.907+0000 7f74db7fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:34:51.916 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:51.907+0000 7f74db7fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:34:51.916 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:51.907+0000 7f74d3fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:34:51.916 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:51.907+0000 7f74d3fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:34:51.916 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:51.907+0000 7f74d3fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:34:51.916 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:51.907+0000 7f74db7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:34:51.916 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:51.907+0000 7f74db7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:34:51.916 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:51.907+0000 7f74db7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:34:51.916 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:51.907+0000 7f74dbfff700 10 monclient(hunting): handle_auth_done global_id 4707 payload 274 2023-09-30T01:34:51.917 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:51.907+0000 7f74dbfff700 1 -- 172.21.15.88:0/3759309567 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f74dc09ffd0 msgr2=0x7f74dc0a03a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:34:51.917 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:51.907+0000 7f74dbfff700 1 --2- 172.21.15.88:0/3759309567 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f74dc09ffd0 0x7f74dc0a03a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:34:51.917 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:51.907+0000 7f74dbfff700 1 -- 172.21.15.88:0/3759309567 >> v2:172.21.15.107:3300/0 conn(0x7f74dc0a3380 msgr2=0x7f74dc0a48b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:34:51.917 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:51.907+0000 7f74dbfff700 1 --2- 172.21.15.88:0/3759309567 >> v2:172.21.15.107:3300/0 conn(0x7f74dc0a3380 0x7f74dc0a48b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:34:51.917 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:51.907+0000 7f74dbfff700 10 monclient: _finish_hunting 0 2023-09-30T01:34:51.917 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:51.907+0000 7f74dbfff700 1 monclient: found mon.noname-a 2023-09-30T01:34:51.917 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:51.907+0000 7f74dbfff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:34:51.917 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:51.907+0000 7f74dbfff700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.88:3300/0 2023-09-30T01:34:51.917 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:51.907+0000 7f74dbfff700 1 -- 172.21.15.88:0/3759309567 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74dc04bb20 con 0x7f74dc1333b0 2023-09-30T01:34:51.917 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:51.907+0000 7f74dbfff700 1 --2- 172.21.15.88:0/3759309567 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f74dc1333b0 0x7f74dc133820 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f74dc0a39c0 tx=0x7f74d4009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c3a6750fbc081e28 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:34:51.917 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:51.907+0000 7f74daffd700 1 -- 172.21.15.88:0/3759309567 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f74d400e070 con 0x7f74dc1333b0 2023-09-30T01:34:51.917 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:51.907+0000 7f74daffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:34:51.918 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:51.907+0000 7f74daffd700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T01:34:51.918 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:51.907+0000 7f74daffd700 10 monclient: dump: 2023-09-30T01:34:51.918 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:34:51.918 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:34:51.918 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:34:51.918 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:34:51.918 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:34:51.918 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:34:51.918 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:34:51.918 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:34:51.918 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:34:51.918 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:34:51.919 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:34:51.919 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:34:51.919 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:34:51.919 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:34:51.919 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:34:51.919 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:34:51.919 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:51.907+0000 7f74daffd700 10 monclient: _finish_auth 0 2023-09-30T01:34:51.919 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:51.907+0000 7f74daffd700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:34:51.919 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:51.907+0000 7f74e0ad1700 5 monclient: authenticate success, global_id 4707 2023-09-30T01:34:51.919 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:51.907+0000 7f74daffd700 1 -- 172.21.15.88:0/3759309567 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f74d400bdb0 con 0x7f74dc1333b0 2023-09-30T01:34:51.919 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:51.907+0000 7f74daffd700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:34:51.919 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:51.907+0000 7f74e0ad1700 10 monclient: get_monmap_and_config success 2023-09-30T01:34:51.919 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:51.907+0000 7f74e0ad1700 4 set_mon_vals no callback set 2023-09-30T01:34:51.920 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:51.907+0000 7f74daffd700 1 -- 172.21.15.88:0/3759309567 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f74d40092e0 con 0x7f74dc1333b0 2023-09-30T01:34:51.920 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:51.907+0000 7f74daffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:34:51.920 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:51.907+0000 7f74daffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:34:51.920 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:51.907+0000 7f74daffd700 10 monclient: dump: 2023-09-30T01:34:51.920 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:34:51.920 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:34:51.920 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:34:51.920 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:34:51.920 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:34:51.920 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:34:51.920 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:34:51.920 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:34:51.921 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:34:51.921 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:34:51.921 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:34:51.921 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:34:51.921 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:34:51.921 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:34:51.921 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:34:51.921 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:34:51.921 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:51.907+0000 7f74e0ad1700 10 monclient: shutdown 2023-09-30T01:34:51.921 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:51.907+0000 7f74e0ad1700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:34:51.921 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:51.907+0000 7f74e0ad1700 1 -- 172.21.15.88:0/3759309567 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f74dc1333b0 msgr2=0x7f74dc133820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:34:51.921 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:51.907+0000 7f74e0ad1700 1 --2- 172.21.15.88:0/3759309567 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f74dc1333b0 0x7f74dc133820 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f74dc0a39c0 tx=0x7f74d4009840 comp rx=0 tx=0).stop 2023-09-30T01:34:51.921 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:51.907+0000 7f74e0ad1700 1 -- 172.21.15.88:0/3759309567 shutdown_connections 2023-09-30T01:34:51.922 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:51.907+0000 7f74e0ad1700 1 --2- 172.21.15.88:0/3759309567 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f74dc1333b0 0x7f74dc133820 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:34:51.922 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:51.907+0000 7f74e0ad1700 1 --2- 172.21.15.88:0/3759309567 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f74dc09ffd0 0x7f74dc0a03a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:34:51.922 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:51.907+0000 7f74e0ad1700 1 --2- 172.21.15.88:0/3759309567 >> v2:172.21.15.107:3300/0 conn(0x7f74dc0a3380 0x7f74dc0a48b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:34:51.922 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:51.907+0000 7f74e0ad1700 1 -- 172.21.15.88:0/3759309567 >> 172.21.15.88:0/3759309567 conn(0x7f74dc06e5e0 msgr2=0x7f74dc06e9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:34:51.922 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:51.907+0000 7f74e0ad1700 1 -- 172.21.15.88:0/3759309567 shutdown_connections 2023-09-30T01:34:51.922 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:51.907+0000 7f74e0ad1700 1 -- 172.21.15.88:0/3759309567 wait complete. 2023-09-30T01:34:51.922 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:51.911+0000 7f74e0ad1700 10 monclient: build_initial_monmap 2023-09-30T01:34:51.922 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:51.911+0000 7f74e0ad1700 10 monclient: monmap: 2023-09-30T01:34:51.922 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2023-09-30T01:34:51.922 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:34:51.922 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 0.000000 2023-09-30T01:34:51.922 INFO:teuthology.orchestra.run.smithi088.stderr:created 0.000000 2023-09-30T01:34:51.923 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2023-09-30T01:34:51.923 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 1 2023-09-30T01:34:51.923 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.noname-a 2023-09-30T01:34:51.923 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.noname-f 2023-09-30T01:34:51.923 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.noname-b 2023-09-30T01:34:51.923 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.noname-g 2023-09-30T01:34:51.923 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.noname-c 2023-09-30T01:34:51.923 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.noname-h 2023-09-30T01:34:51.923 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.noname-d 2023-09-30T01:34:51.923 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.noname-i 2023-09-30T01:34:51.923 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.noname-e 2023-09-30T01:34:51.923 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:34:51.924 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:51.911+0000 7f74e0ad1700 1 Processor -- start 2023-09-30T01:34:51.924 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:51.911+0000 7f74e0ad1700 1 -- start start 2023-09-30T01:34:51.924 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:51.911+0000 7f74e0ad1700 10 monclient: init 2023-09-30T01:34:51.927 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:51.911+0000 7f74e0ad1700 10 monclient: _reopen_session rank -1 2023-09-30T01:34:51.927 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:51.911+0000 7f74e0ad1700 10 monclient: _add_conns ranks=[1,5,4,8,7,3,2,6,0] 2023-09-30T01:34:51.928 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:51.911+0000 7f74e0ad1700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f74dc09ffd0 0x7f74dc142650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:34:51.928 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:51.911+0000 7f74e0ad1700 10 monclient(hunting): picked mon.noname-f con 0x7f74dc09ffd0 addr [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] 2023-09-30T01:34:51.928 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:51.911+0000 7f74e0ad1700 1 --2- >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7f74dc0a3380 0x7f74dc09cd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:34:51.928 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:51.911+0000 7f74e0ad1700 10 monclient(hunting): picked mon.noname-h con 0x7f74dc0a3380 addr [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] 2023-09-30T01:34:51.928 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:51.911+0000 7f74e0ad1700 1 --2- >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f74dc1333b0 0x7f74dc09d280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:34:51.928 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:51.911+0000 7f74e0ad1700 10 monclient(hunting): picked mon.noname-c con 0x7f74dc1333b0 addr [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] 2023-09-30T01:34:51.928 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:51.911+0000 7f74e0ad1700 10 monclient(hunting): start opening mon connection 2023-09-30T01:34:51.928 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:51.911+0000 7f74e0ad1700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74dc097ce0 con 0x7f74dc09ffd0 2023-09-30T01:34:51.928 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:51.911+0000 7f74e0ad1700 10 monclient(hunting): start opening mon connection 2023-09-30T01:34:51.928 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:51.911+0000 7f74e0ad1700 1 -- --> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] -- mon_getmap magic: 0 v1 -- 0x7f74dc09f390 con 0x7f74dc1333b0 2023-09-30T01:34:51.928 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:51.911+0000 7f74e0ad1700 10 monclient(hunting): start opening mon connection 2023-09-30T01:34:51.928 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:51.911+0000 7f74e0ad1700 1 -- --> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] -- mon_getmap magic: 0 v1 -- 0x7f74dc0a0ac0 con 0x7f74dc0a3380 2023-09-30T01:34:51.929 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:51.911+0000 7f74e0ad1700 10 monclient(hunting): _renew_subs 2023-09-30T01:34:51.929 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:51.911+0000 7f74e0ad1700 10 monclient(hunting): authenticate will time out at 963.949984s 2023-09-30T01:34:51.929 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:51.911+0000 7f74dbfff700 1 --2- >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f74dc1333b0 0x7f74dc09d280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:34:51.929 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:51.911+0000 7f74dbfff700 1 --2- >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f74dc1333b0 0x7f74dc09d280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3302/0 says I am v2:172.21.15.88:39154/0 (socket says 172.21.15.88:39154) 2023-09-30T01:34:51.929 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:51.911+0000 7f74dbfff700 1 -- 172.21.15.88:0/150912103 learned_addr learned my addr 172.21.15.88:0/150912103 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-09-30T01:34:51.929 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:51.911+0000 7f74dbfff700 10 monclient(hunting): get_auth_request con 0x7f74dc1333b0 auth_method 0 2023-09-30T01:34:51.929 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:51.911+0000 7f74dbfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:34:51.929 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:51.911+0000 7f74dbfff700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:34:51.929 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:51.911+0000 7f74dbfff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:34:51.929 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:51.911+0000 7f74dbfff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:34:51.929 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:51.911+0000 7f74dbfff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:34:51.929 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:51.911+0000 7f74dbfff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:34:51.930 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:51.911+0000 7f74db7fe700 1 --2- 172.21.15.88:0/150912103 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f74dc09ffd0 0x7f74dc142650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:34:51.930 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:51.911+0000 7f74d3fff700 1 --2- 172.21.15.88:0/150912103 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7f74dc0a3380 0x7f74dc09cd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:34:51.930 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:51.911+0000 7f74d3fff700 10 monclient(hunting): get_auth_request con 0x7f74dc0a3380 auth_method 0 2023-09-30T01:34:51.930 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:51.911+0000 7f74d3fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:34:51.930 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:51.911+0000 7f74d3fff700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:34:51.930 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:51.911+0000 7f74d3fff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:34:51.930 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:51.911+0000 7f74db7fe700 10 monclient(hunting): get_auth_request con 0x7f74dc09ffd0 auth_method 0 2023-09-30T01:34:51.930 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:51.911+0000 7f74db7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:34:51.930 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:51.911+0000 7f74db7fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:34:51.930 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:51.911+0000 7f74db7fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:34:51.930 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:51.911+0000 7f74d3fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:34:51.931 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:51.911+0000 7f74d3fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:34:51.931 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:51.911+0000 7f74d3fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:34:51.931 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:51.911+0000 7f74db7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:34:51.931 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:51.911+0000 7f74db7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:34:51.931 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:51.911+0000 7f74db7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:34:51.931 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:51.911+0000 7f74dbfff700 10 monclient(hunting): handle_auth_done global_id 4540 payload 824 2023-09-30T01:34:51.931 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:51.911+0000 7f74dbfff700 1 -- 172.21.15.88:0/150912103 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7f74dc0a3380 msgr2=0x7f74dc09cd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:34:51.931 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:51.911+0000 7f74dbfff700 1 --2- 172.21.15.88:0/150912103 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7f74dc0a3380 0x7f74dc09cd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:34:51.931 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:51.911+0000 7f74dbfff700 1 -- 172.21.15.88:0/150912103 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f74dc09ffd0 msgr2=0x7f74dc142650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:34:51.931 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:51.911+0000 7f74dbfff700 1 --2- 172.21.15.88:0/150912103 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f74dc09ffd0 0x7f74dc142650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:34:51.931 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:51.911+0000 7f74dbfff700 10 monclient: _finish_hunting 0 2023-09-30T01:34:51.931 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:51.911+0000 7f74dbfff700 1 monclient: found mon.noname-c 2023-09-30T01:34:51.932 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:51.911+0000 7f74dbfff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:34:51.932 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:51.911+0000 7f74dbfff700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.88:3302/0 2023-09-30T01:34:51.932 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:51.911+0000 7f74dbfff700 1 -- 172.21.15.88:0/150912103 --> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74d4009ae0 con 0x7f74dc1333b0 2023-09-30T01:34:51.932 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:51.911+0000 7f74dbfff700 1 --2- 172.21.15.88:0/150912103 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f74dc1333b0 0x7f74dc09d280 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f74d4012000 tx=0x7f74d4018000 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:34:51.932 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:51.911+0000 7f74d37fe700 1 -- 172.21.15.88:0/150912103 <== mon.4 v2:172.21.15.88:3302/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f74d4003300 con 0x7f74dc1333b0 2023-09-30T01:34:51.932 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:51.911+0000 7f74d37fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:34:51.932 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:51.911+0000 7f74d37fe700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T01:34:51.932 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:51.911+0000 7f74d37fe700 10 monclient: dump: 2023-09-30T01:34:51.932 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:34:51.932 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:34:51.932 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:34:51.932 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:34:51.933 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:34:51.933 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:34:51.933 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:34:51.933 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:34:51.933 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:34:51.933 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:34:51.933 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:34:51.933 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:34:51.933 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:34:51.933 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:34:51.933 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:34:51.933 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:34:51.934 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:51.911+0000 7f74d37fe700 10 monclient: _finish_auth 0 2023-09-30T01:34:51.934 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:51.911+0000 7f74d37fe700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:34:51.934 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:51.911+0000 7f74e0ad1700 5 monclient: authenticate success, global_id 4540 2023-09-30T01:34:51.934 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:51.911+0000 7f74d37fe700 1 -- 172.21.15.88:0/150912103 <== mon.4 v2:172.21.15.88:3302/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f74d40092e0 con 0x7f74dc1333b0 2023-09-30T01:34:51.934 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:51.911+0000 7f74e0ad1700 10 monclient: _renew_subs 2023-09-30T01:34:51.934 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:51.911+0000 7f74e0ad1700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.88:3302/0 2023-09-30T01:34:51.934 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:51.911+0000 7f74e0ad1700 1 -- 172.21.15.88:0/150912103 --> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f74dc09eea0 con 0x7f74dc1333b0 2023-09-30T01:34:51.934 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:51.911+0000 7f74d37fe700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:34:51.934 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:51.911+0000 7f74d37fe700 1 -- 172.21.15.88:0/150912103 <== mon.4 v2:172.21.15.88:3302/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f74d401c890 con 0x7f74dc1333b0 2023-09-30T01:34:51.934 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:51.911+0000 7f74d37fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:34:51.934 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:51.911+0000 7f74da7fc700 4 set_mon_vals no callback set 2023-09-30T01:34:51.934 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:51.911+0000 7f74d37fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T01:34:51.935 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:51.911+0000 7f74d37fe700 10 monclient: dump: 2023-09-30T01:34:51.935 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:34:51.935 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:34:51.935 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:34:51.935 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:34:51.935 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:34:51.935 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:34:51.935 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:34:51.935 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:34:51.935 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:34:51.935 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:34:51.936 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:34:51.936 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:34:51.936 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:34:51.936 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:34:51.936 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:34:51.936 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:34:51.936 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:51.911+0000 7f74e0ad1700 10 monclient: _renew_subs 2023-09-30T01:34:51.936 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:51.911+0000 7f74e0ad1700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.88:3302/0 2023-09-30T01:34:51.936 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:51.911+0000 7f74e0ad1700 1 -- 172.21.15.88:0/150912103 --> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f74dc09f560 con 0x7f74dc1333b0 2023-09-30T01:34:51.936 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:51.915+0000 7f74d37fe700 1 -- 172.21.15.88:0/150912103 <== mon.4 v2:172.21.15.88:3302/0 4 ==== mgrmap(e 5) v1 ==== 40144+0+0 (secure 0 0 0) 0x7f74d4003050 con 0x7f74dc1333b0 2023-09-30T01:34:51.936 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:51.915+0000 7f74e0ad1700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:34:51.936 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:51.915+0000 7f74e0ad1700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:34:51.937 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:51.915+0000 7f74e0ad1700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.88:3302/0 2023-09-30T01:34:51.937 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:51.915+0000 7f74d37fe700 1 --2- 172.21.15.88:0/150912103 >> v2:172.21.15.107:6812/29926 conn(0x7f74bc033360 0x7f74bc035810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:34:51.937 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:51.915+0000 7f74e0ad1700 1 -- 172.21.15.88:0/150912103 --> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f74dc0710d0 con 0x7f74dc1333b0 2023-09-30T01:34:51.937 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:51.915+0000 7f74d37fe700 1 -- 172.21.15.88:0/150912103 <== mon.4 v2:172.21.15.88:3302/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 3502+0+0 (secure 0 0 0) 0x7f74d4044990 con 0x7f74dc1333b0 2023-09-30T01:34:51.937 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:51.915+0000 7f74db7fe700 1 --2- 172.21.15.88:0/150912103 >> v2:172.21.15.107:6812/29926 conn(0x7f74bc033360 0x7f74bc035810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:34:51.937 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:51.915+0000 7f74db7fe700 10 monclient: get_auth_request con 0x7f74bc033360 auth_method 0 2023-09-30T01:34:51.937 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:51.915+0000 7f74db7fe700 1 --2- 172.21.15.88:0/150912103 >> v2:172.21.15.107:6812/29926 conn(0x7f74bc033360 0x7f74bc035810 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f74cc009db0 tx=0x7f74cc0091a0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:34:51.937 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:51.923+0000 7f74d37fe700 1 -- 172.21.15.88:0/150912103 <== mon.4 v2:172.21.15.88:3302/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f74d4007b00 con 0x7f74dc1333b0 2023-09-30T01:34:51.937 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:51.923+0000 7f74d37fe700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:34:51.937 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:51.923+0000 7f74d37fe700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:34:52.117 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:52.115+0000 7f74e0ad1700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T01:34:52.117 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:52.115+0000 7f74e0ad1700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:34:52.117 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:52.115+0000 7f74e0ad1700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.88:3302/0 2023-09-30T01:34:52.117 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:52.115+0000 7f74e0ad1700 1 -- 172.21.15.88:0/150912103 --> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f74dc067b60 con 0x7f74dc1333b0 2023-09-30T01:34:52.118 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:52.115+0000 7f74d37fe700 1 -- 172.21.15.88:0/150912103 <== mon.4 v2:172.21.15.88:3302/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f74d401abb0 con 0x7f74dc1333b0 2023-09-30T01:34:52.118 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:52.115+0000 7f74d37fe700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:34:52.118 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:52.115+0000 7f74d37fe700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:34:52.118 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-09-30T01:34:52.118 INFO:teuthology.orchestra.run.smithi088.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:34:52.120 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:52.119+0000 7f74e0ad1700 1 -- 172.21.15.88:0/150912103 >> v2:172.21.15.107:6812/29926 conn(0x7f74bc033360 msgr2=0x7f74bc035810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:34:52.120 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:52.119+0000 7f74e0ad1700 1 --2- 172.21.15.88:0/150912103 >> v2:172.21.15.107:6812/29926 conn(0x7f74bc033360 0x7f74bc035810 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f74cc009db0 tx=0x7f74cc0091a0 comp rx=0 tx=0).stop 2023-09-30T01:34:52.120 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:52.119+0000 7f74e0ad1700 10 monclient: shutdown 2023-09-30T01:34:52.120 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:52.119+0000 7f74e0ad1700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:34:52.120 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:52.119+0000 7f74e0ad1700 1 -- 172.21.15.88:0/150912103 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f74dc1333b0 msgr2=0x7f74dc09d280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:34:52.120 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:52.119+0000 7f74e0ad1700 1 --2- 172.21.15.88:0/150912103 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f74dc1333b0 0x7f74dc09d280 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f74d4012000 tx=0x7f74d4018000 comp rx=0 tx=0).stop 2023-09-30T01:34:52.121 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:52.119+0000 7f74e0ad1700 1 -- 172.21.15.88:0/150912103 shutdown_connections 2023-09-30T01:34:52.121 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:52.119+0000 7f74e0ad1700 1 --2- 172.21.15.88:0/150912103 >> v2:172.21.15.107:6812/29926 conn(0x7f74bc033360 0x7f74bc035810 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:34:52.121 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:52.119+0000 7f74e0ad1700 1 --2- 172.21.15.88:0/150912103 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f74dc1333b0 0x7f74dc09d280 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:34:52.121 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:52.119+0000 7f74e0ad1700 1 --2- 172.21.15.88:0/150912103 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7f74dc0a3380 0x7f74dc09cd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:34:52.121 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:52.119+0000 7f74e0ad1700 1 --2- 172.21.15.88:0/150912103 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f74dc09ffd0 0x7f74dc142650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:34:52.121 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:52.119+0000 7f74e0ad1700 1 -- 172.21.15.88:0/150912103 >> 172.21.15.88:0/150912103 conn(0x7f74dc06e5e0 msgr2=0x7f74dc131e60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:34:52.122 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:52.119+0000 7f74e0ad1700 1 -- 172.21.15.88:0/150912103 shutdown_connections 2023-09-30T01:34:52.122 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:52.119+0000 7f74e0ad1700 1 -- 172.21.15.88:0/150912103 wait complete. 2023-09-30T01:34:55.133 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:34:55.255 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:55.243+0000 7fe86d782700 10 monclient: get_monmap_and_config 2023-09-30T01:34:55.255 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:55.243+0000 7fe86d782700 10 monclient: build_initial_monmap 2023-09-30T01:34:55.256 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:55.243+0000 7fe86d782700 10 monclient: monmap: 2023-09-30T01:34:55.256 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2023-09-30T01:34:55.256 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:34:55.256 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:34:55.245519+0000 2023-09-30T01:34:55.256 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:34:55.245519+0000 2023-09-30T01:34:55.256 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2023-09-30T01:34:55.256 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:34:55.256 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a-legacy 2023-09-30T01:34:55.256 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.107:6789/0 mon.noname-f-legacy 2023-09-30T01:34:55.256 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.noname-b 2023-09-30T01:34:55.257 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.noname-g 2023-09-30T01:34:55.257 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.noname-c 2023-09-30T01:34:55.257 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.noname-h 2023-09-30T01:34:55.257 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.noname-d 2023-09-30T01:34:55.257 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.noname-i 2023-09-30T01:34:55.257 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.noname-e 2023-09-30T01:34:55.257 INFO:teuthology.orchestra.run.smithi088.stderr:9: v2:172.21.15.88:3300/0 mon.noname-a 2023-09-30T01:34:55.257 INFO:teuthology.orchestra.run.smithi088.stderr:10: v2:172.21.15.107:3300/0 mon.noname-f 2023-09-30T01:34:55.257 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:34:55.257 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:55.243+0000 7fe86d782700 1 Processor -- start 2023-09-30T01:34:55.257 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:55.243+0000 7fe86d782700 1 -- start start 2023-09-30T01:34:55.258 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:55.243+0000 7fe86d782700 10 monclient: init 2023-09-30T01:34:55.258 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:55.243+0000 7fe86d782700 10 monclient: _reopen_session rank -1 2023-09-30T01:34:55.258 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:55.243+0000 7fe86d782700 10 monclient: _add_conns ranks=[4,3,7,2,1,9,5,0,10,8,6] 2023-09-30T01:34:55.258 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:55.243+0000 7fe86d782700 1 --2- >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7fe868097a40 0x7fe868132700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:34:55.258 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:55.243+0000 7fe86d782700 10 monclient(hunting): picked mon.noname-c con 0x7fe868097a40 addr [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] 2023-09-30T01:34:55.258 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:55.243+0000 7fe86d782700 1 --2- >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7fe8681311c0 0x7fe868132c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:34:55.258 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:55.243+0000 7fe86d782700 10 monclient(hunting): picked mon.noname-g con 0x7fe8681311c0 addr [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] 2023-09-30T01:34:55.258 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:55.243+0000 7fe86d782700 1 --2- >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7fe8681332e0 0x7fe868133710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:34:55.258 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:55.243+0000 7fe86d782700 10 monclient(hunting): picked mon.noname-i con 0x7fe8681332e0 addr [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] 2023-09-30T01:34:55.258 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:55.243+0000 7fe86d782700 10 monclient(hunting): start opening mon connection 2023-09-30T01:34:55.259 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:55.243+0000 7fe86d782700 1 -- --> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe868131590 con 0x7fe8681311c0 2023-09-30T01:34:55.259 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:55.243+0000 7fe86d782700 10 monclient(hunting): start opening mon connection 2023-09-30T01:34:55.259 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:55.243+0000 7fe86d782700 1 -- --> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] -- mon_getmap magic: 0 v1 -- 0x7fe86809a5d0 con 0x7fe868097a40 2023-09-30T01:34:55.259 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:55.243+0000 7fe86d782700 10 monclient(hunting): start opening mon connection 2023-09-30T01:34:55.259 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:55.243+0000 7fe86d782700 1 -- --> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] -- mon_getmap magic: 0 v1 -- 0x7fe868097e10 con 0x7fe8681332e0 2023-09-30T01:34:55.259 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:55.243+0000 7fe86d782700 10 monclient(hunting): _renew_subs 2023-09-30T01:34:55.259 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:55.243+0000 7fe86d782700 10 monclient(hunting): authenticate will time out at 967.283153s 2023-09-30T01:34:55.259 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:55.243+0000 7fe867fff700 1 --2- >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7fe868097a40 0x7fe868132700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:34:55.260 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:55.243+0000 7fe867fff700 1 --2- >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7fe868097a40 0x7fe868132700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3302/0 says I am v2:172.21.15.88:39156/0 (socket says 172.21.15.88:39156) 2023-09-30T01:34:55.260 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:55.243+0000 7fe867fff700 1 -- 172.21.15.88:0/1610096351 learned_addr learned my addr 172.21.15.88:0/1610096351 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-09-30T01:34:55.260 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:55.243+0000 7fe867fff700 10 monclient(hunting): get_auth_request con 0x7fe868097a40 auth_method 0 2023-09-30T01:34:55.260 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:55.243+0000 7fe867fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:34:55.260 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:55.243+0000 7fe867fff700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:34:55.260 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:55.243+0000 7fe867fff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:34:55.260 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:55.243+0000 7fe85ffff700 1 --2- 172.21.15.88:0/1610096351 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7fe8681311c0 0x7fe868132c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:34:55.260 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:55.243+0000 7fe867fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:34:55.260 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:55.243+0000 7fe867fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:34:55.260 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:55.243+0000 7fe867fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:34:55.260 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:55.243+0000 7fe86cf81700 1 --2- 172.21.15.88:0/1610096351 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7fe8681332e0 0x7fe868133710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:34:55.260 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:55.243+0000 7fe85ffff700 10 monclient(hunting): get_auth_request con 0x7fe8681311c0 auth_method 0 2023-09-30T01:34:55.261 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:55.243+0000 7fe85ffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:34:55.261 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:55.243+0000 7fe85ffff700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:34:55.261 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:55.243+0000 7fe85ffff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:34:55.261 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:55.243+0000 7fe86cf81700 10 monclient(hunting): get_auth_request con 0x7fe8681332e0 auth_method 0 2023-09-30T01:34:55.261 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:55.243+0000 7fe86cf81700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:34:55.261 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:55.243+0000 7fe86cf81700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:34:55.261 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:55.243+0000 7fe86cf81700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:34:55.261 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:55.243+0000 7fe867fff700 10 monclient(hunting): handle_auth_done global_id 4549 payload 274 2023-09-30T01:34:55.261 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:55.243+0000 7fe867fff700 1 -- 172.21.15.88:0/1610096351 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7fe8681332e0 msgr2=0x7fe868133710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:34:55.261 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:55.243+0000 7fe867fff700 1 --2- 172.21.15.88:0/1610096351 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7fe8681332e0 0x7fe868133710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:34:55.261 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:55.243+0000 7fe867fff700 1 -- 172.21.15.88:0/1610096351 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7fe8681311c0 msgr2=0x7fe868132c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:34:55.261 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:55.243+0000 7fe867fff700 1 --2- 172.21.15.88:0/1610096351 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7fe8681311c0 0x7fe868132c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:34:55.262 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:55.243+0000 7fe867fff700 10 monclient: _finish_hunting 0 2023-09-30T01:34:55.262 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:55.243+0000 7fe867fff700 1 monclient: found mon.noname-c 2023-09-30T01:34:55.262 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:55.243+0000 7fe867fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:34:55.262 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:55.243+0000 7fe867fff700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.88:3302/0 2023-09-30T01:34:55.262 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:55.243+0000 7fe867fff700 1 -- 172.21.15.88:0/1610096351 --> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe86804bb20 con 0x7fe868097a40 2023-09-30T01:34:55.262 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:55.247+0000 7fe85ffff700 1 --2- 172.21.15.88:0/1610096351 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7fe8681311c0 0x7fe868132c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-30T01:34:55.262 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:55.247+0000 7fe867fff700 1 --2- 172.21.15.88:0/1610096351 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7fe868097a40 0x7fe868132700 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fe8500099e0 tx=0x7fe850009250 comp rx=0 tx=0).ready entity=mon.4 client_cookie=13145196699fee3a server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:34:55.262 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:55.247+0000 7fe8677fe700 1 -- 172.21.15.88:0/1610096351 <== mon.4 v2:172.21.15.88:3302/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7fe85000bce0 con 0x7fe868097a40 2023-09-30T01:34:55.262 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:55.247+0000 7fe8677fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:34:55.262 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:55.247+0000 7fe8677fe700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T01:34:55.262 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:55.247+0000 7fe8677fe700 10 monclient: dump: 2023-09-30T01:34:55.262 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:34:55.263 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:34:55.263 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:34:55.263 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:34:55.263 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:34:55.263 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:34:55.263 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:34:55.263 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:34:55.263 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:34:55.263 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:34:55.263 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:34:55.263 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:34:55.263 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:34:55.263 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:34:55.264 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:34:55.264 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:34:55.264 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:55.247+0000 7fe8677fe700 10 monclient: _finish_auth 0 2023-09-30T01:34:55.264 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:55.247+0000 7fe8677fe700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:34:55.264 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:55.247+0000 7fe86d782700 5 monclient: authenticate success, global_id 4549 2023-09-30T01:34:55.264 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:55.247+0000 7fe86d782700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:34:55.264 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:55.247+0000 7fe8677fe700 1 -- 172.21.15.88:0/1610096351 <== mon.4 v2:172.21.15.88:3302/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe85000be60 con 0x7fe868097a40 2023-09-30T01:34:55.264 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:55.247+0000 7fe8677fe700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:34:55.264 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:55.247+0000 7fe86d782700 10 monclient: get_monmap_and_config success 2023-09-30T01:34:55.264 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:55.247+0000 7fe8677fe700 1 -- 172.21.15.88:0/1610096351 <== mon.4 v2:172.21.15.88:3302/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7fe8500118a0 con 0x7fe868097a40 2023-09-30T01:34:55.264 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:55.247+0000 7fe86d782700 4 set_mon_vals no callback set 2023-09-30T01:34:55.264 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:55.247+0000 7fe8677fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:34:55.265 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:55.247+0000 7fe8677fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T01:34:55.265 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:55.247+0000 7fe8677fe700 10 monclient: dump: 2023-09-30T01:34:55.265 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:34:55.265 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:34:55.265 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:34:55.265 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:34:55.265 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:34:55.265 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:34:55.265 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:34:55.265 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:34:55.265 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:34:55.265 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:34:55.265 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:34:55.266 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:34:55.266 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:34:55.266 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:34:55.266 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:34:55.266 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:34:55.266 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:55.247+0000 7fe86d782700 10 monclient: shutdown 2023-09-30T01:34:55.266 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:55.247+0000 7fe86d782700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:34:55.266 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:55.247+0000 7fe86d782700 1 -- 172.21.15.88:0/1610096351 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7fe868097a40 msgr2=0x7fe868132700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:34:55.266 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:55.247+0000 7fe86d782700 1 --2- 172.21.15.88:0/1610096351 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7fe868097a40 0x7fe868132700 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fe8500099e0 tx=0x7fe850009250 comp rx=0 tx=0).stop 2023-09-30T01:34:55.266 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:55.247+0000 7fe86d782700 1 -- 172.21.15.88:0/1610096351 shutdown_connections 2023-09-30T01:34:55.266 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:55.247+0000 7fe86d782700 1 --2- 172.21.15.88:0/1610096351 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7fe8681332e0 0x7fe868133710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:34:55.266 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:55.247+0000 7fe86d782700 1 --2- 172.21.15.88:0/1610096351 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7fe8681311c0 0x7fe868132c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:34:55.266 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:55.247+0000 7fe86d782700 1 --2- 172.21.15.88:0/1610096351 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7fe868097a40 0x7fe868132700 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:34:55.267 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:55.247+0000 7fe86d782700 1 -- 172.21.15.88:0/1610096351 >> 172.21.15.88:0/1610096351 conn(0x7fe86806e5e0 msgr2=0x7fe86806e9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:34:55.267 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:55.247+0000 7fe86d782700 1 -- 172.21.15.88:0/1610096351 shutdown_connections 2023-09-30T01:34:55.267 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:55.247+0000 7fe86d782700 1 -- 172.21.15.88:0/1610096351 wait complete. 2023-09-30T01:34:55.267 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:55.247+0000 7fe86d782700 10 monclient: build_initial_monmap 2023-09-30T01:34:55.267 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:55.247+0000 7fe86d782700 10 monclient: monmap: 2023-09-30T01:34:55.267 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2023-09-30T01:34:55.267 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:34:55.267 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 0.000000 2023-09-30T01:34:55.267 INFO:teuthology.orchestra.run.smithi088.stderr:created 0.000000 2023-09-30T01:34:55.267 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2023-09-30T01:34:55.267 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 1 2023-09-30T01:34:55.267 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.noname-a 2023-09-30T01:34:55.268 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.noname-f 2023-09-30T01:34:55.268 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.noname-b 2023-09-30T01:34:55.268 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.noname-g 2023-09-30T01:34:55.268 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.noname-c 2023-09-30T01:34:55.268 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.noname-h 2023-09-30T01:34:55.268 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.noname-d 2023-09-30T01:34:55.268 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.noname-i 2023-09-30T01:34:55.268 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.noname-e 2023-09-30T01:34:55.268 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:34:55.268 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:55.247+0000 7fe86d782700 1 Processor -- start 2023-09-30T01:34:55.268 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:55.247+0000 7fe86d782700 1 -- start start 2023-09-30T01:34:55.268 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:55.247+0000 7fe86d782700 10 monclient: init 2023-09-30T01:34:55.268 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:55.247+0000 7fe86d782700 10 monclient: _reopen_session rank -1 2023-09-30T01:34:55.269 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:55.247+0000 7fe86d782700 10 monclient: _add_conns ranks=[7,4,6,2,0,1,3,5,8] 2023-09-30T01:34:55.269 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:55.247+0000 7fe86d782700 1 --2- >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7fe868097a40 0x7fe868142790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:34:55.269 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:55.247+0000 7fe86d782700 10 monclient(hunting): picked mon.noname-i con 0x7fe868097a40 addr [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] 2023-09-30T01:34:55.269 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:55.247+0000 7fe86d782700 1 --2- >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7fe8681311c0 0x7fe868142cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:34:55.269 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:55.247+0000 7fe86d782700 10 monclient(hunting): picked mon.noname-c con 0x7fe8681311c0 addr [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] 2023-09-30T01:34:55.269 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:55.247+0000 7fe86d782700 1 --2- >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7fe8681332e0 0x7fe86811cd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:34:55.269 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:55.247+0000 7fe86d782700 10 monclient(hunting): picked mon.noname-d con 0x7fe8681332e0 addr [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] 2023-09-30T01:34:55.269 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:55.247+0000 7fe86d782700 10 monclient(hunting): start opening mon connection 2023-09-30T01:34:55.269 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:55.247+0000 7fe86d782700 1 -- --> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] -- mon_getmap magic: 0 v1 -- 0x7fe8680996b0 con 0x7fe8681311c0 2023-09-30T01:34:55.269 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:55.247+0000 7fe86d782700 10 monclient(hunting): start opening mon connection 2023-09-30T01:34:55.269 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:55.247+0000 7fe86d782700 1 -- --> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] -- mon_getmap magic: 0 v1 -- 0x7fe8681202d0 con 0x7fe8681332e0 2023-09-30T01:34:55.270 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:55.247+0000 7fe86d782700 10 monclient(hunting): start opening mon connection 2023-09-30T01:34:55.270 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:55.247+0000 7fe86d782700 1 -- --> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] -- mon_getmap magic: 0 v1 -- 0x7fe86811f100 con 0x7fe868097a40 2023-09-30T01:34:55.270 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:55.247+0000 7fe86d782700 10 monclient(hunting): _renew_subs 2023-09-30T01:34:55.270 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:55.247+0000 7fe86d782700 10 monclient(hunting): authenticate will time out at 967.286920s 2023-09-30T01:34:55.270 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:55.247+0000 7fe85ffff700 1 --2- >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7fe8681311c0 0x7fe868142cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:34:55.270 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:55.247+0000 7fe867fff700 1 --2- >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7fe868097a40 0x7fe868142790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:34:55.270 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:55.247+0000 7fe85ffff700 1 --2- >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7fe8681311c0 0x7fe868142cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3302/0 says I am v2:172.21.15.88:39166/0 (socket says 172.21.15.88:39166) 2023-09-30T01:34:55.270 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:55.247+0000 7fe85ffff700 1 -- 172.21.15.88:0/1708508663 learned_addr learned my addr 172.21.15.88:0/1708508663 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-09-30T01:34:55.270 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:55.247+0000 7fe85ffff700 10 monclient(hunting): get_auth_request con 0x7fe8681311c0 auth_method 0 2023-09-30T01:34:55.271 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:55.247+0000 7fe85ffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:34:55.271 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:55.247+0000 7fe85ffff700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:34:55.271 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:55.247+0000 7fe85ffff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:34:55.271 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:55.247+0000 7fe86cf81700 1 --2- 172.21.15.88:0/1708508663 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7fe8681332e0 0x7fe86811cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:34:55.271 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:55.247+0000 7fe867fff700 10 monclient(hunting): get_auth_request con 0x7fe868097a40 auth_method 0 2023-09-30T01:34:55.271 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:55.247+0000 7fe867fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:34:55.271 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:55.247+0000 7fe867fff700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:34:55.271 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:55.247+0000 7fe867fff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:34:55.271 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:55.247+0000 7fe86cf81700 10 monclient(hunting): get_auth_request con 0x7fe8681332e0 auth_method 0 2023-09-30T01:34:55.271 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:55.247+0000 7fe86cf81700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:34:55.271 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:55.247+0000 7fe86cf81700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:34:55.271 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:55.247+0000 7fe86cf81700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:34:55.272 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:55.247+0000 7fe867fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:34:55.272 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:55.247+0000 7fe867fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:34:55.272 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:55.247+0000 7fe867fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:34:55.272 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:55.247+0000 7fe85ffff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:34:55.272 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:55.247+0000 7fe85ffff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:34:55.272 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:55.247+0000 7fe85ffff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:34:55.272 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:55.247+0000 7fe86cf81700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:34:55.272 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:55.247+0000 7fe86cf81700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:34:55.272 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:55.247+0000 7fe86cf81700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:34:55.272 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:55.247+0000 7fe85ffff700 10 monclient(hunting): handle_auth_done global_id 4558 payload 824 2023-09-30T01:34:55.272 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:55.247+0000 7fe85ffff700 1 -- 172.21.15.88:0/1708508663 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7fe868097a40 msgr2=0x7fe868142790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:34:55.272 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:55.247+0000 7fe85ffff700 1 --2- 172.21.15.88:0/1708508663 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7fe868097a40 0x7fe868142790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:34:55.272 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:55.247+0000 7fe85ffff700 1 -- 172.21.15.88:0/1708508663 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7fe8681332e0 msgr2=0x7fe86811cd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:34:55.273 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:55.247+0000 7fe85ffff700 1 --2- 172.21.15.88:0/1708508663 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7fe8681332e0 0x7fe86811cd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:34:55.273 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:55.251+0000 7fe85ffff700 10 monclient: _finish_hunting 0 2023-09-30T01:34:55.273 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:55.251+0000 7fe85ffff700 1 monclient: found mon.noname-c 2023-09-30T01:34:55.273 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:55.251+0000 7fe85ffff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:34:55.273 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:55.251+0000 7fe85ffff700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.88:3302/0 2023-09-30T01:34:55.273 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:55.251+0000 7fe85ffff700 1 -- 172.21.15.88:0/1708508663 --> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8500094f0 con 0x7fe8681311c0 2023-09-30T01:34:55.273 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:55.251+0000 7fe867fff700 1 --2- 172.21.15.88:0/1708508663 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7fe868097a40 0x7fe868142790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:34:55.273 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:55.251+0000 7fe86cf81700 1 --2- 172.21.15.88:0/1708508663 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7fe8681332e0 0x7fe86811cd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:34:55.273 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:55.251+0000 7fe85ffff700 1 --2- 172.21.15.88:0/1708508663 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7fe8681311c0 0x7fe868142cd0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fe85800e8d0 tx=0x7fe858008040 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:34:55.273 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:55.251+0000 7fe864ff9700 1 -- 172.21.15.88:0/1708508663 <== mon.4 v2:172.21.15.88:3302/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7fe8580093b0 con 0x7fe8681311c0 2023-09-30T01:34:55.273 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:55.251+0000 7fe864ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:34:55.273 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:55.251+0000 7fe864ff9700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T01:34:55.273 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:55.251+0000 7fe864ff9700 10 monclient: dump: 2023-09-30T01:34:55.274 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:34:55.274 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:34:55.274 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:34:55.274 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:34:55.274 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:34:55.274 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:34:55.274 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:34:55.274 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:34:55.274 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:34:55.274 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:34:55.274 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:34:55.274 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:34:55.275 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:34:55.275 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:34:55.275 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:34:55.275 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:34:55.275 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:55.251+0000 7fe864ff9700 10 monclient: _finish_auth 0 2023-09-30T01:34:55.275 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:55.251+0000 7fe864ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:34:55.275 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:55.251+0000 7fe864ff9700 1 -- 172.21.15.88:0/1708508663 <== mon.4 v2:172.21.15.88:3302/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe858013890 con 0x7fe8681311c0 2023-09-30T01:34:55.275 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:55.251+0000 7fe86d782700 5 monclient: authenticate success, global_id 4558 2023-09-30T01:34:55.275 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:55.251+0000 7fe86d782700 10 monclient: _renew_subs 2023-09-30T01:34:55.275 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:55.251+0000 7fe86d782700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.88:3302/0 2023-09-30T01:34:55.275 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:55.251+0000 7fe86d782700 1 -- 172.21.15.88:0/1708508663 --> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe86811ec80 con 0x7fe8681311c0 2023-09-30T01:34:55.275 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:55.251+0000 7fe864ff9700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:34:55.276 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:55.251+0000 7fe864ff9700 1 -- 172.21.15.88:0/1708508663 <== mon.4 v2:172.21.15.88:3302/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7fe858013e80 con 0x7fe8681311c0 2023-09-30T01:34:55.276 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:55.251+0000 7fe864ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:34:55.276 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:55.251+0000 7fe864ff9700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T01:34:55.276 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:55.251+0000 7fe864ff9700 10 monclient: dump: 2023-09-30T01:34:55.276 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:34:55.276 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:34:55.276 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:34:55.276 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:34:55.276 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:34:55.276 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:34:55.276 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:34:55.276 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:34:55.276 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:34:55.277 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:34:55.277 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:34:55.277 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:34:55.277 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:34:55.277 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:34:55.277 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:34:55.277 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:34:55.277 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:55.251+0000 7fe866ffd700 4 set_mon_vals no callback set 2023-09-30T01:34:55.277 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:55.251+0000 7fe86d782700 10 monclient: _renew_subs 2023-09-30T01:34:55.277 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:55.251+0000 7fe86d782700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.88:3302/0 2023-09-30T01:34:55.277 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:55.251+0000 7fe86d782700 1 -- 172.21.15.88:0/1708508663 --> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe86811e4c0 con 0x7fe8681311c0 2023-09-30T01:34:55.277 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:55.251+0000 7fe864ff9700 1 -- 172.21.15.88:0/1708508663 <== mon.4 v2:172.21.15.88:3302/0 4 ==== mgrmap(e 5) v1 ==== 40144+0+0 (secure 0 0 0) 0x7fe858013a00 con 0x7fe8681311c0 2023-09-30T01:34:55.277 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:55.251+0000 7fe86d782700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:34:55.278 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:55.251+0000 7fe86d782700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:34:55.278 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:55.251+0000 7fe86d782700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.88:3302/0 2023-09-30T01:34:55.278 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:55.251+0000 7fe86d782700 1 -- 172.21.15.88:0/1708508663 --> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe868120440 con 0x7fe8681311c0 2023-09-30T01:34:55.278 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:55.251+0000 7fe864ff9700 1 --2- 172.21.15.88:0/1708508663 >> v2:172.21.15.107:6812/29926 conn(0x7fe848033360 0x7fe848035810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:34:55.278 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:55.251+0000 7fe864ff9700 1 -- 172.21.15.88:0/1708508663 <== mon.4 v2:172.21.15.88:3302/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 3502+0+0 (secure 0 0 0) 0x7fe8580430e0 con 0x7fe8681311c0 2023-09-30T01:34:55.278 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:55.251+0000 7fe867fff700 1 --2- 172.21.15.88:0/1708508663 >> v2:172.21.15.107:6812/29926 conn(0x7fe848033360 0x7fe848035810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:34:55.278 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:55.251+0000 7fe867fff700 10 monclient: get_auth_request con 0x7fe848033360 auth_method 0 2023-09-30T01:34:55.278 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:55.251+0000 7fe867fff700 1 --2- 172.21.15.88:0/1708508663 >> v2:172.21.15.107:6812/29926 conn(0x7fe848033360 0x7fe848035810 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fe85000bfd0 tx=0x7fe850017000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:34:55.279 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:55.259+0000 7fe864ff9700 1 -- 172.21.15.88:0/1708508663 <== mon.4 v2:172.21.15.88:3302/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fe8580228f0 con 0x7fe8681311c0 2023-09-30T01:34:55.279 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:55.259+0000 7fe864ff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:34:55.279 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:55.259+0000 7fe864ff9700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:34:55.461 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:55.459+0000 7fe86d782700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T01:34:55.461 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:55.459+0000 7fe86d782700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:34:55.461 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:55.459+0000 7fe86d782700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.88:3302/0 2023-09-30T01:34:55.461 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:55.459+0000 7fe86d782700 1 -- 172.21.15.88:0/1708508663 --> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fe868071960 con 0x7fe8681311c0 2023-09-30T01:34:55.462 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:55.459+0000 7fe864ff9700 1 -- 172.21.15.88:0/1708508663 <== mon.4 v2:172.21.15.88:3302/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7fe858020070 con 0x7fe8681311c0 2023-09-30T01:34:55.462 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:55.459+0000 7fe864ff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:34:55.462 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:55.459+0000 7fe864ff9700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:34:55.462 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-09-30T01:34:55.462 INFO:teuthology.orchestra.run.smithi088.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:34:55.464 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:55.463+0000 7fe86d782700 1 -- 172.21.15.88:0/1708508663 >> v2:172.21.15.107:6812/29926 conn(0x7fe848033360 msgr2=0x7fe848035810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:34:55.464 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:55.463+0000 7fe86d782700 1 --2- 172.21.15.88:0/1708508663 >> v2:172.21.15.107:6812/29926 conn(0x7fe848033360 0x7fe848035810 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fe85000bfd0 tx=0x7fe850017000 comp rx=0 tx=0).stop 2023-09-30T01:34:55.464 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:55.463+0000 7fe86d782700 10 monclient: shutdown 2023-09-30T01:34:55.465 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:55.463+0000 7fe86d782700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:34:55.465 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:55.463+0000 7fe86d782700 1 -- 172.21.15.88:0/1708508663 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7fe8681311c0 msgr2=0x7fe868142cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:34:55.465 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:55.463+0000 7fe86d782700 1 --2- 172.21.15.88:0/1708508663 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7fe8681311c0 0x7fe868142cd0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fe85800e8d0 tx=0x7fe858008040 comp rx=0 tx=0).stop 2023-09-30T01:34:55.465 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:55.463+0000 7fe86d782700 1 -- 172.21.15.88:0/1708508663 shutdown_connections 2023-09-30T01:34:55.465 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:55.463+0000 7fe86d782700 1 --2- 172.21.15.88:0/1708508663 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7fe8681332e0 0x7fe86811cd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:34:55.465 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:55.463+0000 7fe86d782700 1 --2- 172.21.15.88:0/1708508663 >> v2:172.21.15.107:6812/29926 conn(0x7fe848033360 0x7fe848035810 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:34:55.465 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:55.463+0000 7fe86d782700 1 --2- 172.21.15.88:0/1708508663 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7fe8681311c0 0x7fe868142cd0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:34:55.465 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:55.463+0000 7fe86d782700 1 --2- 172.21.15.88:0/1708508663 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7fe868097a40 0x7fe868142790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:34:55.465 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:55.463+0000 7fe86d782700 1 -- 172.21.15.88:0/1708508663 >> 172.21.15.88:0/1708508663 conn(0x7fe86806e5e0 msgr2=0x7fe8680a4940 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:34:55.465 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:55.463+0000 7fe86d782700 1 -- 172.21.15.88:0/1708508663 shutdown_connections 2023-09-30T01:34:55.465 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:55.463+0000 7fe86d782700 1 -- 172.21.15.88:0/1708508663 wait complete. 2023-09-30T01:34:58.477 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:34:58.587 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:58.583+0000 7f6e91dbf700 10 monclient: get_monmap_and_config 2023-09-30T01:34:58.587 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:58.583+0000 7f6e91dbf700 10 monclient: build_initial_monmap 2023-09-30T01:34:58.587 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:58.583+0000 7f6e91dbf700 10 monclient: monmap: 2023-09-30T01:34:58.587 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2023-09-30T01:34:58.587 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:34:58.587 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:34:58.587303+0000 2023-09-30T01:34:58.587 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:34:58.587303+0000 2023-09-30T01:34:58.587 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2023-09-30T01:34:58.588 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:34:58.588 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a-legacy 2023-09-30T01:34:58.588 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.107:6789/0 mon.noname-f-legacy 2023-09-30T01:34:58.588 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.noname-b 2023-09-30T01:34:58.588 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.noname-g 2023-09-30T01:34:58.588 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.noname-c 2023-09-30T01:34:58.588 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.noname-h 2023-09-30T01:34:58.588 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.noname-d 2023-09-30T01:34:58.588 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.noname-i 2023-09-30T01:34:58.588 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.noname-e 2023-09-30T01:34:58.589 INFO:teuthology.orchestra.run.smithi088.stderr:9: v2:172.21.15.88:3300/0 mon.noname-a 2023-09-30T01:34:58.589 INFO:teuthology.orchestra.run.smithi088.stderr:10: v2:172.21.15.107:3300/0 mon.noname-f 2023-09-30T01:34:58.589 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:34:58.593 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:58.587+0000 7f6e91dbf700 1 Processor -- start 2023-09-30T01:34:58.593 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:58.587+0000 7f6e91dbf700 1 -- start start 2023-09-30T01:34:58.593 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:58.587+0000 7f6e91dbf700 10 monclient: init 2023-09-30T01:34:58.593 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:58.587+0000 7f6e91dbf700 10 monclient: _reopen_session rank -1 2023-09-30T01:34:58.593 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:58.587+0000 7f6e91dbf700 10 monclient: _add_conns ranks=[9,2,8,4,3,10,6,0,7,1,5] 2023-09-30T01:34:58.593 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:58.587+0000 7f6e91dbf700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f6e8c0966a0 0x7f6e8c096a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:34:58.593 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:58.587+0000 7f6e91dbf700 10 monclient(hunting): picked mon.noname-a con 0x7f6e8c0966a0 addr v2:172.21.15.88:3300/0 2023-09-30T01:34:58.594 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:58.587+0000 7f6e91dbf700 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6e8c097cd0 0x7f6e8c0980a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:34:58.594 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:58.587+0000 7f6e91dbf700 10 monclient(hunting): picked mon.noname-b con 0x7f6e8c097cd0 addr [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] 2023-09-30T01:34:58.594 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:58.587+0000 7f6e91dbf700 1 --2- >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f6e8c09ed80 0x7f6e8c09f1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:34:58.594 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:58.587+0000 7f6e91dbf700 10 monclient(hunting): picked mon.noname-e con 0x7f6e8c09ed80 addr [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] 2023-09-30T01:34:58.594 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:58.587+0000 7f6e91dbf700 10 monclient(hunting): start opening mon connection 2023-09-30T01:34:58.594 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:58.587+0000 7f6e91dbf700 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6e8c097920 con 0x7f6e8c0966a0 2023-09-30T01:34:58.594 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:58.587+0000 7f6e91dbf700 10 monclient(hunting): start opening mon connection 2023-09-30T01:34:58.594 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:58.587+0000 7f6e91dbf700 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e8c09a340 con 0x7f6e8c097cd0 2023-09-30T01:34:58.594 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:58.587+0000 7f6e91dbf700 10 monclient(hunting): start opening mon connection 2023-09-30T01:34:58.594 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:58.587+0000 7f6e91dbf700 1 -- --> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] -- mon_getmap magic: 0 v1 -- 0x7f6e8c0975d0 con 0x7f6e8c09ed80 2023-09-30T01:34:58.595 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:58.587+0000 7f6e91dbf700 10 monclient(hunting): _renew_subs 2023-09-30T01:34:58.595 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:58.587+0000 7f6e91dbf700 10 monclient(hunting): authenticate will time out at 970.626068s 2023-09-30T01:34:58.595 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:58.587+0000 7f6e83fff700 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6e8c097cd0 0x7f6e8c0980a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:34:58.595 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:58.587+0000 7f6e90dbd700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f6e8c0966a0 0x7f6e8c096a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:34:58.595 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:58.587+0000 7f6e83fff700 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6e8c097cd0 0x7f6e8c0980a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3301/0 says I am v2:172.21.15.88:50602/0 (socket says 172.21.15.88:50602) 2023-09-30T01:34:58.595 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:58.587+0000 7f6e83fff700 1 -- 172.21.15.88:0/3804356589 learned_addr learned my addr 172.21.15.88:0/3804356589 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-09-30T01:34:58.595 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:58.587+0000 7f6e83fff700 10 monclient(hunting): get_auth_request con 0x7f6e8c097cd0 auth_method 0 2023-09-30T01:34:58.595 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:58.587+0000 7f6e83fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:34:58.595 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:58.587+0000 7f6e83fff700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:34:58.595 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:58.587+0000 7f6e83fff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:34:58.596 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:58.587+0000 7f6e90dbd700 10 monclient(hunting): get_auth_request con 0x7f6e8c0966a0 auth_method 0 2023-09-30T01:34:58.596 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:58.587+0000 7f6e90dbd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:34:58.596 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:58.587+0000 7f6e90dbd700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:34:58.596 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:58.587+0000 7f6e90dbd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:34:58.596 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:58.587+0000 7f6e83fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:34:58.596 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:58.587+0000 7f6e83fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:34:58.596 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:58.587+0000 7f6e915be700 1 --2- 172.21.15.88:0/3804356589 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f6e8c09ed80 0x7f6e8c09f1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:34:58.596 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:58.587+0000 7f6e83fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:34:58.596 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:58.587+0000 7f6e915be700 10 monclient(hunting): get_auth_request con 0x7f6e8c09ed80 auth_method 0 2023-09-30T01:34:58.596 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:58.587+0000 7f6e915be700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:34:58.597 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:58.587+0000 7f6e915be700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:34:58.597 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:58.587+0000 7f6e915be700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:34:58.597 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:58.587+0000 7f6e90dbd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:34:58.597 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:58.587+0000 7f6e90dbd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:34:58.597 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:58.587+0000 7f6e90dbd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:34:58.597 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:58.587+0000 7f6e915be700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:34:58.597 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:58.587+0000 7f6e915be700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:34:58.597 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:58.587+0000 7f6e915be700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:34:58.597 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:58.587+0000 7f6e90dbd700 10 monclient(hunting): handle_auth_done global_id 4716 payload 274 2023-09-30T01:34:58.597 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:58.587+0000 7f6e90dbd700 1 -- 172.21.15.88:0/3804356589 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f6e8c09ed80 msgr2=0x7f6e8c09f1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:34:58.597 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:58.587+0000 7f6e90dbd700 1 --2- 172.21.15.88:0/3804356589 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f6e8c09ed80 0x7f6e8c09f1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:34:58.598 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:58.587+0000 7f6e90dbd700 1 -- 172.21.15.88:0/3804356589 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6e8c097cd0 msgr2=0x7f6e8c0980a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:34:58.598 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:58.587+0000 7f6e90dbd700 1 --2- 172.21.15.88:0/3804356589 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6e8c097cd0 0x7f6e8c0980a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:34:58.598 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:58.587+0000 7f6e90dbd700 10 monclient: _finish_hunting 0 2023-09-30T01:34:58.598 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:58.587+0000 7f6e90dbd700 1 monclient: found mon.noname-a 2023-09-30T01:34:58.598 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:58.587+0000 7f6e90dbd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:34:58.598 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:58.587+0000 7f6e90dbd700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.88:3300/0 2023-09-30T01:34:58.598 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:58.587+0000 7f6e90dbd700 1 -- 172.21.15.88:0/3804356589 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e8c04bb20 con 0x7f6e8c0966a0 2023-09-30T01:34:58.598 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:58.587+0000 7f6e90dbd700 1 --2- 172.21.15.88:0/3804356589 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6e8c0966a0 0x7f6e8c096a70 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f6e88009960 tx=0x7f6e880091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef32f00635903b3a server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:34:58.598 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:58.587+0000 7f6e837fe700 1 -- 172.21.15.88:0/3804356589 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f6e88011ab0 con 0x7f6e8c0966a0 2023-09-30T01:34:58.598 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:58.587+0000 7f6e837fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:34:58.599 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:58.587+0000 7f6e837fe700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T01:34:58.599 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:58.587+0000 7f6e837fe700 10 monclient: dump: 2023-09-30T01:34:58.599 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:34:58.599 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:34:58.599 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:34:58.599 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:34:58.599 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:34:58.599 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:34:58.599 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:34:58.599 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:34:58.600 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:34:58.600 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:34:58.600 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:34:58.600 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:34:58.600 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:34:58.600 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:34:58.600 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:34:58.600 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:34:58.600 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:58.587+0000 7f6e837fe700 10 monclient: _finish_auth 0 2023-09-30T01:34:58.600 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:58.587+0000 7f6e837fe700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:34:58.600 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:58.587+0000 7f6e91dbf700 5 monclient: authenticate success, global_id 4716 2023-09-30T01:34:58.601 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:58.587+0000 7f6e837fe700 1 -- 172.21.15.88:0/3804356589 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e88011c30 con 0x7f6e8c0966a0 2023-09-30T01:34:58.601 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:58.587+0000 7f6e91dbf700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:34:58.601 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:58.587+0000 7f6e837fe700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:34:58.601 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:58.587+0000 7f6e91dbf700 10 monclient: get_monmap_and_config success 2023-09-30T01:34:58.601 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:58.587+0000 7f6e91dbf700 4 set_mon_vals no callback set 2023-09-30T01:34:58.602 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:58.587+0000 7f6e837fe700 1 -- 172.21.15.88:0/3804356589 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f6e88011da0 con 0x7f6e8c0966a0 2023-09-30T01:34:58.602 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:58.591+0000 7f6e837fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:34:58.602 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:58.591+0000 7f6e837fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:34:58.602 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:58.591+0000 7f6e837fe700 10 monclient: dump: 2023-09-30T01:34:58.602 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:34:58.602 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:34:58.602 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:34:58.603 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:34:58.603 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:34:58.603 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:34:58.603 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:34:58.603 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:34:58.603 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:34:58.603 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:34:58.603 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:34:58.604 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:34:58.604 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:34:58.604 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:34:58.604 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:34:58.604 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:34:58.604 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:58.591+0000 7f6e91dbf700 10 monclient: shutdown 2023-09-30T01:34:58.604 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:58.591+0000 7f6e91dbf700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:34:58.604 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:58.591+0000 7f6e91dbf700 1 -- 172.21.15.88:0/3804356589 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6e8c0966a0 msgr2=0x7f6e8c096a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:34:58.604 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:58.591+0000 7f6e91dbf700 1 --2- 172.21.15.88:0/3804356589 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6e8c0966a0 0x7f6e8c096a70 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f6e88009960 tx=0x7f6e880091a0 comp rx=0 tx=0).stop 2023-09-30T01:34:58.604 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:58.591+0000 7f6e91dbf700 1 -- 172.21.15.88:0/3804356589 shutdown_connections 2023-09-30T01:34:58.604 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:58.591+0000 7f6e91dbf700 1 --2- 172.21.15.88:0/3804356589 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f6e8c09ed80 0x7f6e8c09f1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:34:58.605 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:58.591+0000 7f6e91dbf700 1 --2- 172.21.15.88:0/3804356589 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6e8c097cd0 0x7f6e8c0980a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:34:58.605 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:58.591+0000 7f6e91dbf700 1 --2- 172.21.15.88:0/3804356589 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6e8c0966a0 0x7f6e8c096a70 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:34:58.605 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:58.591+0000 7f6e91dbf700 1 -- 172.21.15.88:0/3804356589 >> 172.21.15.88:0/3804356589 conn(0x7f6e8c0a0910 msgr2=0x7f6e8c0a33c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:34:58.605 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:58.591+0000 7f6e91dbf700 1 -- 172.21.15.88:0/3804356589 shutdown_connections 2023-09-30T01:34:58.605 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:58.591+0000 7f6e91dbf700 1 -- 172.21.15.88:0/3804356589 wait complete. 2023-09-30T01:34:58.610 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:58.591+0000 7f6e91dbf700 10 monclient: build_initial_monmap 2023-09-30T01:34:58.610 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:58.591+0000 7f6e91dbf700 10 monclient: monmap: 2023-09-30T01:34:58.610 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2023-09-30T01:34:58.610 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:34:58.610 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 0.000000 2023-09-30T01:34:58.610 INFO:teuthology.orchestra.run.smithi088.stderr:created 0.000000 2023-09-30T01:34:58.611 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2023-09-30T01:34:58.611 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 1 2023-09-30T01:34:58.611 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.noname-a 2023-09-30T01:34:58.611 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.noname-f 2023-09-30T01:34:58.611 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.noname-b 2023-09-30T01:34:58.611 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.noname-g 2023-09-30T01:34:58.611 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.noname-c 2023-09-30T01:34:58.611 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.noname-h 2023-09-30T01:34:58.611 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.noname-d 2023-09-30T01:34:58.611 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.noname-i 2023-09-30T01:34:58.612 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.noname-e 2023-09-30T01:34:58.612 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:34:58.612 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:58.591+0000 7f6e91dbf700 1 Processor -- start 2023-09-30T01:34:58.612 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:58.591+0000 7f6e91dbf700 1 -- start start 2023-09-30T01:34:58.612 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:58.591+0000 7f6e91dbf700 10 monclient: init 2023-09-30T01:34:58.612 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:58.595+0000 7f6e91dbf700 10 monclient: _reopen_session rank -1 2023-09-30T01:34:58.612 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:58.595+0000 7f6e91dbf700 10 monclient: _add_conns ranks=[0,8,3,7,5,4,6,2,1] 2023-09-30T01:34:58.612 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:58.595+0000 7f6e91dbf700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6e8c0966a0 0x7f6e8c142690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:34:58.612 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:58.595+0000 7f6e91dbf700 10 monclient(hunting): picked mon.noname-a con 0x7f6e8c0966a0 addr [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] 2023-09-30T01:34:58.612 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:58.595+0000 7f6e91dbf700 1 --2- >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f6e8c097cd0 0x7f6e8c142bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:34:58.612 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:58.595+0000 7f6e91dbf700 10 monclient(hunting): picked mon.noname-e con 0x7f6e8c097cd0 addr [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] 2023-09-30T01:34:58.613 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:58.595+0000 7f6e91dbf700 1 --2- >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7f6e8c09ed80 0x7f6e8c143110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:34:58.613 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:58.595+0000 7f6e91dbf700 10 monclient(hunting): picked mon.noname-g con 0x7f6e8c09ed80 addr [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] 2023-09-30T01:34:58.613 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:58.595+0000 7f6e91dbf700 10 monclient(hunting): start opening mon connection 2023-09-30T01:34:58.613 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:58.595+0000 7f6e91dbf700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e8c091060 con 0x7f6e8c0966a0 2023-09-30T01:34:58.613 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:58.595+0000 7f6e91dbf700 10 monclient(hunting): start opening mon connection 2023-09-30T01:34:58.613 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:58.595+0000 7f6e91dbf700 1 -- --> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e8c0996c0 con 0x7f6e8c09ed80 2023-09-30T01:34:58.613 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:58.595+0000 7f6e91dbf700 10 monclient(hunting): start opening mon connection 2023-09-30T01:34:58.613 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:58.595+0000 7f6e91dbf700 1 -- --> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] -- mon_getmap magic: 0 v1 -- 0x7f6e8c063630 con 0x7f6e8c097cd0 2023-09-30T01:34:58.613 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:58.595+0000 7f6e91dbf700 10 monclient(hunting): _renew_subs 2023-09-30T01:34:58.613 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:58.595+0000 7f6e91dbf700 10 monclient(hunting): authenticate will time out at 970.632431s 2023-09-30T01:34:58.614 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:58.595+0000 7f6e90dbd700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6e8c0966a0 0x7f6e8c142690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:34:58.614 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:58.595+0000 7f6e83fff700 1 --2- >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f6e8c097cd0 0x7f6e8c142bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:34:58.614 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:58.595+0000 7f6e83fff700 1 --2- >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f6e8c097cd0 0x7f6e8c142bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3304/0 says I am v2:172.21.15.88:41174/0 (socket says 172.21.15.88:41174) 2023-09-30T01:34:58.614 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:58.595+0000 7f6e83fff700 1 -- 172.21.15.88:0/2477597654 learned_addr learned my addr 172.21.15.88:0/2477597654 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-09-30T01:34:58.614 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:58.595+0000 7f6e83fff700 10 monclient(hunting): get_auth_request con 0x7f6e8c097cd0 auth_method 0 2023-09-30T01:34:58.614 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:58.595+0000 7f6e83fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:34:58.614 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:58.595+0000 7f6e83fff700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:34:58.614 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:58.595+0000 7f6e83fff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:34:58.614 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:58.595+0000 7f6e90dbd700 10 monclient(hunting): get_auth_request con 0x7f6e8c0966a0 auth_method 0 2023-09-30T01:34:58.614 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:58.595+0000 7f6e90dbd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:34:58.615 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:58.595+0000 7f6e90dbd700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:34:58.615 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:58.595+0000 7f6e90dbd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:34:58.615 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:58.595+0000 7f6e83fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:34:58.615 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:58.595+0000 7f6e83fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:34:58.615 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:58.595+0000 7f6e83fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:34:58.615 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:58.595+0000 7f6e90dbd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:34:58.615 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:58.595+0000 7f6e90dbd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:34:58.615 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:58.595+0000 7f6e915be700 1 --2- 172.21.15.88:0/2477597654 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7f6e8c09ed80 0x7f6e8c143110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:34:58.615 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:58.595+0000 7f6e90dbd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:34:58.615 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:58.595+0000 7f6e915be700 10 monclient(hunting): get_auth_request con 0x7f6e8c09ed80 auth_method 0 2023-09-30T01:34:58.616 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:58.595+0000 7f6e915be700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:34:58.616 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:58.595+0000 7f6e915be700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:34:58.616 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:58.595+0000 7f6e915be700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:34:58.616 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:58.595+0000 7f6e83fff700 10 monclient(hunting): handle_auth_done global_id 4598 payload 824 2023-09-30T01:34:58.616 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:58.595+0000 7f6e83fff700 1 -- 172.21.15.88:0/2477597654 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7f6e8c09ed80 msgr2=0x7f6e8c143110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:34:58.616 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:58.595+0000 7f6e83fff700 1 --2- 172.21.15.88:0/2477597654 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7f6e8c09ed80 0x7f6e8c143110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:34:58.616 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:58.595+0000 7f6e83fff700 1 -- 172.21.15.88:0/2477597654 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6e8c0966a0 msgr2=0x7f6e8c142690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:34:58.616 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:58.595+0000 7f6e83fff700 1 --2- 172.21.15.88:0/2477597654 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6e8c0966a0 0x7f6e8c142690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:34:58.616 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:58.595+0000 7f6e83fff700 10 monclient: _finish_hunting 0 2023-09-30T01:34:58.616 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:58.595+0000 7f6e83fff700 1 monclient: found mon.noname-e 2023-09-30T01:34:58.616 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:58.595+0000 7f6e83fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:34:58.617 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:58.595+0000 7f6e83fff700 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.88:3304/0 2023-09-30T01:34:58.617 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:58.595+0000 7f6e83fff700 1 -- 172.21.15.88:0/2477597654 --> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e88009420 con 0x7f6e8c097cd0 2023-09-30T01:34:58.617 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:58.595+0000 7f6e915be700 1 --2- 172.21.15.88:0/2477597654 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7f6e8c09ed80 0x7f6e8c143110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-30T01:34:58.617 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:58.595+0000 7f6e90dbd700 1 --2- 172.21.15.88:0/2477597654 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6e8c0966a0 0x7f6e8c142690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:34:58.617 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:58.595+0000 7f6e83fff700 1 --2- 172.21.15.88:0/2477597654 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f6e8c097cd0 0x7f6e8c142bd0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f6e7c00ea80 tx=0x7f6e7c009250 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:34:58.617 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:58.595+0000 7f6e80ff9700 1 -- 172.21.15.88:0/2477597654 <== mon.8 v2:172.21.15.88:3304/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f6e7c009640 con 0x7f6e8c097cd0 2023-09-30T01:34:58.617 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:58.595+0000 7f6e80ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:34:58.617 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:58.595+0000 7f6e80ff9700 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2023-09-30T01:34:58.617 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:58.595+0000 7f6e80ff9700 10 monclient: dump: 2023-09-30T01:34:58.617 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:34:58.618 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:34:58.618 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:34:58.618 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:34:58.618 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:34:58.618 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:34:58.618 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:34:58.618 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:34:58.618 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:34:58.618 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:34:58.618 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:34:58.619 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:34:58.619 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:34:58.619 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:34:58.619 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:34:58.619 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:34:58.619 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:58.595+0000 7f6e80ff9700 10 monclient: _finish_auth 0 2023-09-30T01:34:58.619 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:58.595+0000 7f6e80ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:34:58.619 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:58.595+0000 7f6e80ff9700 1 -- 172.21.15.88:0/2477597654 <== mon.8 v2:172.21.15.88:3304/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e7c00cc70 con 0x7f6e8c097cd0 2023-09-30T01:34:58.619 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:58.595+0000 7f6e91dbf700 5 monclient: authenticate success, global_id 4598 2023-09-30T01:34:58.619 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:58.595+0000 7f6e80ff9700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:34:58.619 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:58.595+0000 7f6e91dbf700 10 monclient: _renew_subs 2023-09-30T01:34:58.620 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:58.595+0000 7f6e80ff9700 1 -- 172.21.15.88:0/2477597654 <== mon.8 v2:172.21.15.88:3304/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f6e7c00cde0 con 0x7f6e8c097cd0 2023-09-30T01:34:58.620 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:58.595+0000 7f6e91dbf700 10 monclient: _send_mon_message to mon.e at v2:172.21.15.88:3304/0 2023-09-30T01:34:58.620 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:58.595+0000 7f6e91dbf700 1 -- 172.21.15.88:0/2477597654 --> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e8c08f7b0 con 0x7f6e8c097cd0 2023-09-30T01:34:58.620 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:58.595+0000 7f6e82ffd700 4 set_mon_vals no callback set 2023-09-30T01:34:58.620 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:58.595+0000 7f6e80ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:34:58.620 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:58.595+0000 7f6e80ff9700 10 monclient: got monmap 1 from mon.e (according to old e1) 2023-09-30T01:34:58.620 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:58.595+0000 7f6e80ff9700 10 monclient: dump: 2023-09-30T01:34:58.620 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:34:58.620 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:34:58.620 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:34:58.620 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:34:58.621 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:34:58.621 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:34:58.621 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:34:58.621 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:34:58.621 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:34:58.621 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:34:58.621 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:34:58.621 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:34:58.621 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:34:58.621 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:34:58.621 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:34:58.622 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:34:58.622 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:58.595+0000 7f6e91dbf700 10 monclient: _renew_subs 2023-09-30T01:34:58.622 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:58.595+0000 7f6e91dbf700 10 monclient: _send_mon_message to mon.e at v2:172.21.15.88:3304/0 2023-09-30T01:34:58.622 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:58.595+0000 7f6e91dbf700 1 -- 172.21.15.88:0/2477597654 --> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e8c0903f0 con 0x7f6e8c097cd0 2023-09-30T01:34:58.622 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:58.595+0000 7f6e80ff9700 1 -- 172.21.15.88:0/2477597654 <== mon.8 v2:172.21.15.88:3304/0 4 ==== mgrmap(e 5) v1 ==== 40144+0+0 (secure 0 0 0) 0x7f6e7c00c7f0 con 0x7f6e8c097cd0 2023-09-30T01:34:58.622 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:58.595+0000 7f6e80ff9700 1 --2- 172.21.15.88:0/2477597654 >> v2:172.21.15.107:6812/29926 conn(0x7f6e78033360 0x7f6e78035810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:34:58.622 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:58.595+0000 7f6e80ff9700 1 -- 172.21.15.88:0/2477597654 <== mon.8 v2:172.21.15.88:3304/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 3502+0+0 (secure 0 0 0) 0x7f6e7c043970 con 0x7f6e8c097cd0 2023-09-30T01:34:58.622 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:58.595+0000 7f6e91dbf700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:34:58.622 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:58.595+0000 7f6e91dbf700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:34:58.622 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:58.595+0000 7f6e91dbf700 10 monclient: _send_mon_message to mon.e at v2:172.21.15.88:3304/0 2023-09-30T01:34:58.623 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:58.595+0000 7f6e91dbf700 1 -- 172.21.15.88:0/2477597654 --> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e8c0916e0 con 0x7f6e8c097cd0 2023-09-30T01:34:58.623 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:58.599+0000 7f6e90dbd700 1 --2- 172.21.15.88:0/2477597654 >> v2:172.21.15.107:6812/29926 conn(0x7f6e78033360 0x7f6e78035810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:34:58.623 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:58.599+0000 7f6e90dbd700 10 monclient: get_auth_request con 0x7f6e78033360 auth_method 0 2023-09-30T01:34:58.623 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:58.599+0000 7f6e90dbd700 1 --2- 172.21.15.88:0/2477597654 >> v2:172.21.15.107:6812/29926 conn(0x7f6e78033360 0x7f6e78035810 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f6e88009960 tx=0x7f6e88017000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:34:58.623 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:58.603+0000 7f6e80ff9700 1 -- 172.21.15.88:0/2477597654 <== mon.8 v2:172.21.15.88:3304/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f6e7c00caa0 con 0x7f6e8c097cd0 2023-09-30T01:34:58.623 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:58.603+0000 7f6e80ff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:34:58.623 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:58.603+0000 7f6e80ff9700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:34:58.802 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:58.799+0000 7f6e91dbf700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T01:34:58.803 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:58.799+0000 7f6e91dbf700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:34:58.803 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:58.799+0000 7f6e91dbf700 10 monclient: _send_mon_message to mon.e at v2:172.21.15.88:3304/0 2023-09-30T01:34:58.803 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:58.799+0000 7f6e91dbf700 1 -- 172.21.15.88:0/2477597654 --> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f6e8c067520 con 0x7f6e8c097cd0 2023-09-30T01:34:58.803 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:58.803+0000 7f6e80ff9700 1 -- 172.21.15.88:0/2477597654 <== mon.8 v2:172.21.15.88:3304/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f6e7c029080 con 0x7f6e8c097cd0 2023-09-30T01:34:58.803 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:58.803+0000 7f6e80ff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:34:58.804 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:58.803+0000 7f6e80ff9700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:34:58.804 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-09-30T01:34:58.804 INFO:teuthology.orchestra.run.smithi088.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:34:58.806 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:58.803+0000 7f6e91dbf700 1 -- 172.21.15.88:0/2477597654 >> v2:172.21.15.107:6812/29926 conn(0x7f6e78033360 msgr2=0x7f6e78035810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:34:58.806 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:58.803+0000 7f6e91dbf700 1 --2- 172.21.15.88:0/2477597654 >> v2:172.21.15.107:6812/29926 conn(0x7f6e78033360 0x7f6e78035810 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f6e88009960 tx=0x7f6e88017000 comp rx=0 tx=0).stop 2023-09-30T01:34:58.806 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:58.803+0000 7f6e91dbf700 10 monclient: shutdown 2023-09-30T01:34:58.806 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:58.803+0000 7f6e91dbf700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:34:58.806 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:58.803+0000 7f6e91dbf700 1 -- 172.21.15.88:0/2477597654 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f6e8c097cd0 msgr2=0x7f6e8c142bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:34:58.807 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:58.803+0000 7f6e91dbf700 1 --2- 172.21.15.88:0/2477597654 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f6e8c097cd0 0x7f6e8c142bd0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f6e7c00ea80 tx=0x7f6e7c009250 comp rx=0 tx=0).stop 2023-09-30T01:34:58.807 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:58.803+0000 7f6e91dbf700 1 -- 172.21.15.88:0/2477597654 shutdown_connections 2023-09-30T01:34:58.807 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:58.803+0000 7f6e91dbf700 1 --2- 172.21.15.88:0/2477597654 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7f6e8c09ed80 0x7f6e8c143110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:34:58.807 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:58.803+0000 7f6e91dbf700 1 --2- 172.21.15.88:0/2477597654 >> v2:172.21.15.107:6812/29926 conn(0x7f6e78033360 0x7f6e78035810 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:34:58.807 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:58.803+0000 7f6e91dbf700 1 --2- 172.21.15.88:0/2477597654 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f6e8c097cd0 0x7f6e8c142bd0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:34:58.807 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:58.803+0000 7f6e91dbf700 1 --2- 172.21.15.88:0/2477597654 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6e8c0966a0 0x7f6e8c142690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:34:58.807 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:58.803+0000 7f6e91dbf700 1 -- 172.21.15.88:0/2477597654 >> 172.21.15.88:0/2477597654 conn(0x7f6e8c0a0910 msgr2=0x7f6e8c0a1770 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:34:58.807 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:58.803+0000 7f6e91dbf700 1 -- 172.21.15.88:0/2477597654 shutdown_connections 2023-09-30T01:34:58.807 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:34:58.803+0000 7f6e91dbf700 1 -- 172.21.15.88:0/2477597654 wait complete. 2023-09-30T01:35:01.818 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:35:01.928 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:01.927+0000 7f9f0f59e700 10 monclient: get_monmap_and_config 2023-09-30T01:35:01.929 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:01.927+0000 7f9f0f59e700 10 monclient: build_initial_monmap 2023-09-30T01:35:01.929 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:01.927+0000 7f9f0f59e700 10 monclient: monmap: 2023-09-30T01:35:01.929 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2023-09-30T01:35:01.929 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:35:01.929 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:35:01.929178+0000 2023-09-30T01:35:01.929 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:35:01.929178+0000 2023-09-30T01:35:01.929 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2023-09-30T01:35:01.929 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:35:01.929 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a-legacy 2023-09-30T01:35:01.929 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.107:6789/0 mon.noname-f-legacy 2023-09-30T01:35:01.930 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.noname-b 2023-09-30T01:35:01.930 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.noname-g 2023-09-30T01:35:01.930 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.noname-c 2023-09-30T01:35:01.930 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.noname-h 2023-09-30T01:35:01.930 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.noname-d 2023-09-30T01:35:01.930 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.noname-i 2023-09-30T01:35:01.930 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.noname-e 2023-09-30T01:35:01.930 INFO:teuthology.orchestra.run.smithi088.stderr:9: v2:172.21.15.88:3300/0 mon.noname-a 2023-09-30T01:35:01.930 INFO:teuthology.orchestra.run.smithi088.stderr:10: v2:172.21.15.107:3300/0 mon.noname-f 2023-09-30T01:35:01.930 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:35:01.934 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:01.927+0000 7f9f0f59e700 1 Processor -- start 2023-09-30T01:35:01.934 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:01.927+0000 7f9f0f59e700 1 -- start start 2023-09-30T01:35:01.934 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:01.927+0000 7f9f0f59e700 10 monclient: init 2023-09-30T01:35:01.935 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:01.931+0000 7f9f0f59e700 10 monclient: _reopen_session rank -1 2023-09-30T01:35:01.935 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:01.931+0000 7f9f0f59e700 10 monclient: _add_conns ranks=[1,8,3,6,4,2,0,10,5,9,7] 2023-09-30T01:35:01.935 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:01.931+0000 7f9f0f59e700 10 monclient(hunting): picked mon.noname-f-legacy con 0x7f9f100966a0 addr v1:172.21.15.107:6789/0 2023-09-30T01:35:01.935 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:01.931+0000 7f9f0f59e700 1 --2- >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f9f10097cd0 0x7f9f100980a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:35:01.935 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:01.931+0000 7f9f0f59e700 10 monclient(hunting): picked mon.noname-e con 0x7f9f10097cd0 addr [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] 2023-09-30T01:35:01.935 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:01.931+0000 7f9f0f59e700 1 --2- >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7f9f101385d0 0x7f9f1013a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:35:01.935 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:01.931+0000 7f9f0f59e700 10 monclient(hunting): picked mon.noname-g con 0x7f9f101385d0 addr [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] 2023-09-30T01:35:01.935 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:01.931+0000 7f9f0f59e700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f1004bb20 con 0x7f9f100966a0 2023-09-30T01:35:01.935 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:01.931+0000 7f9f0f59e700 10 monclient(hunting): start opening mon connection 2023-09-30T01:35:01.935 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:01.931+0000 7f9f0f59e700 1 -- --> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f10097180 con 0x7f9f101385d0 2023-09-30T01:35:01.935 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:01.931+0000 7f9f0f59e700 10 monclient(hunting): start opening mon connection 2023-09-30T01:35:01.935 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:01.931+0000 7f9f0f59e700 1 -- --> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] -- mon_getmap magic: 0 v1 -- 0x7f9f10097920 con 0x7f9f10097cd0 2023-09-30T01:35:01.936 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:01.931+0000 7f9f0f59e700 10 monclient(hunting): _renew_subs 2023-09-30T01:35:01.936 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:01.931+0000 7f9f0f59e700 10 monclient(hunting): authenticate will time out at 973.968393s 2023-09-30T01:35:01.936 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:01.931+0000 7f9f0dd9b700 1 --2- >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f9f10097cd0 0x7f9f100980a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:35:01.936 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:01.931+0000 7f9f0dd9b700 1 --2- >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f9f10097cd0 0x7f9f100980a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3304/0 says I am v2:172.21.15.88:56338/0 (socket says 172.21.15.88:56338) 2023-09-30T01:35:01.936 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:01.931+0000 7f9f0dd9b700 1 -- 172.21.15.88:0/2466173157 learned_addr learned my addr 172.21.15.88:0/2466173157 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-09-30T01:35:01.936 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:01.931+0000 7f9f0dd9b700 10 monclient(hunting): get_auth_request con 0x7f9f10097cd0 auth_method 0 2023-09-30T01:35:01.936 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:01.931+0000 7f9f0dd9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:35:01.936 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:01.931+0000 7f9f0dd9b700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:35:01.936 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:01.931+0000 7f9f0dd9b700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:35:01.936 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:01.931+0000 7f9f0ed9d700 1 --2- 172.21.15.88:0/2466173157 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7f9f101385d0 0x7f9f1013a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:35:01.936 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:01.931+0000 7f9f0dd9b700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:35:01.936 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:01.931+0000 7f9f0dd9b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:35:01.937 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:01.931+0000 7f9f0dd9b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:35:01.937 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:01.931+0000 7f9f0ed9d700 10 monclient(hunting): get_auth_request con 0x7f9f101385d0 auth_method 0 2023-09-30T01:35:01.937 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:01.931+0000 7f9f0ed9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:35:01.937 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:01.931+0000 7f9f0ed9d700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:35:01.937 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:01.931+0000 7f9f0ed9d700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:35:01.937 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:01.931+0000 7f9f0ed9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:35:01.937 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:01.931+0000 7f9f0ed9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:35:01.937 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:01.931+0000 7f9f0ed9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:35:01.937 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:01.931+0000 7f9f0dd9b700 10 monclient(hunting): handle_auth_done global_id 4607 payload 274 2023-09-30T01:35:01.937 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:01.931+0000 7f9f0dd9b700 1 -- 172.21.15.88:0/2466173157 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7f9f101385d0 msgr2=0x7f9f1013a9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:35:01.937 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:01.931+0000 7f9f0dd9b700 1 --2- 172.21.15.88:0/2466173157 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7f9f101385d0 0x7f9f1013a9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:35:01.937 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:01.931+0000 7f9f0dd9b700 1 -- 172.21.15.88:0/2466173157 >> v1:172.21.15.107:6789/0 conn(0x7f9f100966a0 legacy=0x7f9f10096a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:35:01.937 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:01.931+0000 7f9f0dd9b700 10 monclient: _finish_hunting 0 2023-09-30T01:35:01.938 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:01.931+0000 7f9f0dd9b700 1 monclient: found mon.noname-e 2023-09-30T01:35:01.938 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:01.931+0000 7f9f0dd9b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:35:01.938 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:01.931+0000 7f9f0dd9b700 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.88:3304/0 2023-09-30T01:35:01.938 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:01.931+0000 7f9f0dd9b700 1 -- 172.21.15.88:0/2466173157 --> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f1009a340 con 0x7f9f10097cd0 2023-09-30T01:35:01.938 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:01.931+0000 7f9f0dd9b700 1 --2- 172.21.15.88:0/2466173157 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f9f10097cd0 0x7f9f100980a0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f9f04009a10 tx=0x7f9f04009250 comp rx=0 tx=0).ready entity=mon.8 client_cookie=a36a8bb9c24f04ef server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:35:01.938 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:01.931+0000 7f9f0d59a700 1 -- 172.21.15.88:0/2466173157 <== mon.8 v2:172.21.15.88:3304/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f9f0400bca0 con 0x7f9f10097cd0 2023-09-30T01:35:01.938 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:01.931+0000 7f9f0d59a700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:35:01.938 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:01.931+0000 7f9f0d59a700 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2023-09-30T01:35:01.938 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:01.931+0000 7f9f0d59a700 10 monclient: dump: 2023-09-30T01:35:01.938 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:35:01.938 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:35:01.938 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:35:01.939 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:35:01.939 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:35:01.939 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:35:01.939 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:35:01.939 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:35:01.939 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:35:01.939 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:35:01.939 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:35:01.939 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:35:01.939 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:35:01.939 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:35:01.939 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:35:01.939 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:35:01.940 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:01.931+0000 7f9f0d59a700 10 monclient: _finish_auth 0 2023-09-30T01:35:01.940 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:01.931+0000 7f9f0d59a700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:35:01.940 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:01.931+0000 7f9f0d59a700 1 -- 172.21.15.88:0/2466173157 <== mon.8 v2:172.21.15.88:3304/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f0400be20 con 0x7f9f10097cd0 2023-09-30T01:35:01.940 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:01.931+0000 7f9f0f59e700 5 monclient: authenticate success, global_id 4607 2023-09-30T01:35:01.940 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:01.931+0000 7f9f0f59e700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:35:01.940 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:01.931+0000 7f9f0d59a700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:35:01.940 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:01.931+0000 7f9f0f59e700 10 monclient: get_monmap_and_config success 2023-09-30T01:35:01.940 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:01.931+0000 7f9f0d59a700 1 -- 172.21.15.88:0/2466173157 <== mon.8 v2:172.21.15.88:3304/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f9f040118a0 con 0x7f9f10097cd0 2023-09-30T01:35:01.940 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:01.931+0000 7f9f0d59a700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:35:01.940 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:01.931+0000 7f9f0f59e700 4 set_mon_vals no callback set 2023-09-30T01:35:01.940 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:01.931+0000 7f9f0d59a700 10 monclient: got monmap 1 from mon.e (according to old e1) 2023-09-30T01:35:01.940 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:01.931+0000 7f9f0d59a700 10 monclient: dump: 2023-09-30T01:35:01.941 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:35:01.941 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:35:01.941 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:35:01.941 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:35:01.941 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:35:01.941 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:35:01.941 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:35:01.941 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:35:01.941 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:35:01.941 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:35:01.941 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:35:01.941 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:35:01.941 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:35:01.942 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:35:01.942 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:35:01.942 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:35:01.942 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:01.931+0000 7f9f0f59e700 10 monclient: shutdown 2023-09-30T01:35:01.942 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:01.931+0000 7f9f0f59e700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:35:01.942 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:01.931+0000 7f9f0f59e700 1 -- 172.21.15.88:0/2466173157 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f9f10097cd0 msgr2=0x7f9f100980a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:35:01.942 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:01.931+0000 7f9f0f59e700 1 --2- 172.21.15.88:0/2466173157 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f9f10097cd0 0x7f9f100980a0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f9f04009a10 tx=0x7f9f04009250 comp rx=0 tx=0).stop 2023-09-30T01:35:01.942 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:01.931+0000 7f9f0f59e700 1 -- 172.21.15.88:0/2466173157 shutdown_connections 2023-09-30T01:35:01.942 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:01.931+0000 7f9f0f59e700 1 --2- 172.21.15.88:0/2466173157 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7f9f101385d0 0x7f9f1013a9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:35:01.942 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:01.931+0000 7f9f0f59e700 1 --2- 172.21.15.88:0/2466173157 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f9f10097cd0 0x7f9f100980a0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:35:01.942 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:01.931+0000 7f9f0f59e700 1 -- 172.21.15.88:0/2466173157 >> 172.21.15.88:0/2466173157 conn(0x7f9f100a0910 msgr2=0x7f9f100a33c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:35:01.942 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:01.931+0000 7f9f0f59e700 1 -- 172.21.15.88:0/2466173157 shutdown_connections 2023-09-30T01:35:01.943 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:01.931+0000 7f9f0f59e700 1 -- 172.21.15.88:0/2466173157 wait complete. 2023-09-30T01:35:01.948 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:01.935+0000 7f9f0f59e700 10 monclient: build_initial_monmap 2023-09-30T01:35:01.948 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:01.935+0000 7f9f0f59e700 10 monclient: monmap: 2023-09-30T01:35:01.948 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2023-09-30T01:35:01.948 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:35:01.948 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 0.000000 2023-09-30T01:35:01.948 INFO:teuthology.orchestra.run.smithi088.stderr:created 0.000000 2023-09-30T01:35:01.948 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2023-09-30T01:35:01.948 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 1 2023-09-30T01:35:01.948 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.noname-a 2023-09-30T01:35:01.948 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.noname-f 2023-09-30T01:35:01.948 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.noname-b 2023-09-30T01:35:01.949 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.noname-g 2023-09-30T01:35:01.949 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.noname-c 2023-09-30T01:35:01.949 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.noname-h 2023-09-30T01:35:01.949 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.noname-d 2023-09-30T01:35:01.949 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.noname-i 2023-09-30T01:35:01.949 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.noname-e 2023-09-30T01:35:01.949 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:35:01.949 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:01.935+0000 7f9f0f59e700 1 Processor -- start 2023-09-30T01:35:01.949 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:01.935+0000 7f9f0f59e700 1 -- start start 2023-09-30T01:35:01.949 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:01.935+0000 7f9f0f59e700 10 monclient: init 2023-09-30T01:35:01.949 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:01.935+0000 7f9f0f59e700 10 monclient: _reopen_session rank -1 2023-09-30T01:35:01.949 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:01.935+0000 7f9f0f59e700 10 monclient: _add_conns ranks=[2,4,5,8,6,7,1,3,0] 2023-09-30T01:35:01.949 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:01.935+0000 7f9f0f59e700 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f9f100966a0 0x7f9f101375e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:35:01.949 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:01.935+0000 7f9f0f59e700 10 monclient(hunting): picked mon.noname-b con 0x7f9f100966a0 addr [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] 2023-09-30T01:35:01.950 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:01.935+0000 7f9f0f59e700 1 --2- >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f9f10097cd0 0x7f9f10137b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:35:01.950 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:01.935+0000 7f9f0f59e700 10 monclient(hunting): picked mon.noname-c con 0x7f9f10097cd0 addr [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] 2023-09-30T01:35:01.950 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:01.935+0000 7f9f0f59e700 1 --2- >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7f9f101385d0 0x7f9f10138060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:35:01.950 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:01.935+0000 7f9f0f59e700 10 monclient(hunting): picked mon.noname-h con 0x7f9f101385d0 addr [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] 2023-09-30T01:35:01.950 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:01.935+0000 7f9f0f59e700 10 monclient(hunting): start opening mon connection 2023-09-30T01:35:01.950 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:01.935+0000 7f9f0f59e700 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f10091500 con 0x7f9f100966a0 2023-09-30T01:35:01.950 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:01.935+0000 7f9f0f59e700 10 monclient(hunting): start opening mon connection 2023-09-30T01:35:01.950 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:01.935+0000 7f9f0f59e700 1 -- --> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] -- mon_getmap magic: 0 v1 -- 0x7f9f100996c0 con 0x7f9f10097cd0 2023-09-30T01:35:01.950 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:01.935+0000 7f9f0f59e700 10 monclient(hunting): start opening mon connection 2023-09-30T01:35:01.950 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:01.935+0000 7f9f0f59e700 1 -- --> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] -- mon_getmap magic: 0 v1 -- 0x7f9f10063630 con 0x7f9f101385d0 2023-09-30T01:35:01.950 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:01.935+0000 7f9f0f59e700 10 monclient(hunting): _renew_subs 2023-09-30T01:35:01.950 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:01.935+0000 7f9f0f59e700 10 monclient(hunting): authenticate will time out at 973.974141s 2023-09-30T01:35:01.951 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:01.935+0000 7f9f0e59c700 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f9f100966a0 0x7f9f101375e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:35:01.951 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:01.935+0000 7f9f0dd9b700 1 --2- >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f9f10097cd0 0x7f9f10137b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:35:01.951 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:01.935+0000 7f9f0dd9b700 1 --2- >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f9f10097cd0 0x7f9f10137b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3302/0 says I am v2:172.21.15.88:33284/0 (socket says 172.21.15.88:33284) 2023-09-30T01:35:01.951 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:01.935+0000 7f9f0dd9b700 1 -- 172.21.15.88:0/3375991878 learned_addr learned my addr 172.21.15.88:0/3375991878 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-09-30T01:35:01.951 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:01.935+0000 7f9f0dd9b700 10 monclient(hunting): get_auth_request con 0x7f9f10097cd0 auth_method 0 2023-09-30T01:35:01.951 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:01.935+0000 7f9f0dd9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:35:01.951 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:01.935+0000 7f9f0dd9b700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:35:01.951 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:01.935+0000 7f9f0dd9b700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:35:01.951 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:01.935+0000 7f9f0e59c700 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f9f100966a0 0x7f9f101375e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3301/0 says I am v2:172.21.15.88:36612/0 (socket says 172.21.15.88:36612) 2023-09-30T01:35:01.951 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:01.935+0000 7f9f0e59c700 10 monclient(hunting): get_auth_request con 0x7f9f100966a0 auth_method 0 2023-09-30T01:35:01.951 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:01.935+0000 7f9f0e59c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:35:01.951 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:01.935+0000 7f9f0e59c700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:35:01.952 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:01.935+0000 7f9f0e59c700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:35:01.952 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:01.935+0000 7f9f0ed9d700 1 --2- 172.21.15.88:0/3375991878 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7f9f101385d0 0x7f9f10138060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:35:01.952 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:01.935+0000 7f9f0ed9d700 10 monclient(hunting): get_auth_request con 0x7f9f101385d0 auth_method 0 2023-09-30T01:35:01.952 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:01.935+0000 7f9f0ed9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:35:01.952 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:01.935+0000 7f9f0ed9d700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:35:01.952 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:01.935+0000 7f9f0ed9d700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:35:01.952 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:01.935+0000 7f9f0dd9b700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:35:01.952 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:01.935+0000 7f9f0dd9b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:35:01.952 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:01.935+0000 7f9f0dd9b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:35:01.952 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:01.935+0000 7f9f0e59c700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:35:01.952 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:01.935+0000 7f9f0e59c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:35:01.952 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:01.935+0000 7f9f0e59c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:35:01.953 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:01.935+0000 7f9f0ed9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:35:01.953 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:01.935+0000 7f9f0ed9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:35:01.953 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:01.935+0000 7f9f0ed9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:35:01.953 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:01.935+0000 7f9f0e59c700 10 monclient(hunting): handle_auth_done global_id 4529 payload 824 2023-09-30T01:35:01.953 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:01.935+0000 7f9f0e59c700 1 -- 172.21.15.88:0/3375991878 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7f9f101385d0 msgr2=0x7f9f10138060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:35:01.953 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:01.935+0000 7f9f0e59c700 1 --2- 172.21.15.88:0/3375991878 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7f9f101385d0 0x7f9f10138060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:35:01.953 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:01.935+0000 7f9f0e59c700 1 -- 172.21.15.88:0/3375991878 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f9f10097cd0 msgr2=0x7f9f10137b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:35:01.953 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:01.935+0000 7f9f0e59c700 1 --2- 172.21.15.88:0/3375991878 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f9f10097cd0 0x7f9f10137b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:35:01.953 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:01.935+0000 7f9f0e59c700 10 monclient: _finish_hunting 0 2023-09-30T01:35:01.953 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:01.935+0000 7f9f0e59c700 1 monclient: found mon.noname-b 2023-09-30T01:35:01.953 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:01.935+0000 7f9f0e59c700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:35:01.953 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:01.935+0000 7f9f0e59c700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.88:3301/0 2023-09-30T01:35:01.954 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:01.935+0000 7f9f0e59c700 1 -- 172.21.15.88:0/3375991878 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f040094f0 con 0x7f9f100966a0 2023-09-30T01:35:01.954 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:01.935+0000 7f9f0ed9d700 1 --2- 172.21.15.88:0/3375991878 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7f9f101385d0 0x7f9f10138060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:35:01.954 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:01.935+0000 7f9f0dd9b700 1 --2- 172.21.15.88:0/3375991878 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f9f10097cd0 0x7f9f10137b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:35:01.954 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:01.935+0000 7f9f0e59c700 1 --2- 172.21.15.88:0/3375991878 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f9f100966a0 0x7f9f101375e0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f9f0800c590 tx=0x7f9f08007e90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:35:01.954 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:01.935+0000 7f9efe7fc700 1 -- 172.21.15.88:0/3375991878 <== mon.2 v2:172.21.15.88:3301/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f9f08001170 con 0x7f9f100966a0 2023-09-30T01:35:01.954 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:01.935+0000 7f9efe7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:35:01.954 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:01.935+0000 7f9efe7fc700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T01:35:01.954 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:01.935+0000 7f9efe7fc700 10 monclient: dump: 2023-09-30T01:35:01.954 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:35:01.954 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:35:01.954 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:35:01.955 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:35:01.955 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:35:01.955 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:35:01.955 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:35:01.955 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:35:01.955 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:35:01.955 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:35:01.955 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:35:01.955 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:35:01.955 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:35:01.955 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:35:01.955 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:35:01.955 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:35:01.956 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:01.939+0000 7f9efe7fc700 10 monclient: _finish_auth 0 2023-09-30T01:35:01.956 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:01.939+0000 7f9efe7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:35:01.956 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:01.939+0000 7f9f0f59e700 5 monclient: authenticate success, global_id 4529 2023-09-30T01:35:01.956 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:01.939+0000 7f9efe7fc700 1 -- 172.21.15.88:0/3375991878 <== mon.2 v2:172.21.15.88:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f0800ce60 con 0x7f9f100966a0 2023-09-30T01:35:01.956 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:01.939+0000 7f9f0f59e700 10 monclient: _renew_subs 2023-09-30T01:35:01.956 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:01.939+0000 7f9f0f59e700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.88:3301/0 2023-09-30T01:35:01.956 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:01.939+0000 7f9f0f59e700 1 -- 172.21.15.88:0/3375991878 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f10090330 con 0x7f9f100966a0 2023-09-30T01:35:01.956 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:01.939+0000 7f9efe7fc700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:35:01.956 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:01.939+0000 7f9efe7fc700 1 -- 172.21.15.88:0/3375991878 <== mon.2 v2:172.21.15.88:3301/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f9f08011e90 con 0x7f9f100966a0 2023-09-30T01:35:01.956 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:01.939+0000 7f9efe7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:35:01.956 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:01.939+0000 7f9efe7fc700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T01:35:01.956 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:01.939+0000 7f9f0cd99700 4 set_mon_vals no callback set 2023-09-30T01:35:01.957 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:01.939+0000 7f9efe7fc700 10 monclient: dump: 2023-09-30T01:35:01.957 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:35:01.957 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:35:01.957 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:35:01.957 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:35:01.957 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:35:01.957 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:35:01.957 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:35:01.957 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:35:01.957 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:35:01.957 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:35:01.957 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:35:01.958 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:35:01.958 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:35:01.958 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:35:01.958 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:35:01.958 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:35:01.958 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:01.939+0000 7f9f0f59e700 10 monclient: _renew_subs 2023-09-30T01:35:01.958 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:01.939+0000 7f9f0f59e700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.88:3301/0 2023-09-30T01:35:01.958 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:01.939+0000 7f9f0f59e700 1 -- 172.21.15.88:0/3375991878 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f1008f730 con 0x7f9f100966a0 2023-09-30T01:35:01.958 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:01.939+0000 7f9efe7fc700 1 -- 172.21.15.88:0/3375991878 <== mon.2 v2:172.21.15.88:3301/0 4 ==== mgrmap(e 5) v1 ==== 40144+0+0 (secure 0 0 0) 0x7f9f08011590 con 0x7f9f100966a0 2023-09-30T01:35:01.958 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:01.939+0000 7f9f0f59e700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:35:01.958 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:01.939+0000 7f9f0f59e700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:35:01.958 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:01.939+0000 7f9f0f59e700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.88:3301/0 2023-09-30T01:35:01.959 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:01.939+0000 7f9efe7fc700 1 --2- 172.21.15.88:0/3375991878 >> v2:172.21.15.107:6812/29926 conn(0x7f9ef0033360 0x7f9ef0035810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:35:01.959 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:01.939+0000 7f9f0f59e700 1 -- 172.21.15.88:0/3375991878 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9f1008f8c0 con 0x7f9f100966a0 2023-09-30T01:35:01.959 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:01.939+0000 7f9efe7fc700 1 -- 172.21.15.88:0/3375991878 <== mon.2 v2:172.21.15.88:3301/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 3502+0+0 (secure 0 0 0) 0x7f9f08042260 con 0x7f9f100966a0 2023-09-30T01:35:01.959 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:01.939+0000 7f9f0dd9b700 1 --2- 172.21.15.88:0/3375991878 >> v2:172.21.15.107:6812/29926 conn(0x7f9ef0033360 0x7f9ef0035810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:35:01.959 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:01.939+0000 7f9f0dd9b700 10 monclient: get_auth_request con 0x7f9ef0033360 auth_method 0 2023-09-30T01:35:01.959 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:01.939+0000 7f9f0dd9b700 1 --2- 172.21.15.88:0/3375991878 >> v2:172.21.15.107:6812/29926 conn(0x7f9ef0033360 0x7f9ef0035810 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f9f040094c0 tx=0x7f9f04017000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:35:01.959 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:01.947+0000 7f9efe7fc700 1 -- 172.21.15.88:0/3375991878 <== mon.2 v2:172.21.15.88:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f9f080173f0 con 0x7f9f100966a0 2023-09-30T01:35:01.959 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:01.947+0000 7f9efe7fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:35:01.959 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:01.947+0000 7f9efe7fc700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:35:02.145 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:02.143+0000 7f9f0f59e700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T01:35:02.146 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:02.143+0000 7f9f0f59e700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:35:02.146 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:02.143+0000 7f9f0f59e700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.88:3301/0 2023-09-30T01:35:02.146 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:02.143+0000 7f9f0f59e700 1 -- 172.21.15.88:0/3375991878 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f9f10067520 con 0x7f9f100966a0 2023-09-30T01:35:02.146 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:02.143+0000 7f9efe7fc700 1 -- 172.21.15.88:0/3375991878 <== mon.2 v2:172.21.15.88:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f9f08017030 con 0x7f9f100966a0 2023-09-30T01:35:02.146 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:02.143+0000 7f9efe7fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:35:02.146 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:02.143+0000 7f9efe7fc700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:35:02.147 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-09-30T01:35:02.147 INFO:teuthology.orchestra.run.smithi088.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:35:02.149 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:02.147+0000 7f9f0f59e700 1 -- 172.21.15.88:0/3375991878 >> v2:172.21.15.107:6812/29926 conn(0x7f9ef0033360 msgr2=0x7f9ef0035810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:35:02.149 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:02.147+0000 7f9f0f59e700 1 --2- 172.21.15.88:0/3375991878 >> v2:172.21.15.107:6812/29926 conn(0x7f9ef0033360 0x7f9ef0035810 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f9f040094c0 tx=0x7f9f04017000 comp rx=0 tx=0).stop 2023-09-30T01:35:02.149 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:02.147+0000 7f9f0f59e700 10 monclient: shutdown 2023-09-30T01:35:02.149 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:02.147+0000 7f9f0f59e700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:35:02.149 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:02.147+0000 7f9f0f59e700 1 -- 172.21.15.88:0/3375991878 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f9f100966a0 msgr2=0x7f9f101375e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:35:02.149 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:02.147+0000 7f9f0f59e700 1 --2- 172.21.15.88:0/3375991878 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f9f100966a0 0x7f9f101375e0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f9f0800c590 tx=0x7f9f08007e90 comp rx=0 tx=0).stop 2023-09-30T01:35:02.149 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:02.147+0000 7f9f0f59e700 1 -- 172.21.15.88:0/3375991878 shutdown_connections 2023-09-30T01:35:02.149 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:02.147+0000 7f9f0f59e700 1 --2- 172.21.15.88:0/3375991878 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7f9f101385d0 0x7f9f10138060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:35:02.149 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:02.147+0000 7f9f0f59e700 1 --2- 172.21.15.88:0/3375991878 >> v2:172.21.15.107:6812/29926 conn(0x7f9ef0033360 0x7f9ef0035810 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:35:02.149 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:02.147+0000 7f9f0f59e700 1 --2- 172.21.15.88:0/3375991878 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f9f10097cd0 0x7f9f10137b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:35:02.149 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:02.147+0000 7f9f0f59e700 1 --2- 172.21.15.88:0/3375991878 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f9f100966a0 0x7f9f101375e0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:35:02.150 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:02.147+0000 7f9f0f59e700 1 -- 172.21.15.88:0/3375991878 >> 172.21.15.88:0/3375991878 conn(0x7f9f100a0910 msgr2=0x7f9f100a1400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:35:02.150 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:02.147+0000 7f9f0f59e700 1 -- 172.21.15.88:0/3375991878 shutdown_connections 2023-09-30T01:35:02.150 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:02.147+0000 7f9f0f59e700 1 -- 172.21.15.88:0/3375991878 wait complete. 2023-09-30T01:35:05.161 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:35:05.272 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:05.271+0000 7fb6a3fff700 10 monclient: get_monmap_and_config 2023-09-30T01:35:05.272 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:05.271+0000 7fb6a3fff700 10 monclient: build_initial_monmap 2023-09-30T01:35:05.272 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:05.271+0000 7fb6a3fff700 10 monclient: monmap: 2023-09-30T01:35:05.272 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2023-09-30T01:35:05.272 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:35:05.272 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:35:05.272110+0000 2023-09-30T01:35:05.272 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:35:05.272110+0000 2023-09-30T01:35:05.272 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2023-09-30T01:35:05.272 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:35:05.273 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a-legacy 2023-09-30T01:35:05.273 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.107:6789/0 mon.noname-f-legacy 2023-09-30T01:35:05.273 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.noname-b 2023-09-30T01:35:05.273 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.noname-g 2023-09-30T01:35:05.273 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.noname-c 2023-09-30T01:35:05.273 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.noname-h 2023-09-30T01:35:05.273 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.noname-d 2023-09-30T01:35:05.273 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.noname-i 2023-09-30T01:35:05.273 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.noname-e 2023-09-30T01:35:05.273 INFO:teuthology.orchestra.run.smithi088.stderr:9: v2:172.21.15.88:3300/0 mon.noname-a 2023-09-30T01:35:05.273 INFO:teuthology.orchestra.run.smithi088.stderr:10: v2:172.21.15.107:3300/0 mon.noname-f 2023-09-30T01:35:05.273 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:35:05.277 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:05.271+0000 7fb6a3fff700 1 Processor -- start 2023-09-30T01:35:05.277 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:05.271+0000 7fb6a3fff700 1 -- start start 2023-09-30T01:35:05.277 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:05.271+0000 7fb6a3fff700 10 monclient: init 2023-09-30T01:35:05.277 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:05.271+0000 7fb6a3fff700 10 monclient: _reopen_session rank -1 2023-09-30T01:35:05.277 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:05.271+0000 7fb6a3fff700 10 monclient: _add_conns ranks=[3,10,1,8,9,4,2,7,6,0,5] 2023-09-30T01:35:05.277 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:05.271+0000 7fb6a3fff700 1 --2- >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7fb6a4097a40 0x7fb6a4132700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:35:05.278 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:05.271+0000 7fb6a3fff700 10 monclient(hunting): picked mon.noname-g con 0x7fb6a4097a40 addr [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] 2023-09-30T01:35:05.278 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:05.271+0000 7fb6a3fff700 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fb6a41311c0 0x7fb6a4132c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:35:05.278 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:05.271+0000 7fb6a3fff700 10 monclient(hunting): picked mon.noname-f con 0x7fb6a41311c0 addr v2:172.21.15.107:3300/0 2023-09-30T01:35:05.278 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:05.271+0000 7fb6a3fff700 10 monclient(hunting): picked mon.noname-f-legacy con 0x7fb6a41332e0 addr v1:172.21.15.107:6789/0 2023-09-30T01:35:05.278 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:05.271+0000 7fb6a3fff700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6a404bb20 con 0x7fb6a41332e0 2023-09-30T01:35:05.278 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:05.271+0000 7fb6a3fff700 10 monclient(hunting): start opening mon connection 2023-09-30T01:35:05.278 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:05.271+0000 7fb6a3fff700 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb6a4131590 con 0x7fb6a41311c0 2023-09-30T01:35:05.278 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:05.271+0000 7fb6a3fff700 10 monclient(hunting): start opening mon connection 2023-09-30T01:35:05.278 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:05.271+0000 7fb6a3fff700 1 -- --> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb6a409a5d0 con 0x7fb6a4097a40 2023-09-30T01:35:05.278 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:05.271+0000 7fb6a3fff700 10 monclient(hunting): _renew_subs 2023-09-30T01:35:05.278 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:05.271+0000 7fb6a3fff700 10 monclient(hunting): authenticate will time out at 977.309557s 2023-09-30T01:35:05.279 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:05.271+0000 7fb69bfff700 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fb6a41311c0 0x7fb6a4132c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:35:05.279 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:05.271+0000 7fb69bfff700 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fb6a41311c0 0x7fb6a4132c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.88:54920/0 (socket says 172.21.15.88:54920) 2023-09-30T01:35:05.279 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:05.271+0000 7fb69bfff700 1 -- 172.21.15.88:0/285696263 learned_addr learned my addr 172.21.15.88:0/285696263 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-09-30T01:35:05.279 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:05.271+0000 7fb6a2ffd700 1 --2- >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7fb6a4097a40 0x7fb6a4132700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:35:05.279 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:05.271+0000 7fb6a37fe700 1 --1- 172.21.15.88:0/285696263 >> v1:172.21.15.107:6789/0 conn(0x7fb6a41332e0 0x7fb6a4133740 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.88:44772/0 (socket says 172.21.15.88:44772) 2023-09-30T01:35:05.279 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:05.271+0000 7fb69bfff700 10 monclient(hunting): get_auth_request con 0x7fb6a41311c0 auth_method 0 2023-09-30T01:35:05.279 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:05.271+0000 7fb69bfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:35:05.279 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:05.271+0000 7fb69bfff700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:35:05.279 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:05.271+0000 7fb69bfff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:35:05.279 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:05.271+0000 7fb6a2ffd700 10 monclient(hunting): get_auth_request con 0x7fb6a4097a40 auth_method 0 2023-09-30T01:35:05.279 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:05.271+0000 7fb6a2ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:35:05.279 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:05.271+0000 7fb6a2ffd700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:35:05.280 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:05.271+0000 7fb6a2ffd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:35:05.280 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:05.271+0000 7fb69bfff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:35:05.280 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:05.271+0000 7fb69bfff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:35:05.280 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:05.271+0000 7fb69bfff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:35:05.280 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:05.271+0000 7fb6a2ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:35:05.280 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:05.271+0000 7fb6a2ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:35:05.280 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:05.271+0000 7fb6a2ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:35:05.280 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:05.271+0000 7fb69bfff700 10 monclient(hunting): handle_auth_done global_id 4762 payload 274 2023-09-30T01:35:05.280 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:05.271+0000 7fb69bfff700 1 -- 172.21.15.88:0/285696263 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7fb6a4097a40 msgr2=0x7fb6a4132700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:35:05.280 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:05.271+0000 7fb69bfff700 1 --2- 172.21.15.88:0/285696263 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7fb6a4097a40 0x7fb6a4132700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:35:05.280 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:05.271+0000 7fb69bfff700 1 -- 172.21.15.88:0/285696263 >> v1:172.21.15.107:6789/0 conn(0x7fb6a41332e0 legacy=0x7fb6a4133740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:35:05.280 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:05.271+0000 7fb69bfff700 10 monclient: _finish_hunting 0 2023-09-30T01:35:05.280 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:05.271+0000 7fb69bfff700 1 monclient: found mon.noname-f 2023-09-30T01:35:05.281 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:05.271+0000 7fb69bfff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:35:05.281 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:05.271+0000 7fb69bfff700 10 monclient: _send_mon_message to mon.noname-f at v2:172.21.15.107:3300/0 2023-09-30T01:35:05.281 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:05.271+0000 7fb69bfff700 1 -- 172.21.15.88:0/285696263 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6a4097e10 con 0x7fb6a41311c0 2023-09-30T01:35:05.281 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:05.271+0000 7fb69bfff700 1 --2- 172.21.15.88:0/285696263 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb6a41311c0 0x7fb6a4132c40 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fb6940099b0 tx=0x7fb694009250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e77906add76d06c server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:35:05.281 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:05.271+0000 7fb6a27fc700 1 -- 172.21.15.88:0/285696263 <== mon.1 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7fb694009ea0 con 0x7fb6a41311c0 2023-09-30T01:35:05.281 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:05.271+0000 7fb6a27fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:35:05.281 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:05.271+0000 7fb6a27fc700 10 monclient: got monmap 1 from mon.noname-f (according to old e1) 2023-09-30T01:35:05.281 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:05.271+0000 7fb6a27fc700 10 monclient: dump: 2023-09-30T01:35:05.281 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:35:05.281 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:35:05.281 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:35:05.281 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:35:05.282 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:35:05.282 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:35:05.282 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:35:05.282 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:35:05.282 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:35:05.282 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:35:05.282 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:35:05.282 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:35:05.282 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:35:05.282 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:35:05.282 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:35:05.282 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:35:05.283 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:05.271+0000 7fb6a27fc700 10 monclient: _finish_auth 0 2023-09-30T01:35:05.283 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:05.271+0000 7fb6a27fc700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:35:05.283 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:05.271+0000 7fb6a27fc700 1 -- 172.21.15.88:0/285696263 <== mon.1 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb69400bce0 con 0x7fb6a41311c0 2023-09-30T01:35:05.283 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:05.271+0000 7fb6a3fff700 5 monclient: authenticate success, global_id 4762 2023-09-30T01:35:05.283 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:05.271+0000 7fb6a3fff700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:35:05.283 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:05.271+0000 7fb6a27fc700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:35:05.283 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:05.271+0000 7fb6a3fff700 10 monclient: get_monmap_and_config success 2023-09-30T01:35:05.283 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:05.271+0000 7fb6a3fff700 4 set_mon_vals no callback set 2023-09-30T01:35:05.283 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:05.271+0000 7fb6a27fc700 1 -- 172.21.15.88:0/285696263 <== mon.1 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7fb69400be50 con 0x7fb6a41311c0 2023-09-30T01:35:05.283 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:05.271+0000 7fb6a27fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:35:05.283 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:05.271+0000 7fb6a27fc700 10 monclient: got monmap 1 from mon.f (according to old e1) 2023-09-30T01:35:05.283 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:05.271+0000 7fb6a27fc700 10 monclient: dump: 2023-09-30T01:35:05.284 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:35:05.284 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:35:05.284 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:35:05.284 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:35:05.284 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:35:05.284 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:35:05.284 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:35:05.284 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:35:05.284 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:35:05.284 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:35:05.284 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:35:05.284 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:35:05.285 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:35:05.285 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:35:05.285 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:35:05.285 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:35:05.285 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:05.275+0000 7fb6a3fff700 10 monclient: shutdown 2023-09-30T01:35:05.285 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:05.275+0000 7fb6a3fff700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:35:05.285 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:05.275+0000 7fb6a3fff700 1 -- 172.21.15.88:0/285696263 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb6a41311c0 msgr2=0x7fb6a4132c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:35:05.285 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:05.275+0000 7fb6a3fff700 1 --2- 172.21.15.88:0/285696263 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb6a41311c0 0x7fb6a4132c40 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fb6940099b0 tx=0x7fb694009250 comp rx=0 tx=0).stop 2023-09-30T01:35:05.285 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:05.275+0000 7fb6a3fff700 1 -- 172.21.15.88:0/285696263 shutdown_connections 2023-09-30T01:35:05.285 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:05.275+0000 7fb6a3fff700 1 --2- 172.21.15.88:0/285696263 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb6a41311c0 0x7fb6a4132c40 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:35:05.285 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:05.275+0000 7fb6a3fff700 1 --2- 172.21.15.88:0/285696263 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7fb6a4097a40 0x7fb6a4132700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:35:05.285 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:05.275+0000 7fb6a3fff700 1 -- 172.21.15.88:0/285696263 >> 172.21.15.88:0/285696263 conn(0x7fb6a406e5e0 msgr2=0x7fb6a406e9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:35:05.285 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:05.275+0000 7fb6a3fff700 1 -- 172.21.15.88:0/285696263 shutdown_connections 2023-09-30T01:35:05.286 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:05.275+0000 7fb6a3fff700 1 -- 172.21.15.88:0/285696263 wait complete. 2023-09-30T01:35:05.286 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:05.275+0000 7fb6a3fff700 10 monclient: build_initial_monmap 2023-09-30T01:35:05.286 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:05.275+0000 7fb6a3fff700 10 monclient: monmap: 2023-09-30T01:35:05.286 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2023-09-30T01:35:05.286 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:35:05.286 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 0.000000 2023-09-30T01:35:05.286 INFO:teuthology.orchestra.run.smithi088.stderr:created 0.000000 2023-09-30T01:35:05.286 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2023-09-30T01:35:05.286 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 1 2023-09-30T01:35:05.286 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.noname-a 2023-09-30T01:35:05.286 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.noname-f 2023-09-30T01:35:05.286 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.noname-b 2023-09-30T01:35:05.287 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.noname-g 2023-09-30T01:35:05.287 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.noname-c 2023-09-30T01:35:05.287 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.noname-h 2023-09-30T01:35:05.287 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.noname-d 2023-09-30T01:35:05.287 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.noname-i 2023-09-30T01:35:05.287 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.noname-e 2023-09-30T01:35:05.287 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:35:05.291 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:05.275+0000 7fb6a3fff700 1 Processor -- start 2023-09-30T01:35:05.291 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:05.275+0000 7fb6a3fff700 1 -- start start 2023-09-30T01:35:05.291 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:05.275+0000 7fb6a3fff700 10 monclient: init 2023-09-30T01:35:05.291 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:05.279+0000 7fb6a3fff700 10 monclient: _reopen_session rank -1 2023-09-30T01:35:05.291 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:05.279+0000 7fb6a3fff700 10 monclient: _add_conns ranks=[1,0,4,5,7,8,6,2,3] 2023-09-30T01:35:05.291 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:05.279+0000 7fb6a3fff700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb6a4097a40 0x7fb6a411cd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:35:05.291 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:05.279+0000 7fb6a3fff700 10 monclient(hunting): picked mon.noname-f con 0x7fb6a4097a40 addr [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] 2023-09-30T01:35:05.292 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:05.279+0000 7fb6a3fff700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb6a41311c0 0x7fb6a411d290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:35:05.292 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:05.279+0000 7fb6a3fff700 10 monclient(hunting): picked mon.noname-a con 0x7fb6a41311c0 addr [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] 2023-09-30T01:35:05.292 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:05.279+0000 7fb6a3fff700 1 --2- >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7fb6a41332e0 0x7fb6a411d7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:35:05.292 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:05.279+0000 7fb6a3fff700 10 monclient(hunting): picked mon.noname-c con 0x7fb6a41332e0 addr [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] 2023-09-30T01:35:05.292 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:05.279+0000 7fb6a3fff700 10 monclient(hunting): start opening mon connection 2023-09-30T01:35:05.292 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:05.279+0000 7fb6a3fff700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6a40996b0 con 0x7fb6a41311c0 2023-09-30T01:35:05.292 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:05.279+0000 7fb6a3fff700 10 monclient(hunting): start opening mon connection 2023-09-30T01:35:05.292 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:05.279+0000 7fb6a3fff700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6a411f4a0 con 0x7fb6a4097a40 2023-09-30T01:35:05.292 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:05.279+0000 7fb6a3fff700 10 monclient(hunting): start opening mon connection 2023-09-30T01:35:05.292 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:05.279+0000 7fb6a3fff700 1 -- --> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] -- mon_getmap magic: 0 v1 -- 0x7fb6a411ee50 con 0x7fb6a41332e0 2023-09-30T01:35:05.292 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:05.279+0000 7fb6a3fff700 10 monclient(hunting): _renew_subs 2023-09-30T01:35:05.293 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:05.279+0000 7fb6a3fff700 10 monclient(hunting): authenticate will time out at 977.316546s 2023-09-30T01:35:05.293 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:05.279+0000 7fb6a37fe700 1 --2- >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7fb6a41332e0 0x7fb6a411d7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:35:05.293 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:05.279+0000 7fb6a37fe700 1 --2- >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7fb6a41332e0 0x7fb6a411d7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3302/0 says I am v2:172.21.15.88:33300/0 (socket says 172.21.15.88:33300) 2023-09-30T01:35:05.293 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:05.279+0000 7fb6a37fe700 1 -- 172.21.15.88:0/115070882 learned_addr learned my addr 172.21.15.88:0/115070882 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-09-30T01:35:05.293 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:05.279+0000 7fb6a37fe700 10 monclient(hunting): get_auth_request con 0x7fb6a41332e0 auth_method 0 2023-09-30T01:35:05.293 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:05.279+0000 7fb6a37fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:35:05.293 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:05.279+0000 7fb6a37fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:35:05.293 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:05.279+0000 7fb6a37fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:35:05.293 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:05.279+0000 7fb69bfff700 1 --2- 172.21.15.88:0/115070882 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb6a41311c0 0x7fb6a411d290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:35:05.293 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:05.279+0000 7fb69bfff700 10 monclient(hunting): get_auth_request con 0x7fb6a41311c0 auth_method 0 2023-09-30T01:35:05.293 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:05.279+0000 7fb69bfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:35:05.293 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:05.279+0000 7fb69bfff700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:35:05.293 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:05.279+0000 7fb69bfff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:35:05.294 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:05.279+0000 7fb6a37fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:35:05.294 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:05.279+0000 7fb6a37fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:35:05.294 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:05.279+0000 7fb6a2ffd700 1 --2- 172.21.15.88:0/115070882 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb6a4097a40 0x7fb6a411cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:35:05.294 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:05.279+0000 7fb6a37fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:35:05.294 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:05.279+0000 7fb6a2ffd700 10 monclient(hunting): get_auth_request con 0x7fb6a4097a40 auth_method 0 2023-09-30T01:35:05.294 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:05.279+0000 7fb6a2ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:35:05.294 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:05.279+0000 7fb6a2ffd700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:35:05.294 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:05.279+0000 7fb6a2ffd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:35:05.294 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:05.279+0000 7fb69bfff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:35:05.294 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:05.279+0000 7fb69bfff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:35:05.294 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:05.279+0000 7fb69bfff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:35:05.294 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:05.279+0000 7fb6a37fe700 10 monclient(hunting): handle_auth_done global_id 4576 payload 824 2023-09-30T01:35:05.295 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:05.279+0000 7fb6a37fe700 1 -- 172.21.15.88:0/115070882 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb6a4097a40 msgr2=0x7fb6a411cd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:35:05.295 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:05.279+0000 7fb6a37fe700 1 --2- 172.21.15.88:0/115070882 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb6a4097a40 0x7fb6a411cd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:35:05.295 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:05.279+0000 7fb6a37fe700 1 -- 172.21.15.88:0/115070882 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb6a41311c0 msgr2=0x7fb6a411d290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:35:05.295 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:05.279+0000 7fb6a37fe700 1 --2- 172.21.15.88:0/115070882 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb6a41311c0 0x7fb6a411d290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:35:05.295 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:05.279+0000 7fb6a37fe700 10 monclient: _finish_hunting 0 2023-09-30T01:35:05.295 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:05.279+0000 7fb6a37fe700 1 monclient: found mon.noname-c 2023-09-30T01:35:05.295 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:05.279+0000 7fb6a37fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:35:05.295 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:05.279+0000 7fb6a37fe700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.88:3302/0 2023-09-30T01:35:05.295 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:05.279+0000 7fb6a37fe700 1 -- 172.21.15.88:0/115070882 --> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6940094f0 con 0x7fb6a41332e0 2023-09-30T01:35:05.295 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:05.279+0000 7fb6a2ffd700 1 --2- 172.21.15.88:0/115070882 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb6a4097a40 0x7fb6a411cd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-30T01:35:05.295 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:05.279+0000 7fb6a37fe700 1 --2- 172.21.15.88:0/115070882 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7fb6a41332e0 0x7fb6a411d7d0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fb69c00cca0 tx=0x7fb69c001a70 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:35:05.295 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:05.279+0000 7fb69affd700 1 -- 172.21.15.88:0/115070882 <== mon.4 v2:172.21.15.88:3302/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7fb69c005df0 con 0x7fb6a41332e0 2023-09-30T01:35:05.296 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:05.279+0000 7fb69affd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:35:05.296 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:05.279+0000 7fb69affd700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T01:35:05.296 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:05.279+0000 7fb69affd700 10 monclient: dump: 2023-09-30T01:35:05.296 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:35:05.296 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:35:05.296 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:35:05.296 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:35:05.296 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:35:05.296 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:35:05.296 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:35:05.296 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:35:05.296 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:35:05.297 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:35:05.297 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:35:05.297 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:35:05.297 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:35:05.297 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:35:05.297 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:35:05.297 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:35:05.297 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:05.279+0000 7fb69affd700 10 monclient: _finish_auth 0 2023-09-30T01:35:05.297 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:05.279+0000 7fb69affd700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:35:05.297 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:05.279+0000 7fb6a3fff700 5 monclient: authenticate success, global_id 4576 2023-09-30T01:35:05.297 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:05.279+0000 7fb69affd700 1 -- 172.21.15.88:0/115070882 <== mon.4 v2:172.21.15.88:3302/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb69c00ab20 con 0x7fb6a41332e0 2023-09-30T01:35:05.297 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:05.279+0000 7fb6a3fff700 10 monclient: _renew_subs 2023-09-30T01:35:05.297 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:05.279+0000 7fb6a3fff700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.88:3302/0 2023-09-30T01:35:05.298 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:05.279+0000 7fb6a3fff700 1 -- 172.21.15.88:0/115070882 --> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb6a411f130 con 0x7fb6a41332e0 2023-09-30T01:35:05.298 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:05.279+0000 7fb69affd700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:35:05.298 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:05.279+0000 7fb69affd700 1 -- 172.21.15.88:0/115070882 <== mon.4 v2:172.21.15.88:3302/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7fb69c00ac90 con 0x7fb6a41332e0 2023-09-30T01:35:05.298 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:05.279+0000 7fb69affd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:35:05.298 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:05.279+0000 7fb69affd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T01:35:05.298 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:05.279+0000 7fb6a1ffb700 4 set_mon_vals no callback set 2023-09-30T01:35:05.298 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:05.279+0000 7fb69affd700 10 monclient: dump: 2023-09-30T01:35:05.298 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:35:05.298 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:35:05.298 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:35:05.298 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:35:05.298 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:35:05.299 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:35:05.299 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:35:05.299 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:35:05.299 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:35:05.299 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:35:05.299 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:35:05.299 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:35:05.299 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:35:05.299 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:35:05.299 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:35:05.299 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:35:05.299 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:05.279+0000 7fb6a3fff700 10 monclient: _renew_subs 2023-09-30T01:35:05.299 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:05.279+0000 7fb6a3fff700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.88:3302/0 2023-09-30T01:35:05.300 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:05.279+0000 7fb6a3fff700 1 -- 172.21.15.88:0/115070882 --> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb6a411df60 con 0x7fb6a41332e0 2023-09-30T01:35:05.300 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:05.279+0000 7fb69affd700 1 -- 172.21.15.88:0/115070882 <== mon.4 v2:172.21.15.88:3302/0 4 ==== mgrmap(e 5) v1 ==== 40144+0+0 (secure 0 0 0) 0x7fb69c00a6a0 con 0x7fb6a41332e0 2023-09-30T01:35:05.300 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:05.279+0000 7fb6a3fff700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:35:05.300 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:05.283+0000 7fb6a3fff700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:35:05.300 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:05.283+0000 7fb6a3fff700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.88:3302/0 2023-09-30T01:35:05.300 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:05.283+0000 7fb6a3fff700 1 -- 172.21.15.88:0/115070882 --> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb6a4133ff0 con 0x7fb6a41332e0 2023-09-30T01:35:05.300 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:05.283+0000 7fb69affd700 1 --2- 172.21.15.88:0/115070882 >> v2:172.21.15.107:6812/29926 conn(0x7fb684033360 0x7fb684035810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:35:05.300 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:05.283+0000 7fb69affd700 1 -- 172.21.15.88:0/115070882 <== mon.4 v2:172.21.15.88:3302/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 3502+0+0 (secure 0 0 0) 0x7fb69c043470 con 0x7fb6a41332e0 2023-09-30T01:35:05.300 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:05.283+0000 7fb6a2ffd700 1 --2- 172.21.15.88:0/115070882 >> v2:172.21.15.107:6812/29926 conn(0x7fb684033360 0x7fb684035810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:35:05.300 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:05.283+0000 7fb6a2ffd700 10 monclient: get_auth_request con 0x7fb684033360 auth_method 0 2023-09-30T01:35:05.300 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:05.283+0000 7fb6a2ffd700 1 --2- 172.21.15.88:0/115070882 >> v2:172.21.15.107:6812/29926 conn(0x7fb684033360 0x7fb684035810 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fb68c009d80 tx=0x7fb68c008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:35:05.301 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:05.291+0000 7fb69affd700 1 -- 172.21.15.88:0/115070882 <== mon.4 v2:172.21.15.88:3302/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fb69c00a950 con 0x7fb6a41332e0 2023-09-30T01:35:05.301 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:05.291+0000 7fb69affd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:35:05.301 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:05.291+0000 7fb69affd700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:35:05.480 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:05.479+0000 7fb6a3fff700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T01:35:05.481 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:05.479+0000 7fb6a3fff700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:35:05.481 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:05.479+0000 7fb6a3fff700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.88:3302/0 2023-09-30T01:35:05.481 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:05.479+0000 7fb6a3fff700 1 -- 172.21.15.88:0/115070882 --> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fb6a4071960 con 0x7fb6a41332e0 2023-09-30T01:35:05.481 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:05.479+0000 7fb69affd700 1 -- 172.21.15.88:0/115070882 <== mon.4 v2:172.21.15.88:3302/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7fb69c010af0 con 0x7fb6a41332e0 2023-09-30T01:35:05.481 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:05.479+0000 7fb69affd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:35:05.481 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:05.479+0000 7fb69affd700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:35:05.482 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-09-30T01:35:05.482 INFO:teuthology.orchestra.run.smithi088.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:35:05.485 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:05.483+0000 7fb6a3fff700 1 -- 172.21.15.88:0/115070882 >> v2:172.21.15.107:6812/29926 conn(0x7fb684033360 msgr2=0x7fb684035810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:35:05.485 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:05.483+0000 7fb6a3fff700 1 --2- 172.21.15.88:0/115070882 >> v2:172.21.15.107:6812/29926 conn(0x7fb684033360 0x7fb684035810 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fb68c009d80 tx=0x7fb68c008040 comp rx=0 tx=0).stop 2023-09-30T01:35:05.485 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:05.483+0000 7fb6a3fff700 10 monclient: shutdown 2023-09-30T01:35:05.485 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:05.483+0000 7fb6a3fff700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:35:05.485 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:05.483+0000 7fb6a3fff700 1 -- 172.21.15.88:0/115070882 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7fb6a41332e0 msgr2=0x7fb6a411d7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:35:05.485 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:05.483+0000 7fb6a3fff700 1 --2- 172.21.15.88:0/115070882 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7fb6a41332e0 0x7fb6a411d7d0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fb69c00cca0 tx=0x7fb69c001a70 comp rx=0 tx=0).stop 2023-09-30T01:35:05.486 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:05.483+0000 7fb6a3fff700 1 -- 172.21.15.88:0/115070882 shutdown_connections 2023-09-30T01:35:05.486 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:05.483+0000 7fb6a3fff700 1 --2- 172.21.15.88:0/115070882 >> v2:172.21.15.107:6812/29926 conn(0x7fb684033360 0x7fb684035810 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:35:05.486 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:05.483+0000 7fb6a3fff700 1 --2- 172.21.15.88:0/115070882 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7fb6a41332e0 0x7fb6a411d7d0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:35:05.486 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:05.483+0000 7fb6a3fff700 1 --2- 172.21.15.88:0/115070882 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb6a41311c0 0x7fb6a411d290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:35:05.486 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:05.483+0000 7fb6a3fff700 1 --2- 172.21.15.88:0/115070882 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb6a4097a40 0x7fb6a411cd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:35:05.486 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:05.483+0000 7fb6a3fff700 1 -- 172.21.15.88:0/115070882 >> 172.21.15.88:0/115070882 conn(0x7fb6a406e5e0 msgr2=0x7fb6a4131e60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:35:05.486 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:05.483+0000 7fb6a3fff700 1 -- 172.21.15.88:0/115070882 shutdown_connections 2023-09-30T01:35:05.486 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:05.483+0000 7fb6a3fff700 1 -- 172.21.15.88:0/115070882 wait complete. 2023-09-30T01:35:08.502 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:35:08.611 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:08.607+0000 7f3157570700 10 monclient: get_monmap_and_config 2023-09-30T01:35:08.611 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:08.607+0000 7f3157570700 10 monclient: build_initial_monmap 2023-09-30T01:35:08.611 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:08.607+0000 7f3157570700 10 monclient: monmap: 2023-09-30T01:35:08.611 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2023-09-30T01:35:08.611 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:35:08.611 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:35:08.611271+0000 2023-09-30T01:35:08.611 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:35:08.611271+0000 2023-09-30T01:35:08.611 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2023-09-30T01:35:08.612 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:35:08.612 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a-legacy 2023-09-30T01:35:08.612 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.107:6789/0 mon.noname-f-legacy 2023-09-30T01:35:08.612 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.noname-b 2023-09-30T01:35:08.612 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.noname-g 2023-09-30T01:35:08.612 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.noname-c 2023-09-30T01:35:08.612 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.noname-h 2023-09-30T01:35:08.612 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.noname-d 2023-09-30T01:35:08.612 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.noname-i 2023-09-30T01:35:08.612 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.noname-e 2023-09-30T01:35:08.612 INFO:teuthology.orchestra.run.smithi088.stderr:9: v2:172.21.15.88:3300/0 mon.noname-a 2023-09-30T01:35:08.613 INFO:teuthology.orchestra.run.smithi088.stderr:10: v2:172.21.15.107:3300/0 mon.noname-f 2023-09-30T01:35:08.613 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:35:08.618 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:08.611+0000 7f3157570700 1 Processor -- start 2023-09-30T01:35:08.618 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:08.611+0000 7f3157570700 1 -- start start 2023-09-30T01:35:08.619 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:08.611+0000 7f3157570700 10 monclient: init 2023-09-30T01:35:08.619 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:08.611+0000 7f3157570700 10 monclient: _reopen_session rank -1 2023-09-30T01:35:08.619 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:08.611+0000 7f3157570700 10 monclient: _add_conns ranks=[0,1,8,5,9,6,2,10,7,4,3] 2023-09-30T01:35:08.619 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:08.611+0000 7f3157570700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f315009ed80 addr v1:172.21.15.88:6789/0 2023-09-30T01:35:08.619 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:08.611+0000 7f3157570700 10 monclient(hunting): picked mon.noname-f-legacy con 0x7f315009f860 addr v1:172.21.15.107:6789/0 2023-09-30T01:35:08.619 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:08.611+0000 7f3157570700 1 --2- >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f31500637f0 0x7f3150139ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:35:08.619 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:08.611+0000 7f3157570700 10 monclient(hunting): picked mon.noname-e con 0x7f31500637f0 addr [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] 2023-09-30T01:35:08.619 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:08.611+0000 7f3157570700 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f315004bb20 con 0x7f315009ed80 2023-09-30T01:35:08.619 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:08.611+0000 7f3157570700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f315009d830 con 0x7f315009f860 2023-09-30T01:35:08.619 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:08.611+0000 7f3157570700 10 monclient(hunting): start opening mon connection 2023-09-30T01:35:08.620 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:08.611+0000 7f3157570700 1 -- --> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] -- mon_getmap magic: 0 v1 -- 0x7f315009a110 con 0x7f31500637f0 2023-09-30T01:35:08.620 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:08.611+0000 7f3157570700 10 monclient(hunting): _renew_subs 2023-09-30T01:35:08.620 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:08.611+0000 7f3157570700 10 monclient(hunting): authenticate will time out at 980.649353s 2023-09-30T01:35:08.620 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:08.611+0000 7f3156d6f700 1 --2- >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f31500637f0 0x7f3150139ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:35:08.620 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:08.611+0000 7f3156d6f700 1 --2- >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f31500637f0 0x7f3150139ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3304/0 says I am v2:172.21.15.88:56352/0 (socket says 172.21.15.88:56352) 2023-09-30T01:35:08.620 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:08.611+0000 7f3156d6f700 1 -- 172.21.15.88:0/1014491099 learned_addr learned my addr 172.21.15.88:0/1014491099 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-09-30T01:35:08.620 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:08.611+0000 7f3156d6f700 10 monclient(hunting): get_auth_request con 0x7f31500637f0 auth_method 0 2023-09-30T01:35:08.620 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:08.611+0000 7f3156d6f700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:35:08.620 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:08.611+0000 7f3156d6f700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:35:08.620 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:08.611+0000 7f315656e700 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f315009ed80 0x7f315009f150 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:33222/0 (socket says 172.21.15.88:33222) 2023-09-30T01:35:08.620 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:08.611+0000 7f3156d6f700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:35:08.620 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:08.611+0000 7f3156d6f700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:35:08.621 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:08.611+0000 7f3156d6f700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:35:08.621 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:08.611+0000 7f3156d6f700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:35:08.621 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:08.611+0000 7f3156d6f700 10 monclient(hunting): handle_auth_done global_id 4616 payload 274 2023-09-30T01:35:08.621 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:08.611+0000 7f3156d6f700 1 -- 172.21.15.88:0/1014491099 >> v1:172.21.15.107:6789/0 conn(0x7f315009f860 legacy=0x7f31501367c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:35:08.621 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:08.611+0000 7f3156d6f700 1 -- 172.21.15.88:0/1014491099 >> v1:172.21.15.88:6789/0 conn(0x7f315009ed80 legacy=0x7f315009f150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:35:08.621 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:08.611+0000 7f3156d6f700 10 monclient: _finish_hunting 0 2023-09-30T01:35:08.621 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:08.611+0000 7f3156d6f700 1 monclient: found mon.noname-e 2023-09-30T01:35:08.621 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:08.611+0000 7f3156d6f700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:35:08.621 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:08.611+0000 7f3156d6f700 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.88:3304/0 2023-09-30T01:35:08.621 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:08.611+0000 7f3156d6f700 1 -- 172.21.15.88:0/1014491099 --> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3150063440 con 0x7f31500637f0 2023-09-30T01:35:08.621 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:08.611+0000 7f3156d6f700 1 --2- 172.21.15.88:0/1014491099 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f31500637f0 0x7f3150139ef0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f3148009f50 tx=0x7f3148009840 comp rx=0 tx=0).ready entity=mon.8 client_cookie=61635791fc88b258 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:35:08.622 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:08.611+0000 7f315556c700 1 -- 172.21.15.88:0/1014491099 <== mon.8 v2:172.21.15.88:3304/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f314800e070 con 0x7f31500637f0 2023-09-30T01:35:08.622 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:08.611+0000 7f315556c700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:35:08.622 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:08.611+0000 7f315556c700 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2023-09-30T01:35:08.622 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:08.611+0000 7f315556c700 10 monclient: dump: 2023-09-30T01:35:08.622 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:35:08.622 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:35:08.622 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:35:08.622 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:35:08.622 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:35:08.622 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:35:08.622 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:35:08.623 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:35:08.623 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:35:08.623 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:35:08.623 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:35:08.623 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:35:08.623 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:35:08.623 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:35:08.623 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:35:08.623 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:35:08.623 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:08.611+0000 7f315556c700 10 monclient: _finish_auth 0 2023-09-30T01:35:08.623 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:08.611+0000 7f315556c700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:35:08.623 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:08.611+0000 7f3157570700 5 monclient: authenticate success, global_id 4616 2023-09-30T01:35:08.623 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:08.611+0000 7f315556c700 1 -- 172.21.15.88:0/1014491099 <== mon.8 v2:172.21.15.88:3304/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3148007400 con 0x7f31500637f0 2023-09-30T01:35:08.624 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:08.611+0000 7f3157570700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:35:08.624 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:08.611+0000 7f315556c700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:35:08.624 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:08.611+0000 7f315556c700 1 -- 172.21.15.88:0/1014491099 <== mon.8 v2:172.21.15.88:3304/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f31480092e0 con 0x7f31500637f0 2023-09-30T01:35:08.624 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:08.611+0000 7f3157570700 10 monclient: get_monmap_and_config success 2023-09-30T01:35:08.624 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:08.611+0000 7f3157570700 4 set_mon_vals no callback set 2023-09-30T01:35:08.624 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:08.611+0000 7f315556c700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:35:08.624 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:08.611+0000 7f315556c700 10 monclient: got monmap 1 from mon.e (according to old e1) 2023-09-30T01:35:08.624 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:08.611+0000 7f315556c700 10 monclient: dump: 2023-09-30T01:35:08.624 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:35:08.624 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:35:08.624 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:35:08.624 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:35:08.625 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:35:08.625 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:35:08.625 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:35:08.625 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:35:08.625 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:35:08.625 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:35:08.625 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:35:08.625 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:35:08.625 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:35:08.625 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:35:08.625 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:35:08.625 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:35:08.626 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:08.611+0000 7f3157570700 10 monclient: shutdown 2023-09-30T01:35:08.626 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:08.611+0000 7f3157570700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:35:08.626 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:08.611+0000 7f3157570700 1 -- 172.21.15.88:0/1014491099 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f31500637f0 msgr2=0x7f3150139ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:35:08.626 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:08.611+0000 7f3157570700 1 --2- 172.21.15.88:0/1014491099 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f31500637f0 0x7f3150139ef0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f3148009f50 tx=0x7f3148009840 comp rx=0 tx=0).stop 2023-09-30T01:35:08.626 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:08.611+0000 7f3157570700 1 -- 172.21.15.88:0/1014491099 shutdown_connections 2023-09-30T01:35:08.626 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:08.611+0000 7f3157570700 1 --2- 172.21.15.88:0/1014491099 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f31500637f0 0x7f3150139ef0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:35:08.626 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:08.611+0000 7f3157570700 1 -- 172.21.15.88:0/1014491099 >> 172.21.15.88:0/1014491099 conn(0x7f31500a0910 msgr2=0x7f31500a33c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:35:08.626 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:08.611+0000 7f3157570700 1 -- 172.21.15.88:0/1014491099 shutdown_connections 2023-09-30T01:35:08.626 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:08.611+0000 7f3157570700 1 -- 172.21.15.88:0/1014491099 wait complete. 2023-09-30T01:35:08.626 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:08.615+0000 7f3157570700 10 monclient: build_initial_monmap 2023-09-30T01:35:08.626 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:08.615+0000 7f3157570700 10 monclient: monmap: 2023-09-30T01:35:08.627 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2023-09-30T01:35:08.627 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:35:08.627 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 0.000000 2023-09-30T01:35:08.627 INFO:teuthology.orchestra.run.smithi088.stderr:created 0.000000 2023-09-30T01:35:08.627 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2023-09-30T01:35:08.627 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 1 2023-09-30T01:35:08.627 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.noname-a 2023-09-30T01:35:08.627 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.noname-f 2023-09-30T01:35:08.627 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.noname-b 2023-09-30T01:35:08.627 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.noname-g 2023-09-30T01:35:08.627 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.noname-c 2023-09-30T01:35:08.627 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.noname-h 2023-09-30T01:35:08.627 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.noname-d 2023-09-30T01:35:08.628 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.noname-i 2023-09-30T01:35:08.628 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.noname-e 2023-09-30T01:35:08.628 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:35:08.628 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:08.615+0000 7f3157570700 1 Processor -- start 2023-09-30T01:35:08.628 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:08.615+0000 7f3157570700 1 -- start start 2023-09-30T01:35:08.628 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:08.615+0000 7f3157570700 10 monclient: init 2023-09-30T01:35:08.628 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:08.615+0000 7f3157570700 10 monclient: _reopen_session rank -1 2023-09-30T01:35:08.628 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:08.615+0000 7f3157570700 10 monclient: _add_conns ranks=[5,2,8,3,4,6,1,0,7] 2023-09-30T01:35:08.628 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:08.615+0000 7f3157570700 1 --2- >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7f31500637f0 0x7f3150140670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:35:08.628 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:08.615+0000 7f3157570700 10 monclient(hunting): picked mon.noname-h con 0x7f31500637f0 addr [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] 2023-09-30T01:35:08.628 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:08.615+0000 7f3157570700 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f315009ed80 0x7f3150140bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:35:08.628 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:08.615+0000 7f3157570700 10 monclient(hunting): picked mon.noname-b con 0x7f315009ed80 addr [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] 2023-09-30T01:35:08.629 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:08.615+0000 7f3157570700 1 --2- >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f315009f860 0x7f31501410f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:35:08.629 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:08.615+0000 7f3157570700 10 monclient(hunting): picked mon.noname-e con 0x7f315009f860 addr [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] 2023-09-30T01:35:08.629 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:08.615+0000 7f3157570700 10 monclient(hunting): start opening mon connection 2023-09-30T01:35:08.629 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:08.615+0000 7f3157570700 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f315009ccd0 con 0x7f315009ed80 2023-09-30T01:35:08.629 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:08.615+0000 7f3157570700 10 monclient(hunting): start opening mon connection 2023-09-30T01:35:08.629 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:08.615+0000 7f3157570700 1 -- --> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] -- mon_getmap magic: 0 v1 -- 0x7f31500970b0 con 0x7f31500637f0 2023-09-30T01:35:08.629 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:08.615+0000 7f3157570700 10 monclient(hunting): start opening mon connection 2023-09-30T01:35:08.629 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:08.615+0000 7f3157570700 1 -- --> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] -- mon_getmap magic: 0 v1 -- 0x7f3150090250 con 0x7f315009f860 2023-09-30T01:35:08.629 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:08.615+0000 7f3157570700 10 monclient(hunting): _renew_subs 2023-09-30T01:35:08.629 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:08.615+0000 7f3157570700 10 monclient(hunting): authenticate will time out at 980.653423s 2023-09-30T01:35:08.629 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:08.615+0000 7f3155d6d700 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f315009ed80 0x7f3150140bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:35:08.629 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:08.615+0000 7f3156d6f700 1 --2- >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f315009f860 0x7f31501410f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:35:08.630 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:08.615+0000 7f3155d6d700 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f315009ed80 0x7f3150140bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3301/0 says I am v2:172.21.15.88:36616/0 (socket says 172.21.15.88:36616) 2023-09-30T01:35:08.630 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:08.615+0000 7f3156d6f700 1 --2- >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f315009f860 0x7f31501410f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3304/0 says I am v2:172.21.15.88:56368/0 (socket says 172.21.15.88:56368) 2023-09-30T01:35:08.630 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:08.615+0000 7f3155d6d700 1 -- 172.21.15.88:0/3129039132 learned_addr learned my addr 172.21.15.88:0/3129039132 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-09-30T01:35:08.630 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:08.615+0000 7f3155d6d700 10 monclient(hunting): get_auth_request con 0x7f315009ed80 auth_method 0 2023-09-30T01:35:08.630 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:08.615+0000 7f3155d6d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:35:08.630 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:08.615+0000 7f3155d6d700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:35:08.630 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:08.615+0000 7f3155d6d700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:35:08.630 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:08.615+0000 7f3156d6f700 10 monclient(hunting): get_auth_request con 0x7f315009f860 auth_method 0 2023-09-30T01:35:08.630 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:08.615+0000 7f3156d6f700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:35:08.630 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:08.615+0000 7f3156d6f700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:35:08.630 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:08.615+0000 7f3156d6f700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:35:08.630 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:08.615+0000 7f315656e700 1 --2- 172.21.15.88:0/3129039132 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7f31500637f0 0x7f3150140670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:35:08.631 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:08.615+0000 7f315656e700 10 monclient(hunting): get_auth_request con 0x7f31500637f0 auth_method 0 2023-09-30T01:35:08.631 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:08.615+0000 7f315656e700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:35:08.631 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:08.615+0000 7f315656e700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:35:08.631 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:08.615+0000 7f315656e700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:35:08.631 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:08.615+0000 7f3155d6d700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:35:08.631 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:08.615+0000 7f3155d6d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:35:08.631 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:08.615+0000 7f3155d6d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:35:08.631 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:08.615+0000 7f3156d6f700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:35:08.631 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:08.615+0000 7f3156d6f700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:35:08.631 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:08.615+0000 7f3156d6f700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:35:08.631 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:08.615+0000 7f315656e700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:35:08.631 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:08.615+0000 7f315656e700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:35:08.632 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:08.615+0000 7f315656e700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:35:08.632 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:08.615+0000 7f3155d6d700 10 monclient(hunting): handle_auth_done global_id 4538 payload 824 2023-09-30T01:35:08.632 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:08.615+0000 7f3155d6d700 1 -- 172.21.15.88:0/3129039132 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f315009f860 msgr2=0x7f31501410f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:35:08.632 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:08.615+0000 7f3155d6d700 1 --2- 172.21.15.88:0/3129039132 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f315009f860 0x7f31501410f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:35:08.632 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:08.615+0000 7f3155d6d700 1 -- 172.21.15.88:0/3129039132 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7f31500637f0 msgr2=0x7f3150140670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:35:08.632 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:08.615+0000 7f3155d6d700 1 --2- 172.21.15.88:0/3129039132 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7f31500637f0 0x7f3150140670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:35:08.632 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:08.615+0000 7f3155d6d700 10 monclient: _finish_hunting 0 2023-09-30T01:35:08.632 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:08.615+0000 7f3155d6d700 1 monclient: found mon.noname-b 2023-09-30T01:35:08.632 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:08.615+0000 7f3155d6d700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:35:08.632 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:08.615+0000 7f3155d6d700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.88:3301/0 2023-09-30T01:35:08.632 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:08.615+0000 7f3155d6d700 1 -- 172.21.15.88:0/3129039132 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3148009ae0 con 0x7f315009ed80 2023-09-30T01:35:08.633 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:08.615+0000 7f3156d6f700 1 --2- 172.21.15.88:0/3129039132 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f315009f860 0x7f31501410f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:35:08.633 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:08.615+0000 7f3155d6d700 1 --2- 172.21.15.88:0/3129039132 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f315009ed80 0x7f3150140bb0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f314400c4e0 tx=0x7f314400c7f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:35:08.633 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:08.615+0000 7f31427fc700 1 -- 172.21.15.88:0/3129039132 <== mon.2 v2:172.21.15.88:3301/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f3144001080 con 0x7f315009ed80 2023-09-30T01:35:08.633 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:08.615+0000 7f31427fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:35:08.633 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:08.615+0000 7f31427fc700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T01:35:08.633 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:08.615+0000 7f31427fc700 10 monclient: dump: 2023-09-30T01:35:08.633 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:35:08.633 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:35:08.633 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:35:08.634 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:35:08.634 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:35:08.634 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:35:08.634 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:35:08.634 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:35:08.634 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:35:08.634 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:35:08.634 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:35:08.634 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:35:08.634 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:35:08.634 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:35:08.634 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:35:08.635 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:35:08.635 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:08.615+0000 7f31427fc700 10 monclient: _finish_auth 0 2023-09-30T01:35:08.635 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:08.615+0000 7f31427fc700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:35:08.635 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:08.615+0000 7f31427fc700 1 -- 172.21.15.88:0/3129039132 <== mon.2 v2:172.21.15.88:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3144011a10 con 0x7f315009ed80 2023-09-30T01:35:08.635 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:08.615+0000 7f3157570700 5 monclient: authenticate success, global_id 4538 2023-09-30T01:35:08.635 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:08.615+0000 7f3157570700 10 monclient: _renew_subs 2023-09-30T01:35:08.635 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:08.615+0000 7f3157570700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.88:3301/0 2023-09-30T01:35:08.635 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:08.615+0000 7f3157570700 1 -- 172.21.15.88:0/3129039132 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3150091320 con 0x7f315009ed80 2023-09-30T01:35:08.635 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:08.615+0000 7f31427fc700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:35:08.635 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:08.615+0000 7f31427fc700 1 -- 172.21.15.88:0/3129039132 <== mon.2 v2:172.21.15.88:3301/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f31440102e0 con 0x7f315009ed80 2023-09-30T01:35:08.635 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:08.615+0000 7f31427fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:35:08.636 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:08.615+0000 7f31427fc700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T01:35:08.636 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:08.615+0000 7f31427fc700 10 monclient: dump: 2023-09-30T01:35:08.636 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:35:08.636 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:35:08.636 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:35:08.636 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:35:08.636 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:35:08.636 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:35:08.636 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:35:08.636 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:35:08.636 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:35:08.636 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:35:08.637 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:35:08.637 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:35:08.637 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:35:08.637 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:35:08.637 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:35:08.637 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:35:08.637 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:08.615+0000 7f3157570700 10 monclient: _renew_subs 2023-09-30T01:35:08.637 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:08.615+0000 7f3154d6b700 4 set_mon_vals no callback set 2023-09-30T01:35:08.637 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:08.615+0000 7f3157570700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.88:3301/0 2023-09-30T01:35:08.637 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:08.615+0000 7f3157570700 1 -- 172.21.15.88:0/3129039132 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f315008f120 con 0x7f315009ed80 2023-09-30T01:35:08.638 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:08.615+0000 7f3157570700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:35:08.639 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:08.615+0000 7f3157570700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:35:08.639 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:08.615+0000 7f3157570700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.88:3301/0 2023-09-30T01:35:08.639 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:08.615+0000 7f31427fc700 1 -- 172.21.15.88:0/3129039132 <== mon.2 v2:172.21.15.88:3301/0 4 ==== mgrmap(e 5) v1 ==== 40144+0+0 (secure 0 0 0) 0x7f3144011b80 con 0x7f315009ed80 2023-09-30T01:35:08.639 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:08.615+0000 7f3157570700 1 -- 172.21.15.88:0/3129039132 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f31500903c0 con 0x7f315009ed80 2023-09-30T01:35:08.639 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:08.619+0000 7f31427fc700 1 --2- 172.21.15.88:0/3129039132 >> v2:172.21.15.107:6812/29926 conn(0x7f31340332c0 0x7f3134035770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:35:08.639 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:08.619+0000 7f31427fc700 1 -- 172.21.15.88:0/3129039132 <== mon.2 v2:172.21.15.88:3301/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 3502+0+0 (secure 0 0 0) 0x7f3144042310 con 0x7f315009ed80 2023-09-30T01:35:08.639 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:08.619+0000 7f315656e700 1 --2- 172.21.15.88:0/3129039132 >> v2:172.21.15.107:6812/29926 conn(0x7f31340332c0 0x7f3134035770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:35:08.639 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:08.619+0000 7f315656e700 10 monclient: get_auth_request con 0x7f31340332c0 auth_method 0 2023-09-30T01:35:08.639 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:08.619+0000 7f315656e700 1 --2- 172.21.15.88:0/3129039132 >> v2:172.21.15.107:6812/29926 conn(0x7f31340332c0 0x7f3134035770 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f314c000c00 tx=0x7f314c009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:35:08.639 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:08.623+0000 7f31427fc700 1 -- 172.21.15.88:0/3129039132 <== mon.2 v2:172.21.15.88:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f3144018680 con 0x7f315009ed80 2023-09-30T01:35:08.639 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:08.623+0000 7f31427fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:35:08.639 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:08.623+0000 7f31427fc700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:35:08.819 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:08.815+0000 7f3157570700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T01:35:08.819 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:08.815+0000 7f3157570700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:35:08.819 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:08.815+0000 7f3157570700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.88:3301/0 2023-09-30T01:35:08.819 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:08.815+0000 7f3157570700 1 -- 172.21.15.88:0/3129039132 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f3150071960 con 0x7f315009ed80 2023-09-30T01:35:08.820 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:08.819+0000 7f31427fc700 1 -- 172.21.15.88:0/3129039132 <== mon.2 v2:172.21.15.88:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f3144017020 con 0x7f315009ed80 2023-09-30T01:35:08.820 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:08.819+0000 7f31427fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:35:08.820 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:08.819+0000 7f31427fc700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:35:08.820 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-09-30T01:35:08.820 INFO:teuthology.orchestra.run.smithi088.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:35:08.822 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:08.819+0000 7f3157570700 1 -- 172.21.15.88:0/3129039132 >> v2:172.21.15.107:6812/29926 conn(0x7f31340332c0 msgr2=0x7f3134035770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:35:08.822 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:08.819+0000 7f3157570700 1 --2- 172.21.15.88:0/3129039132 >> v2:172.21.15.107:6812/29926 conn(0x7f31340332c0 0x7f3134035770 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f314c000c00 tx=0x7f314c009040 comp rx=0 tx=0).stop 2023-09-30T01:35:08.822 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:08.819+0000 7f3157570700 10 monclient: shutdown 2023-09-30T01:35:08.822 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:08.819+0000 7f3157570700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:35:08.822 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:08.819+0000 7f3157570700 1 -- 172.21.15.88:0/3129039132 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f315009ed80 msgr2=0x7f3150140bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:35:08.823 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:08.819+0000 7f3157570700 1 --2- 172.21.15.88:0/3129039132 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f315009ed80 0x7f3150140bb0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f314400c4e0 tx=0x7f314400c7f0 comp rx=0 tx=0).stop 2023-09-30T01:35:08.823 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:08.819+0000 7f3157570700 1 -- 172.21.15.88:0/3129039132 shutdown_connections 2023-09-30T01:35:08.823 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:08.819+0000 7f3157570700 1 --2- 172.21.15.88:0/3129039132 >> v2:172.21.15.107:6812/29926 conn(0x7f31340332c0 0x7f3134035770 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:35:08.823 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:08.819+0000 7f3157570700 1 --2- 172.21.15.88:0/3129039132 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f315009f860 0x7f31501410f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:35:08.823 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:08.819+0000 7f3157570700 1 --2- 172.21.15.88:0/3129039132 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f315009ed80 0x7f3150140bb0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:35:08.823 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:08.819+0000 7f3157570700 1 --2- 172.21.15.88:0/3129039132 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7f31500637f0 0x7f3150140670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:35:08.823 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:08.819+0000 7f3157570700 1 -- 172.21.15.88:0/3129039132 >> 172.21.15.88:0/3129039132 conn(0x7f31500a0910 msgr2=0x7f31500a1b90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:35:08.823 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:08.819+0000 7f3157570700 1 -- 172.21.15.88:0/3129039132 shutdown_connections 2023-09-30T01:35:08.823 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:08.819+0000 7f3157570700 1 -- 172.21.15.88:0/3129039132 wait complete. 2023-09-30T01:35:11.838 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:35:11.947 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:11.943+0000 7f84bc429700 10 monclient: get_monmap_and_config 2023-09-30T01:35:11.947 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:11.943+0000 7f84bc429700 10 monclient: build_initial_monmap 2023-09-30T01:35:11.947 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:11.943+0000 7f84bc429700 10 monclient: monmap: 2023-09-30T01:35:11.947 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2023-09-30T01:35:11.947 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:35:11.947 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:35:11.947132+0000 2023-09-30T01:35:11.947 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:35:11.947132+0000 2023-09-30T01:35:11.947 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2023-09-30T01:35:11.947 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:35:11.948 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a-legacy 2023-09-30T01:35:11.948 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.107:6789/0 mon.noname-f-legacy 2023-09-30T01:35:11.948 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.noname-b 2023-09-30T01:35:11.948 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.noname-g 2023-09-30T01:35:11.948 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.noname-c 2023-09-30T01:35:11.948 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.noname-h 2023-09-30T01:35:11.948 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.noname-d 2023-09-30T01:35:11.948 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.noname-i 2023-09-30T01:35:11.948 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.noname-e 2023-09-30T01:35:11.948 INFO:teuthology.orchestra.run.smithi088.stderr:9: v2:172.21.15.88:3300/0 mon.noname-a 2023-09-30T01:35:11.948 INFO:teuthology.orchestra.run.smithi088.stderr:10: v2:172.21.15.107:3300/0 mon.noname-f 2023-09-30T01:35:11.948 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:35:11.953 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:11.947+0000 7f84bc429700 1 Processor -- start 2023-09-30T01:35:11.953 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:11.947+0000 7f84bc429700 1 -- start start 2023-09-30T01:35:11.953 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:11.947+0000 7f84bc429700 10 monclient: init 2023-09-30T01:35:11.953 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:11.947+0000 7f84bc429700 10 monclient: _reopen_session rank -1 2023-09-30T01:35:11.953 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:11.947+0000 7f84bc429700 10 monclient: _add_conns ranks=[0,2,1,4,10,7,3,8,6,5,9] 2023-09-30T01:35:11.953 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:11.947+0000 7f84bc429700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f84b4067130 addr v1:172.21.15.88:6789/0 2023-09-30T01:35:11.953 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:11.947+0000 7f84bc429700 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f84b41327b0 0x7f84b4132b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:35:11.954 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:11.947+0000 7f84bc429700 10 monclient(hunting): picked mon.noname-b con 0x7f84b41327b0 addr [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] 2023-09-30T01:35:11.954 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:11.947+0000 7f84bc429700 10 monclient(hunting): picked mon.noname-f-legacy con 0x7f84b4138840 addr v1:172.21.15.107:6789/0 2023-09-30T01:35:11.954 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:11.947+0000 7f84bc429700 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84b405ecb0 con 0x7f84b4067130 2023-09-30T01:35:11.954 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:11.947+0000 7f84bc429700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84b409f0d0 con 0x7f84b4138840 2023-09-30T01:35:11.954 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:11.947+0000 7f84bc429700 10 monclient(hunting): start opening mon connection 2023-09-30T01:35:11.954 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:11.947+0000 7f84bc429700 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84b41341f0 con 0x7f84b41327b0 2023-09-30T01:35:11.954 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:11.947+0000 7f84bc429700 10 monclient(hunting): _renew_subs 2023-09-30T01:35:11.954 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:11.947+0000 7f84bc429700 10 monclient(hunting): authenticate will time out at 983.985174s 2023-09-30T01:35:11.954 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:11.947+0000 7f84ba1c5700 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f84b4067130 0x7f84b4133ae0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:55916/0 (socket says 172.21.15.88:55916) 2023-09-30T01:35:11.954 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:11.947+0000 7f84ba1c5700 1 -- 172.21.15.88:0/1700667347 learned_addr learned my addr 172.21.15.88:0/1700667347 (peer_addr_for_me v1:172.21.15.88:0/0) 2023-09-30T01:35:11.954 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:11.947+0000 7f84b99c4700 1 --2- 172.21.15.88:0/1700667347 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f84b41327b0 0x7f84b4132b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:35:11.954 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:11.947+0000 7f84b99c4700 10 monclient(hunting): get_auth_request con 0x7f84b41327b0 auth_method 0 2023-09-30T01:35:11.955 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:11.947+0000 7f84b99c4700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:35:11.955 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:11.947+0000 7f84b99c4700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:35:11.955 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:11.947+0000 7f84b99c4700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:35:11.955 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:11.947+0000 7f84b91c3700 1 -- 172.21.15.88:0/1700667347 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3844922855 0 0) 0x7f84b405ecb0 con 0x7f84b4067130 2023-09-30T01:35:11.955 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:11.947+0000 7f84b91c3700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:35:11.955 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:11.947+0000 7f84b91c3700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:35:11.955 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:11.947+0000 7f84b91c3700 10 monclient(hunting): my global_id is 4752 2023-09-30T01:35:11.955 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:11.947+0000 7f84b91c3700 1 -- 172.21.15.88:0/1700667347 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f84a0003570 con 0x7f84b4067130 2023-09-30T01:35:11.955 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:11.947+0000 7f84b99c4700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:35:11.955 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:11.947+0000 7f84b99c4700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:35:11.955 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:11.947+0000 7f84b99c4700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:35:11.955 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:11.947+0000 7f84b91c3700 1 -- 172.21.15.88:0/1700667347 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3833868900 0 0) 0x7f84a0003570 con 0x7f84b4067130 2023-09-30T01:35:11.955 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:11.947+0000 7f84b91c3700 1 -- 172.21.15.88:0/1700667347 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f84b41327b0 msgr2=0x7f84b4132b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:35:11.956 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:11.947+0000 7f84b91c3700 1 --2- 172.21.15.88:0/1700667347 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f84b41327b0 0x7f84b4132b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:35:11.956 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:11.947+0000 7f84b91c3700 1 -- 172.21.15.88:0/1700667347 >> v1:172.21.15.107:6789/0 conn(0x7f84b4138840 legacy=0x7f84b413ac20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:35:11.956 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:11.947+0000 7f84b91c3700 10 monclient: _finish_hunting 0 2023-09-30T01:35:11.956 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:11.947+0000 7f84b91c3700 1 monclient: found mon.noname-a-legacy 2023-09-30T01:35:11.956 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:11.947+0000 7f84b91c3700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:35:11.956 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:11.947+0000 7f84b91c3700 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.88:6789/0 2023-09-30T01:35:11.956 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:11.947+0000 7f84b91c3700 1 -- 172.21.15.88:0/1700667347 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84b409edf0 con 0x7f84b4067130 2023-09-30T01:35:11.956 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:11.947+0000 7f84b91c3700 10 monclient: _finish_auth 0 2023-09-30T01:35:11.956 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:11.947+0000 7f84b91c3700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:35:11.956 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:11.947+0000 7f84b91c3700 1 -- 172.21.15.88:0/1700667347 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (unknown 176869241 0 0) 0x7f84b00018f0 con 0x7f84b4067130 2023-09-30T01:35:11.956 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:11.947+0000 7f84bc429700 5 monclient: authenticate success, global_id 4752 2023-09-30T01:35:11.956 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:11.947+0000 7f84bc429700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:35:11.957 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:11.947+0000 7f84b91c3700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:35:11.957 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:11.947+0000 7f84b91c3700 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2023-09-30T01:35:11.957 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:11.947+0000 7f84b91c3700 10 monclient: dump: 2023-09-30T01:35:11.957 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:35:11.957 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:35:11.957 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:35:11.957 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:35:11.957 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:35:11.957 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:35:11.957 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:35:11.957 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:35:11.957 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:35:11.958 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:35:11.958 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:35:11.958 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:35:11.958 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:35:11.958 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:35:11.958 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:35:11.958 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:35:11.958 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:11.947+0000 7f84b91c3700 1 monclient: mon.a has (v2) addrs [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] but i'm connected to v1:172.21.15.88:6789/0, reconnecting 2023-09-30T01:35:11.958 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:11.947+0000 7f84b91c3700 10 monclient: _reopen_session rank -1 2023-09-30T01:35:11.958 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:11.947+0000 7f84b91c3700 1 -- 172.21.15.88:0/1700667347 >> v1:172.21.15.88:6789/0 conn(0x7f84b4067130 legacy=0x7f84b4133ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:35:11.958 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:11.947+0000 7f84b91c3700 10 monclient: _add_conns ranks=[1,0,7,8,4,3,5,2,6] 2023-09-30T01:35:11.958 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:11.947+0000 7f84b91c3700 1 --2- 172.21.15.88:0/1700667347 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f84a00042e0 0x7f84a0004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:35:11.959 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:11.947+0000 7f84b91c3700 10 monclient(hunting): picked mon.f con 0x7f84a00042e0 addr [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] 2023-09-30T01:35:11.959 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:11.947+0000 7f84b91c3700 1 --2- 172.21.15.88:0/1700667347 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f84a0004da0 0x7f84a0005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:35:11.959 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:11.947+0000 7f84b91c3700 10 monclient(hunting): picked mon.a con 0x7f84a0004da0 addr [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] 2023-09-30T01:35:11.959 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:11.947+0000 7f84b91c3700 1 --2- 172.21.15.88:0/1700667347 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7f84a0009cf0 0x7f84a000c170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:35:11.959 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:11.947+0000 7f84b91c3700 10 monclient(hunting): picked mon.i con 0x7f84a0009cf0 addr [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] 2023-09-30T01:35:11.959 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:11.947+0000 7f84b91c3700 10 monclient(hunting): start opening mon connection 2023-09-30T01:35:11.959 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:11.947+0000 7f84b91c3700 1 -- 172.21.15.88:0/1700667347 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84b41341f0 con 0x7f84a0004da0 2023-09-30T01:35:11.959 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:11.947+0000 7f84b91c3700 10 monclient(hunting): start opening mon connection 2023-09-30T01:35:11.959 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:11.947+0000 7f84b91c3700 1 -- 172.21.15.88:0/1700667347 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84a000ca50 con 0x7f84a00042e0 2023-09-30T01:35:11.959 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:11.947+0000 7f84b91c3700 10 monclient(hunting): start opening mon connection 2023-09-30T01:35:11.959 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:11.947+0000 7f84b91c3700 1 -- 172.21.15.88:0/1700667347 --> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] -- mon_getmap magic: 0 v1 -- 0x7f84a000cbc0 con 0x7f84a0009cf0 2023-09-30T01:35:11.959 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:11.947+0000 7f84b91c3700 10 monclient(hunting): _renew_subs 2023-09-30T01:35:11.960 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:11.947+0000 7f84b91c3700 10 monclient(hunting): _finish_auth 0 2023-09-30T01:35:11.960 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:11.947+0000 7f84bc429700 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2023-09-30T01:35:11.960 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:11.947+0000 7f84b99c4700 1 --2- 172.21.15.88:0/1700667347 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f84a0004da0 0x7f84a0005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:35:11.960 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:11.947+0000 7f84b99c4700 10 monclient(hunting): get_auth_request con 0x7f84a0004da0 auth_method 0 2023-09-30T01:35:11.960 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:11.947+0000 7f84b99c4700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:35:11.960 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:11.947+0000 7f84b99c4700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:35:11.960 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:11.947+0000 7f84b99c4700 10 monclient(hunting): _init_auth already have auth, reseting 2023-09-30T01:35:11.960 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:11.947+0000 7f84b99c4700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:35:11.960 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:11.947+0000 7f84b99c4700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:35:11.960 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:11.947+0000 7f84b99c4700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-09-30T01:35:11.960 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:11.947+0000 7f84ba1c5700 1 --2- 172.21.15.88:0/1700667347 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f84a00042e0 0x7f84a0004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:35:11.960 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:11.947+0000 7f84b99c4700 10 monclient(hunting): handle_auth_done global_id 4752 payload 293 2023-09-30T01:35:11.960 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:11.947+0000 7f84ba9c6700 1 --2- 172.21.15.88:0/1700667347 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7f84a0009cf0 0x7f84a000c170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:35:11.961 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:11.947+0000 7f84b99c4700 1 -- 172.21.15.88:0/1700667347 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7f84a0009cf0 msgr2=0x7f84a000c170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:35:11.961 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:11.947+0000 7f84b99c4700 1 --2- 172.21.15.88:0/1700667347 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7f84a0009cf0 0x7f84a000c170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:35:11.961 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:11.947+0000 7f84b99c4700 1 -- 172.21.15.88:0/1700667347 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f84a00042e0 msgr2=0x7f84a0004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:35:11.961 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:11.947+0000 7f84b99c4700 1 --2- 172.21.15.88:0/1700667347 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f84a00042e0 0x7f84a0004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:35:11.961 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:11.947+0000 7f84b99c4700 10 monclient: _finish_hunting 0 2023-09-30T01:35:11.961 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:11.947+0000 7f84b99c4700 1 monclient: found mon.a 2023-09-30T01:35:11.961 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:11.947+0000 7f84b99c4700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:35:11.961 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:11.947+0000 7f84b99c4700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.88:3300/0 2023-09-30T01:35:11.961 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:11.947+0000 7f84b99c4700 1 -- 172.21.15.88:0/1700667347 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84a0003570 con 0x7f84a0004da0 2023-09-30T01:35:11.961 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:11.947+0000 7f84b99c4700 10 monclient: _finish_auth 0 2023-09-30T01:35:11.961 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:11.947+0000 7f84b99c4700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:35:11.961 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:11.947+0000 7f84ba1c5700 10 monclient: get_auth_request con 0x7f84a00042e0 auth_method 0 2023-09-30T01:35:11.962 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:11.947+0000 7f84ba9c6700 10 monclient: get_auth_request con 0x7f84a0009cf0 auth_method 0 2023-09-30T01:35:11.962 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:11.947+0000 7f84ba1c5700 1 --2- 172.21.15.88:0/1700667347 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f84a00042e0 0x7f84a0004810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-09-30T01:35:11.962 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:11.947+0000 7f84ba9c6700 1 --2- 172.21.15.88:0/1700667347 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7f84a0009cf0 0x7f84a000c170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-09-30T01:35:11.962 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:11.947+0000 7f84ba9c6700 1 -- 172.21.15.88:0/1700667347 reap_dead start 2023-09-30T01:35:11.962 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:11.947+0000 7f84b99c4700 1 --2- 172.21.15.88:0/1700667347 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f84a0004da0 0x7f84a0005210 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f84a4006f00 tx=0x7f84a40091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4da1a3a9a77a61b9 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:35:11.962 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:11.947+0000 7f84b91c3700 1 -- 172.21.15.88:0/1700667347 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f84a400eb90 con 0x7f84a0004da0 2023-09-30T01:35:11.962 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:11.947+0000 7f84b91c3700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:35:11.962 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:11.947+0000 7f84b91c3700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:35:11.962 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:11.947+0000 7f84b91c3700 10 monclient: dump: 2023-09-30T01:35:11.962 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:35:11.962 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:35:11.962 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:35:11.963 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:35:11.963 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:35:11.963 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:35:11.963 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:35:11.963 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:35:11.963 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:35:11.963 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:35:11.963 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:35:11.963 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:35:11.963 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:35:11.963 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:35:11.963 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:35:11.963 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:35:11.964 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:11.947+0000 7f84b91c3700 1 -- 172.21.15.88:0/1700667347 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84a400ed30 con 0x7f84a0004da0 2023-09-30T01:35:11.964 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:11.947+0000 7f84bc429700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:35:11.964 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:11.947+0000 7f84b91c3700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:35:11.964 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:11.947+0000 7f84bc429700 10 monclient: get_monmap_and_config success 2023-09-30T01:35:11.964 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:11.947+0000 7f84bc429700 4 set_mon_vals no callback set 2023-09-30T01:35:11.964 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:11.947+0000 7f84b91c3700 1 -- 172.21.15.88:0/1700667347 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f84a400eea0 con 0x7f84a0004da0 2023-09-30T01:35:11.964 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:11.947+0000 7f84b91c3700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:35:11.964 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:11.947+0000 7f84b91c3700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:35:11.964 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:11.947+0000 7f84b91c3700 10 monclient: dump: 2023-09-30T01:35:11.964 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:35:11.964 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:35:11.964 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:35:11.964 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:35:11.965 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:35:11.965 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:35:11.965 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:35:11.965 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:35:11.965 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:35:11.965 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:35:11.965 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:35:11.965 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:35:11.965 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:35:11.965 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:35:11.965 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:35:11.965 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:35:11.965 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:11.951+0000 7f84bc429700 10 monclient: shutdown 2023-09-30T01:35:11.966 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:11.951+0000 7f84bc429700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:35:11.966 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:11.951+0000 7f84bc429700 1 -- 172.21.15.88:0/1700667347 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f84a0004da0 msgr2=0x7f84a0005210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:35:11.966 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:11.951+0000 7f84bc429700 1 --2- 172.21.15.88:0/1700667347 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f84a0004da0 0x7f84a0005210 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f84a4006f00 tx=0x7f84a40091a0 comp rx=0 tx=0).stop 2023-09-30T01:35:11.966 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:11.951+0000 7f84bc429700 1 -- 172.21.15.88:0/1700667347 shutdown_connections 2023-09-30T01:35:11.966 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:11.951+0000 7f84bc429700 1 --2- 172.21.15.88:0/1700667347 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f84a0004da0 0x7f84a0005210 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:35:11.966 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:11.951+0000 7f84bc429700 1 -- 172.21.15.88:0/1700667347 >> 172.21.15.88:0/1700667347 conn(0x7f84b40a01d0 msgr2=0x7f84b4062c70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:35:11.966 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:11.951+0000 7f84bc429700 1 -- 172.21.15.88:0/1700667347 shutdown_connections 2023-09-30T01:35:11.966 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:11.951+0000 7f84bc429700 1 -- 172.21.15.88:0/1700667347 wait complete. 2023-09-30T01:35:11.971 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:11.951+0000 7f84bc429700 10 monclient: build_initial_monmap 2023-09-30T01:35:11.971 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:11.951+0000 7f84bc429700 10 monclient: monmap: 2023-09-30T01:35:11.971 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2023-09-30T01:35:11.971 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:35:11.971 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 0.000000 2023-09-30T01:35:11.971 INFO:teuthology.orchestra.run.smithi088.stderr:created 0.000000 2023-09-30T01:35:11.972 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2023-09-30T01:35:11.972 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 1 2023-09-30T01:35:11.972 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.noname-a 2023-09-30T01:35:11.972 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.noname-f 2023-09-30T01:35:11.972 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.noname-b 2023-09-30T01:35:11.972 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.noname-g 2023-09-30T01:35:11.972 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.noname-c 2023-09-30T01:35:11.972 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.noname-h 2023-09-30T01:35:11.972 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.noname-d 2023-09-30T01:35:11.972 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.noname-i 2023-09-30T01:35:11.972 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.noname-e 2023-09-30T01:35:11.972 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:35:11.972 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:11.951+0000 7f84bc429700 1 Processor -- start 2023-09-30T01:35:11.973 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:11.951+0000 7f84bc429700 1 -- start start 2023-09-30T01:35:11.973 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:11.951+0000 7f84bc429700 10 monclient: init 2023-09-30T01:35:11.973 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:11.955+0000 7f84bc429700 10 monclient: _reopen_session rank -1 2023-09-30T01:35:11.973 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:11.955+0000 7f84bc429700 10 monclient: _add_conns ranks=[2,6,4,8,5,3,0,1,7] 2023-09-30T01:35:11.973 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:11.955+0000 7f84bc429700 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f84a0004da0 0x7f84b4130fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:35:11.973 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:11.955+0000 7f84bc429700 10 monclient(hunting): picked mon.noname-b con 0x7f84a0004da0 addr [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] 2023-09-30T01:35:11.973 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:11.955+0000 7f84bc429700 1 --2- >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7f84b411bee0 0x7f84b4119700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:35:11.973 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:11.955+0000 7f84bc429700 10 monclient(hunting): picked mon.noname-d con 0x7f84b411bee0 addr [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] 2023-09-30T01:35:11.973 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:11.955+0000 7f84bc429700 1 --2- >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f84b4119c40 0x7f84b411a030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:35:11.973 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:11.955+0000 7f84bc429700 10 monclient(hunting): picked mon.noname-c con 0x7f84b4119c40 addr [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] 2023-09-30T01:35:11.973 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:11.955+0000 7f84bc429700 10 monclient(hunting): start opening mon connection 2023-09-30T01:35:11.973 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:11.955+0000 7f84bc429700 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84b411d420 con 0x7f84a0004da0 2023-09-30T01:35:11.973 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:11.955+0000 7f84bc429700 10 monclient(hunting): start opening mon connection 2023-09-30T01:35:11.974 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:11.955+0000 7f84bc429700 1 -- --> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] -- mon_getmap magic: 0 v1 -- 0x7f84b409e370 con 0x7f84b4119c40 2023-09-30T01:35:11.974 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:11.955+0000 7f84bc429700 10 monclient(hunting): start opening mon connection 2023-09-30T01:35:11.974 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:11.955+0000 7f84bc429700 1 -- --> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] -- mon_getmap magic: 0 v1 -- 0x7f84b406ecd0 con 0x7f84b411bee0 2023-09-30T01:35:11.974 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:11.955+0000 7f84bc429700 10 monclient(hunting): _renew_subs 2023-09-30T01:35:11.974 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:11.955+0000 7f84bc429700 10 monclient(hunting): authenticate will time out at 983.992717s 2023-09-30T01:35:11.974 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:11.955+0000 7f84ba9c6700 1 --2- >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f84b4119c40 0x7f84b411a030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:35:11.974 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:11.955+0000 7f84b99c4700 1 --2- >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7f84b411bee0 0x7f84b4119700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:35:11.974 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:11.955+0000 7f84ba1c5700 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f84a0004da0 0x7f84b4130fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:35:11.974 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:11.955+0000 7f84ba1c5700 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f84a0004da0 0x7f84b4130fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3301/0 says I am v2:172.21.15.88:50768/0 (socket says 172.21.15.88:50768) 2023-09-30T01:35:11.974 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:11.955+0000 7f84b99c4700 1 --2- >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7f84b411bee0 0x7f84b4119700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3303/0 says I am v2:172.21.15.88:51404/0 (socket says 172.21.15.88:51404) 2023-09-30T01:35:11.974 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:11.955+0000 7f84ba1c5700 1 -- 172.21.15.88:0/354972039 learned_addr learned my addr 172.21.15.88:0/354972039 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-09-30T01:35:11.974 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:11.955+0000 7f84ba9c6700 1 --2- >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f84b4119c40 0x7f84b411a030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3302/0 says I am v2:172.21.15.88:42580/0 (socket says 172.21.15.88:42580) 2023-09-30T01:35:11.975 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:11.955+0000 7f84ba1c5700 10 monclient(hunting): get_auth_request con 0x7f84a0004da0 auth_method 0 2023-09-30T01:35:11.975 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:11.955+0000 7f84ba1c5700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:35:11.975 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:11.955+0000 7f84ba1c5700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:35:11.975 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:11.955+0000 7f84ba1c5700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:35:11.975 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:11.955+0000 7f84b99c4700 10 monclient(hunting): get_auth_request con 0x7f84b411bee0 auth_method 0 2023-09-30T01:35:11.975 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:11.955+0000 7f84b99c4700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:35:11.975 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:11.955+0000 7f84b99c4700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:35:11.975 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:11.955+0000 7f84b99c4700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:35:11.975 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:11.955+0000 7f84ba9c6700 10 monclient(hunting): get_auth_request con 0x7f84b4119c40 auth_method 0 2023-09-30T01:35:11.975 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:11.955+0000 7f84ba9c6700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:35:11.975 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:11.955+0000 7f84ba9c6700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:35:11.975 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:11.955+0000 7f84ba9c6700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:35:11.976 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:11.955+0000 7f84ba1c5700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:35:11.976 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:11.955+0000 7f84ba1c5700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:35:11.976 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:11.955+0000 7f84ba1c5700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:35:11.976 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:11.955+0000 7f84b99c4700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:35:11.976 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:11.955+0000 7f84b99c4700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:35:11.976 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:11.955+0000 7f84b99c4700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:35:11.976 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:11.955+0000 7f84ba9c6700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:35:11.976 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:11.955+0000 7f84ba9c6700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:35:11.976 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:11.955+0000 7f84ba9c6700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:35:11.976 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:11.955+0000 7f84ba1c5700 10 monclient(hunting): handle_auth_done global_id 4556 payload 824 2023-09-30T01:35:11.976 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:11.955+0000 7f84ba1c5700 1 -- 172.21.15.88:0/354972039 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7f84b411bee0 msgr2=0x7f84b4119700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:35:11.976 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:11.955+0000 7f84ba1c5700 1 --2- 172.21.15.88:0/354972039 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7f84b411bee0 0x7f84b4119700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:35:11.976 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:11.955+0000 7f84ba1c5700 1 -- 172.21.15.88:0/354972039 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f84b4119c40 msgr2=0x7f84b411a030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:35:11.977 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:11.955+0000 7f84ba1c5700 1 --2- 172.21.15.88:0/354972039 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f84b4119c40 0x7f84b411a030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:35:11.977 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:11.955+0000 7f84ba1c5700 10 monclient: _finish_hunting 0 2023-09-30T01:35:11.977 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:11.955+0000 7f84ba1c5700 1 monclient: found mon.noname-b 2023-09-30T01:35:11.977 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:11.955+0000 7f84ba1c5700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:35:11.977 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:11.955+0000 7f84ba1c5700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.88:3301/0 2023-09-30T01:35:11.977 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:11.955+0000 7f84ba1c5700 1 -- 172.21.15.88:0/354972039 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84b411a570 con 0x7f84a0004da0 2023-09-30T01:35:11.977 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:11.955+0000 7f84ba1c5700 1 --2- 172.21.15.88:0/354972039 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f84a0004da0 0x7f84b4130fe0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f84b000dbb0 tx=0x7f84b0009040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:35:11.977 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:11.955+0000 7f84aa7fc700 1 -- 172.21.15.88:0/354972039 <== mon.2 v2:172.21.15.88:3301/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f84b0007e90 con 0x7f84a0004da0 2023-09-30T01:35:11.977 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:11.955+0000 7f84aa7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:35:11.977 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:11.955+0000 7f84aa7fc700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T01:35:11.977 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:11.955+0000 7f84aa7fc700 10 monclient: dump: 2023-09-30T01:35:11.977 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:35:11.978 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:35:11.978 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:35:11.978 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:35:11.978 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:35:11.978 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:35:11.978 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:35:11.978 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:35:11.978 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:35:11.978 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:35:11.978 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:35:11.978 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:35:11.978 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:35:11.979 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:35:11.979 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:35:11.979 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:35:11.979 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:11.955+0000 7f84aa7fc700 10 monclient: _finish_auth 0 2023-09-30T01:35:11.979 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:11.955+0000 7f84aa7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:35:11.979 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:11.955+0000 7f84bc429700 5 monclient: authenticate success, global_id 4556 2023-09-30T01:35:11.979 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:11.955+0000 7f84aa7fc700 1 -- 172.21.15.88:0/354972039 <== mon.2 v2:172.21.15.88:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84b0004830 con 0x7f84a0004da0 2023-09-30T01:35:11.979 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:11.955+0000 7f84bc429700 10 monclient: _renew_subs 2023-09-30T01:35:11.979 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:11.955+0000 7f84bc429700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.88:3301/0 2023-09-30T01:35:11.979 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:11.955+0000 7f84bc429700 1 -- 172.21.15.88:0/354972039 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84b411cdd0 con 0x7f84a0004da0 2023-09-30T01:35:11.979 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:11.955+0000 7f84aa7fc700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:35:11.979 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:11.955+0000 7f84aa7fc700 1 -- 172.21.15.88:0/354972039 <== mon.2 v2:172.21.15.88:3301/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f84b0004e20 con 0x7f84a0004da0 2023-09-30T01:35:11.979 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:11.955+0000 7f84aa7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:35:11.980 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:11.955+0000 7f84b89c2700 4 set_mon_vals no callback set 2023-09-30T01:35:11.980 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:11.955+0000 7f84aa7fc700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T01:35:11.980 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:11.955+0000 7f84aa7fc700 10 monclient: dump: 2023-09-30T01:35:11.980 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:35:11.980 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:35:11.980 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:35:11.980 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:35:11.980 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:35:11.980 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:35:11.980 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:35:11.980 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:35:11.980 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:35:11.980 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:35:11.981 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:35:11.981 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:35:11.981 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:35:11.981 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:35:11.981 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:35:11.981 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:35:11.981 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:11.955+0000 7f84bc429700 10 monclient: _renew_subs 2023-09-30T01:35:11.981 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:11.955+0000 7f84bc429700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.88:3301/0 2023-09-30T01:35:11.981 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:11.955+0000 7f84bc429700 1 -- 172.21.15.88:0/354972039 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f84b4132140 con 0x7f84a0004da0 2023-09-30T01:35:11.981 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:11.955+0000 7f84bc429700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:35:11.981 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:11.955+0000 7f84bc429700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:35:11.981 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:11.955+0000 7f84bc429700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.88:3301/0 2023-09-30T01:35:11.982 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:11.955+0000 7f84bc429700 1 -- 172.21.15.88:0/354972039 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f84b4071960 con 0x7f84a0004da0 2023-09-30T01:35:11.982 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:11.959+0000 7f84aa7fc700 1 -- 172.21.15.88:0/354972039 <== mon.2 v2:172.21.15.88:3301/0 4 ==== mgrmap(e 5) v1 ==== 40144+0+0 (secure 0 0 0) 0x7f84b00049a0 con 0x7f84a0004da0 2023-09-30T01:35:11.982 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:11.959+0000 7f84aa7fc700 1 --2- 172.21.15.88:0/354972039 >> v2:172.21.15.107:6812/29926 conn(0x7f8498033360 0x7f8498035810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:35:11.982 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:11.959+0000 7f84aa7fc700 1 -- 172.21.15.88:0/354972039 <== mon.2 v2:172.21.15.88:3301/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 3502+0+0 (secure 0 0 0) 0x7f84b0042a40 con 0x7f84a0004da0 2023-09-30T01:35:11.982 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:11.959+0000 7f84b99c4700 1 --2- 172.21.15.88:0/354972039 >> v2:172.21.15.107:6812/29926 conn(0x7f8498033360 0x7f8498035810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:35:11.982 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:11.959+0000 7f84b99c4700 10 monclient: get_auth_request con 0x7f8498033360 auth_method 0 2023-09-30T01:35:11.982 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:11.959+0000 7f84b99c4700 1 --2- 172.21.15.88:0/354972039 >> v2:172.21.15.107:6812/29926 conn(0x7f8498033360 0x7f8498035810 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f84a4006f00 tx=0x7f84a401a000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:35:11.982 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:11.963+0000 7f84aa7fc700 1 -- 172.21.15.88:0/354972039 <== mon.2 v2:172.21.15.88:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f84b0004c50 con 0x7f84a0004da0 2023-09-30T01:35:11.982 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:11.963+0000 7f84aa7fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:35:11.982 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:11.963+0000 7f84aa7fc700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:35:12.159 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:12.155+0000 7f84bc429700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T01:35:12.160 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:12.159+0000 7f84bc429700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:35:12.160 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:12.159+0000 7f84bc429700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.88:3301/0 2023-09-30T01:35:12.160 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:12.159+0000 7f84bc429700 1 -- 172.21.15.88:0/354972039 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f84b411a570 con 0x7f84a0004da0 2023-09-30T01:35:12.160 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:12.159+0000 7f84aa7fc700 1 -- 172.21.15.88:0/354972039 <== mon.2 v2:172.21.15.88:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f84b411a570 con 0x7f84a0004da0 2023-09-30T01:35:12.160 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:12.159+0000 7f84aa7fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:35:12.160 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:12.159+0000 7f84aa7fc700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:35:12.161 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-09-30T01:35:12.161 INFO:teuthology.orchestra.run.smithi088.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:35:12.162 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:12.159+0000 7f84bc429700 1 -- 172.21.15.88:0/354972039 >> v2:172.21.15.107:6812/29926 conn(0x7f8498033360 msgr2=0x7f8498035810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:35:12.163 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:12.159+0000 7f84bc429700 1 --2- 172.21.15.88:0/354972039 >> v2:172.21.15.107:6812/29926 conn(0x7f8498033360 0x7f8498035810 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f84a4006f00 tx=0x7f84a401a000 comp rx=0 tx=0).stop 2023-09-30T01:35:12.163 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:12.159+0000 7f84bc429700 10 monclient: shutdown 2023-09-30T01:35:12.163 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:12.159+0000 7f84bc429700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:35:12.163 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:12.159+0000 7f84bc429700 1 -- 172.21.15.88:0/354972039 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f84a0004da0 msgr2=0x7f84b4130fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:35:12.163 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:12.159+0000 7f84bc429700 1 --2- 172.21.15.88:0/354972039 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f84a0004da0 0x7f84b4130fe0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f84b000dbb0 tx=0x7f84b0009040 comp rx=0 tx=0).stop 2023-09-30T01:35:12.163 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:12.159+0000 7f84bc429700 1 -- 172.21.15.88:0/354972039 shutdown_connections 2023-09-30T01:35:12.163 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:12.159+0000 7f84bc429700 1 --2- 172.21.15.88:0/354972039 >> v2:172.21.15.107:6812/29926 conn(0x7f8498033360 0x7f8498035810 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:35:12.163 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:12.159+0000 7f84bc429700 1 --2- 172.21.15.88:0/354972039 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f84b4119c40 0x7f84b411a030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:35:12.163 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:12.159+0000 7f84bc429700 1 --2- 172.21.15.88:0/354972039 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7f84b411bee0 0x7f84b4119700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:35:12.163 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:12.159+0000 7f84bc429700 1 --2- 172.21.15.88:0/354972039 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f84a0004da0 0x7f84b4130fe0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:35:12.163 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:12.159+0000 7f84bc429700 1 -- 172.21.15.88:0/354972039 >> 172.21.15.88:0/354972039 conn(0x7f84b40a01d0 msgr2=0x7f84b40a1d80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:35:12.164 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:12.159+0000 7f84bc429700 1 -- 172.21.15.88:0/354972039 shutdown_connections 2023-09-30T01:35:12.164 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:12.159+0000 7f84bc429700 1 -- 172.21.15.88:0/354972039 wait complete. 2023-09-30T01:35:15.176 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:35:15.286 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:15.283+0000 7f4c96f23700 10 monclient: get_monmap_and_config 2023-09-30T01:35:15.286 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:15.283+0000 7f4c96f23700 10 monclient: build_initial_monmap 2023-09-30T01:35:15.286 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:15.283+0000 7f4c96f23700 10 monclient: monmap: 2023-09-30T01:35:15.286 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2023-09-30T01:35:15.287 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:35:15.287 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:35:15.286420+0000 2023-09-30T01:35:15.287 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:35:15.286420+0000 2023-09-30T01:35:15.287 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2023-09-30T01:35:15.287 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:35:15.287 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a-legacy 2023-09-30T01:35:15.287 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.107:6789/0 mon.noname-f-legacy 2023-09-30T01:35:15.287 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.noname-b 2023-09-30T01:35:15.287 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.noname-g 2023-09-30T01:35:15.287 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.noname-c 2023-09-30T01:35:15.287 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.noname-h 2023-09-30T01:35:15.287 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.noname-d 2023-09-30T01:35:15.288 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.noname-i 2023-09-30T01:35:15.288 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.noname-e 2023-09-30T01:35:15.288 INFO:teuthology.orchestra.run.smithi088.stderr:9: v2:172.21.15.88:3300/0 mon.noname-a 2023-09-30T01:35:15.288 INFO:teuthology.orchestra.run.smithi088.stderr:10: v2:172.21.15.107:3300/0 mon.noname-f 2023-09-30T01:35:15.288 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:35:15.295 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:15.287+0000 7f4c96f23700 1 Processor -- start 2023-09-30T01:35:15.295 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:15.287+0000 7f4c96f23700 1 -- start start 2023-09-30T01:35:15.295 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:15.287+0000 7f4c96f23700 10 monclient: init 2023-09-30T01:35:15.295 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:15.287+0000 7f4c96f23700 10 monclient: _reopen_session rank -1 2023-09-30T01:35:15.295 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:15.287+0000 7f4c96f23700 10 monclient: _add_conns ranks=[9,10,6,4,1,0,7,2,8,5,3] 2023-09-30T01:35:15.295 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:15.287+0000 7f4c96f23700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f4c90067130 0x7f4c90133ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:35:15.296 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:15.287+0000 7f4c96f23700 10 monclient(hunting): picked mon.noname-a con 0x7f4c90067130 addr v2:172.21.15.88:3300/0 2023-09-30T01:35:15.296 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:15.287+0000 7f4c96f23700 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f4c901327b0 0x7f4c90132b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:35:15.296 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:15.287+0000 7f4c96f23700 10 monclient(hunting): picked mon.noname-f con 0x7f4c901327b0 addr v2:172.21.15.107:3300/0 2023-09-30T01:35:15.296 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:15.287+0000 7f4c96f23700 1 --2- >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7f4c9009be50 0x7f4c9009c2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:35:15.296 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:15.287+0000 7f4c96f23700 10 monclient(hunting): picked mon.noname-d con 0x7f4c9009be50 addr [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] 2023-09-30T01:35:15.296 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:15.287+0000 7f4c96f23700 10 monclient(hunting): start opening mon connection 2023-09-30T01:35:15.296 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:15.287+0000 7f4c96f23700 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4c9009f0d0 con 0x7f4c90067130 2023-09-30T01:35:15.296 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:15.287+0000 7f4c96f23700 10 monclient(hunting): start opening mon connection 2023-09-30T01:35:15.296 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:15.287+0000 7f4c96f23700 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4c9009edf0 con 0x7f4c901327b0 2023-09-30T01:35:15.296 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:15.287+0000 7f4c96f23700 10 monclient(hunting): start opening mon connection 2023-09-30T01:35:15.296 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:15.287+0000 7f4c96f23700 1 -- --> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] -- mon_getmap magic: 0 v1 -- 0x7f4c9006a710 con 0x7f4c9009be50 2023-09-30T01:35:15.296 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:15.287+0000 7f4c96f23700 10 monclient(hunting): _renew_subs 2023-09-30T01:35:15.297 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:15.287+0000 7f4c96f23700 10 monclient(hunting): authenticate will time out at 987.324731s 2023-09-30T01:35:15.297 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:15.287+0000 7f4c954c0700 1 --2- >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7f4c9009be50 0x7f4c9009c2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:35:15.297 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:15.287+0000 7f4c954c0700 1 --2- >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7f4c9009be50 0x7f4c9009c2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3303/0 says I am v2:172.21.15.88:51406/0 (socket says 172.21.15.88:51406) 2023-09-30T01:35:15.297 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:15.287+0000 7f4c954c0700 1 -- 172.21.15.88:0/2015392239 learned_addr learned my addr 172.21.15.88:0/2015392239 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-09-30T01:35:15.297 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:15.287+0000 7f4c954c0700 10 monclient(hunting): get_auth_request con 0x7f4c9009be50 auth_method 0 2023-09-30T01:35:15.297 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:15.287+0000 7f4c954c0700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:35:15.297 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:15.287+0000 7f4c954c0700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:35:15.297 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:15.287+0000 7f4c954c0700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:35:15.297 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:15.287+0000 7f4c94cbf700 1 --2- 172.21.15.88:0/2015392239 >> v2:172.21.15.88:3300/0 conn(0x7f4c90067130 0x7f4c90133ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:35:15.297 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:15.287+0000 7f4c94cbf700 10 monclient(hunting): get_auth_request con 0x7f4c90067130 auth_method 0 2023-09-30T01:35:15.297 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:15.287+0000 7f4c94cbf700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:35:15.297 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:15.287+0000 7f4c94cbf700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:35:15.297 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:15.287+0000 7f4c94cbf700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:35:15.298 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:15.287+0000 7f4c954c0700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:35:15.298 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:15.287+0000 7f4c954c0700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:35:15.298 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:15.287+0000 7f4c954c0700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:35:15.298 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:15.287+0000 7f4c94cbf700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:35:15.298 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:15.287+0000 7f4c94cbf700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:35:15.298 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:15.287+0000 7f4c87fff700 1 --2- 172.21.15.88:0/2015392239 >> v2:172.21.15.107:3300/0 conn(0x7f4c901327b0 0x7f4c90132b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:35:15.298 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:15.287+0000 7f4c94cbf700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:35:15.298 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:15.287+0000 7f4c87fff700 10 monclient(hunting): get_auth_request con 0x7f4c901327b0 auth_method 0 2023-09-30T01:35:15.298 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:15.287+0000 7f4c87fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:35:15.298 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:15.287+0000 7f4c87fff700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:35:15.298 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:15.287+0000 7f4c87fff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:35:15.298 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:15.287+0000 7f4c94cbf700 10 monclient(hunting): handle_auth_done global_id 4761 payload 274 2023-09-30T01:35:15.299 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:15.287+0000 7f4c94cbf700 1 -- 172.21.15.88:0/2015392239 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7f4c9009be50 msgr2=0x7f4c9009c2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:35:15.299 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:15.287+0000 7f4c94cbf700 1 --2- 172.21.15.88:0/2015392239 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7f4c9009be50 0x7f4c9009c2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:35:15.299 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:15.287+0000 7f4c94cbf700 1 -- 172.21.15.88:0/2015392239 >> v2:172.21.15.107:3300/0 conn(0x7f4c901327b0 msgr2=0x7f4c90132b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:35:15.299 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:15.287+0000 7f4c94cbf700 1 --2- 172.21.15.88:0/2015392239 >> v2:172.21.15.107:3300/0 conn(0x7f4c901327b0 0x7f4c90132b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:35:15.299 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:15.287+0000 7f4c94cbf700 10 monclient: _finish_hunting 0 2023-09-30T01:35:15.299 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:15.287+0000 7f4c94cbf700 1 monclient: found mon.noname-a 2023-09-30T01:35:15.299 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:15.287+0000 7f4c94cbf700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:35:15.299 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:15.287+0000 7f4c94cbf700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.88:3300/0 2023-09-30T01:35:15.299 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:15.287+0000 7f4c94cbf700 1 -- 172.21.15.88:0/2015392239 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c9005ecb0 con 0x7f4c90067130 2023-09-30T01:35:15.299 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:15.287+0000 7f4c954c0700 1 --2- 172.21.15.88:0/2015392239 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7f4c9009be50 0x7f4c9009c2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:35:15.299 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:15.287+0000 7f4c94cbf700 1 --2- 172.21.15.88:0/2015392239 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4c90067130 0x7f4c90133ae0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f4c900a3c70 tx=0x7f4c8c0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dc4d96dbe2a3a20f server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:35:15.299 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:15.287+0000 7f4c877fe700 1 -- 172.21.15.88:0/2015392239 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f4c8c011ab0 con 0x7f4c90067130 2023-09-30T01:35:15.299 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:15.287+0000 7f4c877fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:35:15.300 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:15.287+0000 7f4c877fe700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T01:35:15.300 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:15.287+0000 7f4c877fe700 10 monclient: dump: 2023-09-30T01:35:15.300 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:35:15.300 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:35:15.300 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:35:15.300 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:35:15.300 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:35:15.300 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:35:15.300 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:35:15.300 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:35:15.300 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:35:15.300 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:35:15.301 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:35:15.301 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:35:15.301 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:35:15.301 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:35:15.301 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:35:15.301 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:35:15.301 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:15.287+0000 7f4c877fe700 10 monclient: _finish_auth 0 2023-09-30T01:35:15.301 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:15.287+0000 7f4c877fe700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:35:15.301 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:15.287+0000 7f4c96f23700 5 monclient: authenticate success, global_id 4761 2023-09-30T01:35:15.301 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:15.287+0000 7f4c96f23700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:35:15.301 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:15.287+0000 7f4c877fe700 1 -- 172.21.15.88:0/2015392239 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c8c011c30 con 0x7f4c90067130 2023-09-30T01:35:15.301 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:15.287+0000 7f4c877fe700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:35:15.302 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:15.287+0000 7f4c96f23700 10 monclient: get_monmap_and_config success 2023-09-30T01:35:15.302 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:15.287+0000 7f4c96f23700 4 set_mon_vals no callback set 2023-09-30T01:35:15.302 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:15.287+0000 7f4c877fe700 1 -- 172.21.15.88:0/2015392239 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f4c8c011da0 con 0x7f4c90067130 2023-09-30T01:35:15.302 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:15.287+0000 7f4c877fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:35:15.302 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:15.287+0000 7f4c877fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:35:15.302 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:15.287+0000 7f4c877fe700 10 monclient: dump: 2023-09-30T01:35:15.302 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:35:15.302 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:35:15.302 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:35:15.302 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:35:15.302 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:35:15.302 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:35:15.303 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:35:15.303 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:35:15.303 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:35:15.303 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:35:15.303 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:35:15.303 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:35:15.303 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:35:15.303 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:35:15.303 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:35:15.303 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:35:15.303 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:15.287+0000 7f4c96f23700 10 monclient: shutdown 2023-09-30T01:35:15.303 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:15.287+0000 7f4c96f23700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:35:15.303 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:15.287+0000 7f4c96f23700 1 -- 172.21.15.88:0/2015392239 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4c90067130 msgr2=0x7f4c90133ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:35:15.304 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:15.287+0000 7f4c96f23700 1 --2- 172.21.15.88:0/2015392239 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4c90067130 0x7f4c90133ae0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f4c900a3c70 tx=0x7f4c8c0091a0 comp rx=0 tx=0).stop 2023-09-30T01:35:15.304 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:15.287+0000 7f4c96f23700 1 -- 172.21.15.88:0/2015392239 shutdown_connections 2023-09-30T01:35:15.304 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:15.287+0000 7f4c96f23700 1 --2- 172.21.15.88:0/2015392239 >> v2:172.21.15.107:3300/0 conn(0x7f4c901327b0 0x7f4c90132b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:35:15.304 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:15.287+0000 7f4c96f23700 1 --2- 172.21.15.88:0/2015392239 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7f4c9009be50 0x7f4c9009c2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:35:15.304 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:15.287+0000 7f4c96f23700 1 --2- 172.21.15.88:0/2015392239 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4c90067130 0x7f4c90133ae0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:35:15.304 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:15.287+0000 7f4c96f23700 1 -- 172.21.15.88:0/2015392239 >> 172.21.15.88:0/2015392239 conn(0x7f4c900a01d0 msgr2=0x7f4c90062c70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:35:15.304 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:15.287+0000 7f4c96f23700 1 -- 172.21.15.88:0/2015392239 shutdown_connections 2023-09-30T01:35:15.304 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:15.287+0000 7f4c96f23700 1 -- 172.21.15.88:0/2015392239 wait complete. 2023-09-30T01:35:15.304 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:15.287+0000 7f4c96f23700 10 monclient: build_initial_monmap 2023-09-30T01:35:15.304 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:15.287+0000 7f4c96f23700 10 monclient: monmap: 2023-09-30T01:35:15.304 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2023-09-30T01:35:15.304 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:35:15.305 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 0.000000 2023-09-30T01:35:15.305 INFO:teuthology.orchestra.run.smithi088.stderr:created 0.000000 2023-09-30T01:35:15.305 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2023-09-30T01:35:15.305 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 1 2023-09-30T01:35:15.305 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.noname-a 2023-09-30T01:35:15.305 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.noname-f 2023-09-30T01:35:15.305 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.noname-b 2023-09-30T01:35:15.305 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.noname-g 2023-09-30T01:35:15.305 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.noname-c 2023-09-30T01:35:15.305 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.noname-h 2023-09-30T01:35:15.305 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.noname-d 2023-09-30T01:35:15.305 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.noname-i 2023-09-30T01:35:15.306 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.noname-e 2023-09-30T01:35:15.306 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:35:15.306 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:15.291+0000 7f4c96f23700 1 Processor -- start 2023-09-30T01:35:15.306 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:15.291+0000 7f4c96f23700 1 -- start start 2023-09-30T01:35:15.306 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:15.291+0000 7f4c96f23700 10 monclient: init 2023-09-30T01:35:15.306 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:15.291+0000 7f4c96f23700 10 monclient: _reopen_session rank -1 2023-09-30T01:35:15.306 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:15.291+0000 7f4c96f23700 10 monclient: _add_conns ranks=[7,0,8,5,3,2,4,1,6] 2023-09-30T01:35:15.306 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:15.291+0000 7f4c96f23700 1 --2- >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7f4c90067130 0x7f4c901426d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:35:15.306 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:15.291+0000 7f4c96f23700 10 monclient(hunting): picked mon.noname-i con 0x7f4c90067130 addr [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] 2023-09-30T01:35:15.306 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:15.291+0000 7f4c96f23700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4c9009be50 0x7f4c90142c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:35:15.306 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:15.291+0000 7f4c96f23700 10 monclient(hunting): picked mon.noname-a con 0x7f4c9009be50 addr [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] 2023-09-30T01:35:15.306 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:15.291+0000 7f4c96f23700 1 --2- >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f4c901327b0 0x7f4c90143150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:35:15.307 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:15.291+0000 7f4c96f23700 10 monclient(hunting): picked mon.noname-e con 0x7f4c901327b0 addr [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] 2023-09-30T01:35:15.307 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:15.291+0000 7f4c96f23700 10 monclient(hunting): start opening mon connection 2023-09-30T01:35:15.307 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:15.291+0000 7f4c96f23700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c90120030 con 0x7f4c9009be50 2023-09-30T01:35:15.307 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:15.291+0000 7f4c96f23700 10 monclient(hunting): start opening mon connection 2023-09-30T01:35:15.307 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:15.291+0000 7f4c96f23700 1 -- --> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] -- mon_getmap magic: 0 v1 -- 0x7f4c90121330 con 0x7f4c90067130 2023-09-30T01:35:15.307 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:15.291+0000 7f4c96f23700 10 monclient(hunting): start opening mon connection 2023-09-30T01:35:15.307 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:15.291+0000 7f4c96f23700 1 -- --> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] -- mon_getmap magic: 0 v1 -- 0x7f4c90120440 con 0x7f4c901327b0 2023-09-30T01:35:15.307 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:15.291+0000 7f4c96f23700 10 monclient(hunting): _renew_subs 2023-09-30T01:35:15.307 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:15.291+0000 7f4c96f23700 10 monclient(hunting): authenticate will time out at 987.329014s 2023-09-30T01:35:15.307 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:15.291+0000 7f4c87fff700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4c9009be50 0x7f4c90142c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:35:15.307 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:15.291+0000 7f4c954c0700 1 --2- >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f4c901327b0 0x7f4c90143150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:35:15.307 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:15.291+0000 7f4c954c0700 1 --2- >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f4c901327b0 0x7f4c90143150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3304/0 says I am v2:172.21.15.88:53162/0 (socket says 172.21.15.88:53162) 2023-09-30T01:35:15.307 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:15.291+0000 7f4c954c0700 1 -- 172.21.15.88:0/1787025185 learned_addr learned my addr 172.21.15.88:0/1787025185 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-09-30T01:35:15.308 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:15.291+0000 7f4c954c0700 10 monclient(hunting): get_auth_request con 0x7f4c901327b0 auth_method 0 2023-09-30T01:35:15.308 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:15.291+0000 7f4c954c0700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:35:15.308 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:15.291+0000 7f4c954c0700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:35:15.308 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:15.291+0000 7f4c954c0700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:35:15.308 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:15.291+0000 7f4c87fff700 10 monclient(hunting): get_auth_request con 0x7f4c9009be50 auth_method 0 2023-09-30T01:35:15.308 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:15.291+0000 7f4c87fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:35:15.308 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:15.291+0000 7f4c87fff700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:35:15.308 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:15.291+0000 7f4c87fff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:35:15.308 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:15.291+0000 7f4c94cbf700 1 --2- 172.21.15.88:0/1787025185 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7f4c90067130 0x7f4c901426d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:35:15.308 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:15.291+0000 7f4c94cbf700 10 monclient(hunting): get_auth_request con 0x7f4c90067130 auth_method 0 2023-09-30T01:35:15.308 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:15.291+0000 7f4c94cbf700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:35:15.308 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:15.291+0000 7f4c94cbf700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:35:15.308 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:15.291+0000 7f4c94cbf700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:35:15.308 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:15.291+0000 7f4c954c0700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:35:15.309 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:15.291+0000 7f4c954c0700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:35:15.309 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:15.291+0000 7f4c954c0700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:35:15.309 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:15.291+0000 7f4c87fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:35:15.309 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:15.291+0000 7f4c87fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:35:15.309 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:15.291+0000 7f4c87fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:35:15.309 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:15.291+0000 7f4c94cbf700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:35:15.309 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:15.291+0000 7f4c94cbf700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:35:15.309 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:15.291+0000 7f4c94cbf700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:35:15.309 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:15.291+0000 7f4c954c0700 10 monclient(hunting): handle_auth_done global_id 4634 payload 824 2023-09-30T01:35:15.309 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:15.291+0000 7f4c954c0700 1 -- 172.21.15.88:0/1787025185 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7f4c90067130 msgr2=0x7f4c901426d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:35:15.309 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:15.291+0000 7f4c954c0700 1 --2- 172.21.15.88:0/1787025185 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7f4c90067130 0x7f4c901426d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:35:15.309 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:15.291+0000 7f4c954c0700 1 -- 172.21.15.88:0/1787025185 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4c9009be50 msgr2=0x7f4c90142c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:35:15.309 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:15.291+0000 7f4c954c0700 1 --2- 172.21.15.88:0/1787025185 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4c9009be50 0x7f4c90142c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:35:15.310 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:15.291+0000 7f4c954c0700 10 monclient: _finish_hunting 0 2023-09-30T01:35:15.310 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:15.291+0000 7f4c954c0700 1 monclient: found mon.noname-e 2023-09-30T01:35:15.310 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:15.291+0000 7f4c954c0700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:35:15.310 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:15.291+0000 7f4c954c0700 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.88:3304/0 2023-09-30T01:35:15.310 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:15.291+0000 7f4c954c0700 1 -- 172.21.15.88:0/1787025185 --> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c88009ae0 con 0x7f4c901327b0 2023-09-30T01:35:15.310 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:15.291+0000 7f4c954c0700 1 --2- 172.21.15.88:0/1787025185 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f4c901327b0 0x7f4c90143150 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f4c88009fd0 tx=0x7f4c8800b040 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:35:15.310 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:15.291+0000 7f4c84ff9700 1 -- 172.21.15.88:0/1787025185 <== mon.8 v2:172.21.15.88:3304/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f4c88018070 con 0x7f4c901327b0 2023-09-30T01:35:15.310 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:15.291+0000 7f4c84ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:35:15.310 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:15.291+0000 7f4c84ff9700 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2023-09-30T01:35:15.310 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:15.291+0000 7f4c84ff9700 10 monclient: dump: 2023-09-30T01:35:15.310 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:35:15.311 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:35:15.311 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:35:15.311 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:35:15.311 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:35:15.311 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:35:15.311 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:35:15.311 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:35:15.311 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:35:15.311 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:35:15.311 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:35:15.311 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:35:15.311 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:35:15.312 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:35:15.312 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:35:15.312 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:35:15.312 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:15.291+0000 7f4c84ff9700 10 monclient: _finish_auth 0 2023-09-30T01:35:15.312 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:15.291+0000 7f4c84ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:35:15.312 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:15.291+0000 7f4c84ff9700 1 -- 172.21.15.88:0/1787025185 <== mon.8 v2:172.21.15.88:3304/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c8800f040 con 0x7f4c901327b0 2023-09-30T01:35:15.312 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:15.291+0000 7f4c96f23700 5 monclient: authenticate success, global_id 4634 2023-09-30T01:35:15.312 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:15.291+0000 7f4c96f23700 10 monclient: _renew_subs 2023-09-30T01:35:15.312 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:15.291+0000 7f4c96f23700 10 monclient: _send_mon_message to mon.e at v2:172.21.15.88:3304/0 2023-09-30T01:35:15.312 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:15.291+0000 7f4c96f23700 1 -- 172.21.15.88:0/1787025185 --> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c8c009420 con 0x7f4c901327b0 2023-09-30T01:35:15.312 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:15.291+0000 7f4c84ff9700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:35:15.312 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:15.291+0000 7f4c84ff9700 1 -- 172.21.15.88:0/1787025185 <== mon.8 v2:172.21.15.88:3304/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f4c880092e0 con 0x7f4c901327b0 2023-09-30T01:35:15.313 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:15.291+0000 7f4c84ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:35:15.313 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:15.291+0000 7f4c84ff9700 10 monclient: got monmap 1 from mon.e (according to old e1) 2023-09-30T01:35:15.313 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:15.291+0000 7f4c86ffd700 4 set_mon_vals no callback set 2023-09-30T01:35:15.313 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:15.291+0000 7f4c84ff9700 10 monclient: dump: 2023-09-30T01:35:15.313 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:35:15.313 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:35:15.313 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:35:15.313 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:35:15.313 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:35:15.313 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:35:15.313 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:35:15.313 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:35:15.313 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:35:15.313 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:35:15.314 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:35:15.314 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:35:15.314 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:35:15.314 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:35:15.314 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:35:15.314 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:35:15.314 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:15.291+0000 7f4c96f23700 10 monclient: _renew_subs 2023-09-30T01:35:15.314 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:15.291+0000 7f4c96f23700 10 monclient: _send_mon_message to mon.e at v2:172.21.15.88:3304/0 2023-09-30T01:35:15.314 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:15.291+0000 7f4c96f23700 1 -- 172.21.15.88:0/1787025185 --> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c9009e3f0 con 0x7f4c901327b0 2023-09-30T01:35:15.314 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:15.291+0000 7f4c84ff9700 1 -- 172.21.15.88:0/1787025185 <== mon.8 v2:172.21.15.88:3304/0 4 ==== mgrmap(e 5) v1 ==== 40144+0+0 (secure 0 0 0) 0x7f4c88003020 con 0x7f4c901327b0 2023-09-30T01:35:15.314 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:15.291+0000 7f4c96f23700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:35:15.315 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:15.291+0000 7f4c96f23700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:35:15.315 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:15.291+0000 7f4c96f23700 10 monclient: _send_mon_message to mon.e at v2:172.21.15.88:3304/0 2023-09-30T01:35:15.315 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:15.291+0000 7f4c84ff9700 1 --2- 172.21.15.88:0/1787025185 >> v2:172.21.15.107:6812/29926 conn(0x7f4c7c033360 0x7f4c7c035810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:35:15.315 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:15.291+0000 7f4c96f23700 1 -- 172.21.15.88:0/1787025185 --> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4c90071960 con 0x7f4c901327b0 2023-09-30T01:35:15.315 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:15.295+0000 7f4c84ff9700 1 -- 172.21.15.88:0/1787025185 <== mon.8 v2:172.21.15.88:3304/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 3502+0+0 (secure 0 0 0) 0x7f4c88043220 con 0x7f4c901327b0 2023-09-30T01:35:15.315 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:15.295+0000 7f4c94cbf700 1 --2- 172.21.15.88:0/1787025185 >> v2:172.21.15.107:6812/29926 conn(0x7f4c7c033360 0x7f4c7c035810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:35:15.316 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:15.295+0000 7f4c94cbf700 10 monclient: get_auth_request con 0x7f4c7c033360 auth_method 0 2023-09-30T01:35:15.316 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:15.295+0000 7f4c94cbf700 1 --2- 172.21.15.88:0/1787025185 >> v2:172.21.15.107:6812/29926 conn(0x7f4c7c033360 0x7f4c7c035810 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f4c900a3c70 tx=0x7f4c8c017000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:35:15.316 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:15.303+0000 7f4c84ff9700 1 -- 172.21.15.88:0/1787025185 <== mon.8 v2:172.21.15.88:3304/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f4c8801ca10 con 0x7f4c901327b0 2023-09-30T01:35:15.316 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:15.303+0000 7f4c84ff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:35:15.316 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:15.303+0000 7f4c84ff9700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:35:15.500 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:15.495+0000 7f4c96f23700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T01:35:15.500 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:15.499+0000 7f4c96f23700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:35:15.500 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:15.499+0000 7f4c96f23700 10 monclient: _send_mon_message to mon.e at v2:172.21.15.88:3304/0 2023-09-30T01:35:15.500 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:15.499+0000 7f4c96f23700 1 -- 172.21.15.88:0/1787025185 --> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f4c901217a0 con 0x7f4c901327b0 2023-09-30T01:35:15.500 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:15.499+0000 7f4c84ff9700 1 -- 172.21.15.88:0/1787025185 <== mon.8 v2:172.21.15.88:3304/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f4c88030080 con 0x7f4c901327b0 2023-09-30T01:35:15.501 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:15.499+0000 7f4c84ff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:35:15.501 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:15.499+0000 7f4c84ff9700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:35:15.501 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-09-30T01:35:15.501 INFO:teuthology.orchestra.run.smithi088.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:35:15.503 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:15.499+0000 7f4c96f23700 1 -- 172.21.15.88:0/1787025185 >> v2:172.21.15.107:6812/29926 conn(0x7f4c7c033360 msgr2=0x7f4c7c035810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:35:15.503 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:15.499+0000 7f4c96f23700 1 --2- 172.21.15.88:0/1787025185 >> v2:172.21.15.107:6812/29926 conn(0x7f4c7c033360 0x7f4c7c035810 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f4c900a3c70 tx=0x7f4c8c017000 comp rx=0 tx=0).stop 2023-09-30T01:35:15.503 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:15.499+0000 7f4c96f23700 10 monclient: shutdown 2023-09-30T01:35:15.503 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:15.499+0000 7f4c96f23700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:35:15.503 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:15.499+0000 7f4c96f23700 1 -- 172.21.15.88:0/1787025185 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f4c901327b0 msgr2=0x7f4c90143150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:35:15.503 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:15.499+0000 7f4c96f23700 1 --2- 172.21.15.88:0/1787025185 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f4c901327b0 0x7f4c90143150 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f4c88009fd0 tx=0x7f4c8800b040 comp rx=0 tx=0).stop 2023-09-30T01:35:15.503 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:15.499+0000 7f4c96f23700 1 -- 172.21.15.88:0/1787025185 shutdown_connections 2023-09-30T01:35:15.503 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:15.499+0000 7f4c96f23700 1 --2- 172.21.15.88:0/1787025185 >> v2:172.21.15.107:6812/29926 conn(0x7f4c7c033360 0x7f4c7c035810 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:35:15.503 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:15.499+0000 7f4c96f23700 1 --2- 172.21.15.88:0/1787025185 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f4c901327b0 0x7f4c90143150 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:35:15.504 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:15.499+0000 7f4c96f23700 1 --2- 172.21.15.88:0/1787025185 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4c9009be50 0x7f4c90142c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:35:15.504 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:15.499+0000 7f4c96f23700 1 --2- 172.21.15.88:0/1787025185 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7f4c90067130 0x7f4c901426d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:35:15.504 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:15.499+0000 7f4c96f23700 1 -- 172.21.15.88:0/1787025185 >> 172.21.15.88:0/1787025185 conn(0x7f4c900a01d0 msgr2=0x7f4c9006d5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:35:15.504 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:15.499+0000 7f4c96f23700 1 -- 172.21.15.88:0/1787025185 shutdown_connections 2023-09-30T01:35:15.504 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:15.499+0000 7f4c96f23700 1 -- 172.21.15.88:0/1787025185 wait complete. 2023-09-30T01:35:18.514 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:35:18.621 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:18.618+0000 7f3c2e4ca700 10 monclient: get_monmap_and_config 2023-09-30T01:35:18.622 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:18.618+0000 7f3c2e4ca700 10 monclient: build_initial_monmap 2023-09-30T01:35:18.622 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:18.618+0000 7f3c2e4ca700 10 monclient: monmap: 2023-09-30T01:35:18.622 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2023-09-30T01:35:18.622 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:35:18.622 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:35:18.621876+0000 2023-09-30T01:35:18.622 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:35:18.621876+0000 2023-09-30T01:35:18.622 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2023-09-30T01:35:18.622 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:35:18.622 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a-legacy 2023-09-30T01:35:18.622 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.107:6789/0 mon.noname-f-legacy 2023-09-30T01:35:18.622 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.noname-b 2023-09-30T01:35:18.623 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.noname-g 2023-09-30T01:35:18.623 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.noname-c 2023-09-30T01:35:18.623 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.noname-h 2023-09-30T01:35:18.623 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.noname-d 2023-09-30T01:35:18.623 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.noname-i 2023-09-30T01:35:18.623 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.noname-e 2023-09-30T01:35:18.623 INFO:teuthology.orchestra.run.smithi088.stderr:9: v2:172.21.15.88:3300/0 mon.noname-a 2023-09-30T01:35:18.623 INFO:teuthology.orchestra.run.smithi088.stderr:10: v2:172.21.15.107:3300/0 mon.noname-f 2023-09-30T01:35:18.623 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:35:18.626 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:18.618+0000 7f3c2e4ca700 1 Processor -- start 2023-09-30T01:35:18.627 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:18.618+0000 7f3c2e4ca700 1 -- start start 2023-09-30T01:35:18.627 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:18.618+0000 7f3c2e4ca700 10 monclient: init 2023-09-30T01:35:18.627 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:18.622+0000 7f3c2e4ca700 10 monclient: _reopen_session rank -1 2023-09-30T01:35:18.627 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:18.622+0000 7f3c2e4ca700 10 monclient: _add_conns ranks=[9,5,2,4,3,7,0,1,6,8,10] 2023-09-30T01:35:18.627 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:18.622+0000 7f3c2e4ca700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f3c2806e870 0x7f3c2806c710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:35:18.628 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:18.622+0000 7f3c2e4ca700 10 monclient(hunting): picked mon.noname-a con 0x7f3c2806e870 addr v2:172.21.15.88:3300/0 2023-09-30T01:35:18.628 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:18.622+0000 7f3c2e4ca700 1 --2- >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7f3c2809d5f0 0x7f3c2809d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:35:18.628 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:18.622+0000 7f3c2e4ca700 10 monclient(hunting): picked mon.noname-h con 0x7f3c2809d5f0 addr [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] 2023-09-30T01:35:18.628 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:18.622+0000 7f3c2e4ca700 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f3c2809df00 0x7f3c28136460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:35:18.628 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:18.622+0000 7f3c2e4ca700 10 monclient(hunting): picked mon.noname-b con 0x7f3c2809df00 addr [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] 2023-09-30T01:35:18.628 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:18.622+0000 7f3c2e4ca700 10 monclient(hunting): start opening mon connection 2023-09-30T01:35:18.628 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:18.622+0000 7f3c2e4ca700 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3c2806ee90 con 0x7f3c2806e870 2023-09-30T01:35:18.628 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:18.622+0000 7f3c2e4ca700 10 monclient(hunting): start opening mon connection 2023-09-30T01:35:18.628 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:18.622+0000 7f3c2e4ca700 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c2806a670 con 0x7f3c2809df00 2023-09-30T01:35:18.628 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:18.622+0000 7f3c2e4ca700 10 monclient(hunting): start opening mon connection 2023-09-30T01:35:18.628 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:18.622+0000 7f3c2e4ca700 1 -- --> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] -- mon_getmap magic: 0 v1 -- 0x7f3c2806ec40 con 0x7f3c2809d5f0 2023-09-30T01:35:18.629 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:18.622+0000 7f3c2e4ca700 10 monclient(hunting): _renew_subs 2023-09-30T01:35:18.629 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:18.622+0000 7f3c2e4ca700 10 monclient(hunting): authenticate will time out at 990.660942s 2023-09-30T01:35:18.629 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:18.622+0000 7f3c2ca67700 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f3c2809df00 0x7f3c28136460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:35:18.629 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:18.622+0000 7f3c27fff700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f3c2806e870 0x7f3c2806c710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:35:18.629 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:18.622+0000 7f3c2ca67700 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f3c2809df00 0x7f3c28136460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3301/0 says I am v2:172.21.15.88:50784/0 (socket says 172.21.15.88:50784) 2023-09-30T01:35:18.629 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:18.622+0000 7f3c2ca67700 1 -- 172.21.15.88:0/2835859992 learned_addr learned my addr 172.21.15.88:0/2835859992 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-09-30T01:35:18.629 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:18.622+0000 7f3c2ca67700 10 monclient(hunting): get_auth_request con 0x7f3c2809df00 auth_method 0 2023-09-30T01:35:18.629 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:18.622+0000 7f3c2ca67700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:35:18.629 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:18.622+0000 7f3c2ca67700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:35:18.629 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:18.622+0000 7f3c2ca67700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:35:18.629 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:18.622+0000 7f3c27fff700 10 monclient(hunting): get_auth_request con 0x7f3c2806e870 auth_method 0 2023-09-30T01:35:18.629 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:18.622+0000 7f3c27fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:35:18.630 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:18.622+0000 7f3c27fff700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:35:18.630 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:18.622+0000 7f3c27fff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:35:18.630 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:18.622+0000 7f3c2ca67700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:35:18.630 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:18.622+0000 7f3c2ca67700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:35:18.630 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:18.622+0000 7f3c2ca67700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:35:18.630 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:18.622+0000 7f3c27fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:35:18.630 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:18.622+0000 7f3c27fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:35:18.630 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:18.622+0000 7f3c27fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:35:18.630 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:18.622+0000 7f3c277fe700 1 --2- 172.21.15.88:0/2835859992 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7f3c2809d5f0 0x7f3c2809d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:35:18.630 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:18.622+0000 7f3c2ca67700 10 monclient(hunting): handle_auth_done global_id 4565 payload 274 2023-09-30T01:35:18.630 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:18.622+0000 7f3c2ca67700 1 -- 172.21.15.88:0/2835859992 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7f3c2809d5f0 msgr2=0x7f3c2809d9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:35:18.631 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:18.622+0000 7f3c2ca67700 1 --2- 172.21.15.88:0/2835859992 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7f3c2809d5f0 0x7f3c2809d9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:35:18.631 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:18.622+0000 7f3c2ca67700 1 -- 172.21.15.88:0/2835859992 >> v2:172.21.15.88:3300/0 conn(0x7f3c2806e870 msgr2=0x7f3c2806c710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:35:18.631 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:18.622+0000 7f3c2ca67700 1 --2- 172.21.15.88:0/2835859992 >> v2:172.21.15.88:3300/0 conn(0x7f3c2806e870 0x7f3c2806c710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:35:18.631 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:18.622+0000 7f3c2ca67700 10 monclient: _finish_hunting 0 2023-09-30T01:35:18.631 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:18.622+0000 7f3c2ca67700 1 monclient: found mon.noname-b 2023-09-30T01:35:18.631 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:18.622+0000 7f3c2ca67700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:35:18.631 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:18.622+0000 7f3c2ca67700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.88:3301/0 2023-09-30T01:35:18.631 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:18.622+0000 7f3c2ca67700 1 -- 172.21.15.88:0/2835859992 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c2805ecb0 con 0x7f3c2809df00 2023-09-30T01:35:18.631 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:18.622+0000 7f3c277fe700 10 monclient: get_auth_request con 0x7f3c2809d5f0 auth_method 0 2023-09-30T01:35:18.631 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:18.622+0000 7f3c277fe700 1 --2- 172.21.15.88:0/2835859992 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7f3c2809d5f0 0x7f3c2809d9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-09-30T01:35:18.631 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:18.622+0000 7f3c2ca67700 1 --2- 172.21.15.88:0/2835859992 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f3c2809df00 0x7f3c28136460 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f3c200099e0 tx=0x7f3c20009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8701f0c584c6ff9e server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:35:18.631 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:18.622+0000 7f3c26ffd700 1 -- 172.21.15.88:0/2835859992 <== mon.2 v2:172.21.15.88:3301/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f3c2000bca0 con 0x7f3c2809df00 2023-09-30T01:35:18.632 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:18.622+0000 7f3c26ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:35:18.632 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:18.622+0000 7f3c26ffd700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T01:35:18.632 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:18.622+0000 7f3c26ffd700 10 monclient: dump: 2023-09-30T01:35:18.632 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:35:18.632 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:35:18.632 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:35:18.632 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:35:18.632 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:35:18.632 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:35:18.632 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:35:18.632 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:35:18.632 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:35:18.633 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:35:18.633 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:35:18.633 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:35:18.633 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:35:18.633 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:35:18.633 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:35:18.633 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:35:18.633 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:18.622+0000 7f3c26ffd700 10 monclient: _finish_auth 0 2023-09-30T01:35:18.633 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:18.622+0000 7f3c26ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:35:18.633 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:18.622+0000 7f3c26ffd700 1 -- 172.21.15.88:0/2835859992 <== mon.2 v2:172.21.15.88:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c2000be20 con 0x7f3c2809df00 2023-09-30T01:35:18.633 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:18.622+0000 7f3c2e4ca700 5 monclient: authenticate success, global_id 4565 2023-09-30T01:35:18.633 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:18.622+0000 7f3c2e4ca700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:35:18.633 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:18.622+0000 7f3c26ffd700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:35:18.634 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:18.622+0000 7f3c2e4ca700 10 monclient: get_monmap_and_config success 2023-09-30T01:35:18.634 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:18.622+0000 7f3c2e4ca700 4 set_mon_vals no callback set 2023-09-30T01:35:18.634 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:18.622+0000 7f3c26ffd700 1 -- 172.21.15.88:0/2835859992 <== mon.2 v2:172.21.15.88:3301/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f3c20011420 con 0x7f3c2809df00 2023-09-30T01:35:18.634 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:18.622+0000 7f3c26ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:35:18.634 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:18.622+0000 7f3c26ffd700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T01:35:18.634 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:18.622+0000 7f3c26ffd700 10 monclient: dump: 2023-09-30T01:35:18.634 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:35:18.634 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:35:18.634 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:35:18.634 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:35:18.634 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:35:18.634 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:35:18.634 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:35:18.635 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:35:18.635 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:35:18.635 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:35:18.635 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:35:18.635 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:35:18.635 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:35:18.635 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:35:18.635 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:35:18.635 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:35:18.640 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:18.622+0000 7f3c2e4ca700 10 monclient: shutdown 2023-09-30T01:35:18.640 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:18.622+0000 7f3c2e4ca700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:35:18.640 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:18.622+0000 7f3c2e4ca700 1 -- 172.21.15.88:0/2835859992 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f3c2809df00 msgr2=0x7f3c28136460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:35:18.640 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:18.622+0000 7f3c2e4ca700 1 --2- 172.21.15.88:0/2835859992 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f3c2809df00 0x7f3c28136460 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f3c200099e0 tx=0x7f3c20009250 comp rx=0 tx=0).stop 2023-09-30T01:35:18.640 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:18.622+0000 7f3c2e4ca700 1 -- 172.21.15.88:0/2835859992 shutdown_connections 2023-09-30T01:35:18.640 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:18.622+0000 7f3c2e4ca700 1 --2- 172.21.15.88:0/2835859992 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f3c2809df00 0x7f3c28136460 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:35:18.640 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:18.622+0000 7f3c2e4ca700 1 --2- 172.21.15.88:0/2835859992 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7f3c2809d5f0 0x7f3c2809d9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:35:18.640 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:18.622+0000 7f3c2e4ca700 1 --2- 172.21.15.88:0/2835859992 >> v2:172.21.15.88:3300/0 conn(0x7f3c2806e870 0x7f3c2806c710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:35:18.640 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:18.622+0000 7f3c2e4ca700 1 -- 172.21.15.88:0/2835859992 >> 172.21.15.88:0/2835859992 conn(0x7f3c280a01d0 msgr2=0x7f3c28062c70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:35:18.641 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:18.622+0000 7f3c2e4ca700 1 -- 172.21.15.88:0/2835859992 shutdown_connections 2023-09-30T01:35:18.641 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:18.622+0000 7f3c2e4ca700 1 -- 172.21.15.88:0/2835859992 wait complete. 2023-09-30T01:35:18.641 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:18.622+0000 7f3c2e4ca700 10 monclient: build_initial_monmap 2023-09-30T01:35:18.641 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:18.622+0000 7f3c2e4ca700 10 monclient: monmap: 2023-09-30T01:35:18.641 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2023-09-30T01:35:18.641 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:35:18.641 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 0.000000 2023-09-30T01:35:18.641 INFO:teuthology.orchestra.run.smithi088.stderr:created 0.000000 2023-09-30T01:35:18.641 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2023-09-30T01:35:18.641 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 1 2023-09-30T01:35:18.642 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.noname-a 2023-09-30T01:35:18.642 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.noname-f 2023-09-30T01:35:18.642 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.noname-b 2023-09-30T01:35:18.642 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.noname-g 2023-09-30T01:35:18.642 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.noname-c 2023-09-30T01:35:18.642 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.noname-h 2023-09-30T01:35:18.642 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.noname-d 2023-09-30T01:35:18.642 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.noname-i 2023-09-30T01:35:18.642 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.noname-e 2023-09-30T01:35:18.642 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:35:18.642 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:18.626+0000 7f3c2e4ca700 1 Processor -- start 2023-09-30T01:35:18.643 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:18.626+0000 7f3c2e4ca700 1 -- start start 2023-09-30T01:35:18.643 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:18.626+0000 7f3c2e4ca700 10 monclient: init 2023-09-30T01:35:18.643 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:18.626+0000 7f3c2e4ca700 10 monclient: _reopen_session rank -1 2023-09-30T01:35:18.643 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:18.626+0000 7f3c2e4ca700 10 monclient: _add_conns ranks=[5,4,8,7,2,1,0,6,3] 2023-09-30T01:35:18.643 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:18.626+0000 7f3c2e4ca700 1 --2- >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7f3c2806e870 0x7f3c281368c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:35:18.643 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:18.626+0000 7f3c2e4ca700 10 monclient(hunting): picked mon.noname-h con 0x7f3c2806e870 addr [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] 2023-09-30T01:35:18.643 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:18.626+0000 7f3c2e4ca700 1 --2- >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f3c2809d5f0 0x7f3c28097f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:35:18.643 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:18.626+0000 7f3c2e4ca700 10 monclient(hunting): picked mon.noname-c con 0x7f3c2809d5f0 addr [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] 2023-09-30T01:35:18.643 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:18.626+0000 7f3c2e4ca700 1 --2- >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f3c2809df00 0x7f3c2809bdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:35:18.643 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:18.626+0000 7f3c2e4ca700 10 monclient(hunting): picked mon.noname-e con 0x7f3c2809df00 addr [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] 2023-09-30T01:35:18.643 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:18.626+0000 7f3c2e4ca700 10 monclient(hunting): start opening mon connection 2023-09-30T01:35:18.644 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:18.626+0000 7f3c2e4ca700 1 -- --> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] -- mon_getmap magic: 0 v1 -- 0x7f3c2806ae10 con 0x7f3c2809d5f0 2023-09-30T01:35:18.644 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:18.626+0000 7f3c2e4ca700 10 monclient(hunting): start opening mon connection 2023-09-30T01:35:18.644 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:18.626+0000 7f3c2e4ca700 1 -- --> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] -- mon_getmap magic: 0 v1 -- 0x7f3c2809a5a0 con 0x7f3c2806e870 2023-09-30T01:35:18.644 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:18.626+0000 7f3c2e4ca700 10 monclient(hunting): start opening mon connection 2023-09-30T01:35:18.644 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:18.626+0000 7f3c2e4ca700 1 -- --> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] -- mon_getmap magic: 0 v1 -- 0x7f3c2809a2c0 con 0x7f3c2809df00 2023-09-30T01:35:18.644 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:18.626+0000 7f3c2e4ca700 10 monclient(hunting): _renew_subs 2023-09-30T01:35:18.644 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:18.626+0000 7f3c2e4ca700 10 monclient(hunting): authenticate will time out at 990.665621s 2023-09-30T01:35:18.644 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:18.626+0000 7f3c277fe700 1 --2- >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f3c2809d5f0 0x7f3c28097f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:35:18.644 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:18.626+0000 7f3c2ca67700 1 --2- >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f3c2809df00 0x7f3c2809bdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:35:18.644 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:18.626+0000 7f3c277fe700 1 --2- >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f3c2809d5f0 0x7f3c28097f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3302/0 says I am v2:172.21.15.88:42594/0 (socket says 172.21.15.88:42594) 2023-09-30T01:35:18.644 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:18.626+0000 7f3c277fe700 1 -- 172.21.15.88:0/3625700547 learned_addr learned my addr 172.21.15.88:0/3625700547 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-09-30T01:35:18.645 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:18.626+0000 7f3c277fe700 10 monclient(hunting): get_auth_request con 0x7f3c2809d5f0 auth_method 0 2023-09-30T01:35:18.645 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:18.626+0000 7f3c277fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:35:18.645 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:18.626+0000 7f3c277fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:35:18.645 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:18.626+0000 7f3c277fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:35:18.645 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:18.626+0000 7f3c2ca67700 10 monclient(hunting): get_auth_request con 0x7f3c2809df00 auth_method 0 2023-09-30T01:35:18.645 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:18.626+0000 7f3c2ca67700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:35:18.645 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:18.626+0000 7f3c2ca67700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:35:18.645 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:18.626+0000 7f3c2ca67700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:35:18.645 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:18.626+0000 7f3c277fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:35:18.645 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:18.626+0000 7f3c277fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:35:18.645 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:18.626+0000 7f3c27fff700 1 --2- 172.21.15.88:0/3625700547 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7f3c2806e870 0x7f3c281368c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:35:18.645 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:18.626+0000 7f3c277fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:35:18.646 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:18.626+0000 7f3c2ca67700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:35:18.646 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:18.626+0000 7f3c2ca67700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:35:18.646 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:18.626+0000 7f3c2ca67700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:35:18.646 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:18.626+0000 7f3c27fff700 10 monclient(hunting): get_auth_request con 0x7f3c2806e870 auth_method 0 2023-09-30T01:35:18.646 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:18.626+0000 7f3c27fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:35:18.646 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:18.626+0000 7f3c27fff700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:35:18.646 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:18.626+0000 7f3c27fff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:35:18.646 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:18.626+0000 7f3c27fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:35:18.646 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:18.626+0000 7f3c27fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:35:18.646 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:18.626+0000 7f3c27fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:35:18.646 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:18.626+0000 7f3c277fe700 10 monclient(hunting): handle_auth_done global_id 4594 payload 824 2023-09-30T01:35:18.646 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:18.626+0000 7f3c277fe700 1 -- 172.21.15.88:0/3625700547 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f3c2809df00 msgr2=0x7f3c2809bdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:35:18.647 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:18.626+0000 7f3c277fe700 1 --2- 172.21.15.88:0/3625700547 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f3c2809df00 0x7f3c2809bdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:35:18.647 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:18.626+0000 7f3c277fe700 1 -- 172.21.15.88:0/3625700547 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7f3c2806e870 msgr2=0x7f3c281368c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:35:18.647 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:18.626+0000 7f3c277fe700 1 --2- 172.21.15.88:0/3625700547 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7f3c2806e870 0x7f3c281368c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:35:18.647 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:18.626+0000 7f3c277fe700 10 monclient: _finish_hunting 0 2023-09-30T01:35:18.647 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:18.626+0000 7f3c277fe700 1 monclient: found mon.noname-c 2023-09-30T01:35:18.647 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:18.626+0000 7f3c277fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:35:18.647 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:18.626+0000 7f3c277fe700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.88:3302/0 2023-09-30T01:35:18.647 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:18.626+0000 7f3c277fe700 1 -- 172.21.15.88:0/3625700547 --> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c200094f0 con 0x7f3c2809d5f0 2023-09-30T01:35:18.647 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:18.626+0000 7f3c277fe700 1 --2- 172.21.15.88:0/3625700547 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f3c2809d5f0 0x7f3c28097f10 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f3c1c00c860 tx=0x7f3c1c008040 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:35:18.647 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:18.626+0000 7f3c0ffff700 1 -- 172.21.15.88:0/3625700547 <== mon.4 v2:172.21.15.88:3302/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f3c1c011a80 con 0x7f3c2809d5f0 2023-09-30T01:35:18.647 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:18.626+0000 7f3c0ffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:35:18.647 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:18.626+0000 7f3c0ffff700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T01:35:18.648 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:18.626+0000 7f3c0ffff700 10 monclient: dump: 2023-09-30T01:35:18.648 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:35:18.648 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:35:18.648 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:35:18.648 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:35:18.648 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:35:18.648 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:35:18.648 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:35:18.648 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:35:18.648 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:35:18.648 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:35:18.648 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:35:18.648 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:35:18.649 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:35:18.649 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:35:18.649 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:35:18.649 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:35:18.649 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:18.626+0000 7f3c0ffff700 10 monclient: _finish_auth 0 2023-09-30T01:35:18.649 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:18.626+0000 7f3c0ffff700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:35:18.649 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:18.626+0000 7f3c2e4ca700 5 monclient: authenticate success, global_id 4594 2023-09-30T01:35:18.649 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:18.626+0000 7f3c0ffff700 1 -- 172.21.15.88:0/3625700547 <== mon.4 v2:172.21.15.88:3302/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c1c011c00 con 0x7f3c2809d5f0 2023-09-30T01:35:18.649 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:18.626+0000 7f3c0ffff700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:35:18.649 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:18.626+0000 7f3c0ffff700 1 -- 172.21.15.88:0/3625700547 <== mon.4 v2:172.21.15.88:3302/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f3c1c011d70 con 0x7f3c2809d5f0 2023-09-30T01:35:18.649 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:18.626+0000 7f3c2e4ca700 10 monclient: _renew_subs 2023-09-30T01:35:18.649 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:18.626+0000 7f3c2e4ca700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.88:3302/0 2023-09-30T01:35:18.649 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:18.626+0000 7f3c2e4ca700 1 -- 172.21.15.88:0/3625700547 --> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c28098c80 con 0x7f3c2809d5f0 2023-09-30T01:35:18.650 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:18.626+0000 7f3c0ffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:35:18.650 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:18.626+0000 7f3c0ffff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T01:35:18.650 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:18.626+0000 7f3c267fc700 4 set_mon_vals no callback set 2023-09-30T01:35:18.650 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:18.626+0000 7f3c0ffff700 10 monclient: dump: 2023-09-30T01:35:18.650 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:35:18.650 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:35:18.650 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:35:18.650 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:35:18.650 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:35:18.650 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:35:18.650 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:35:18.651 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:35:18.651 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:35:18.651 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:35:18.651 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:35:18.651 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:35:18.651 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:35:18.651 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:35:18.651 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:35:18.651 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:35:18.651 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:18.626+0000 7f3c2e4ca700 10 monclient: _renew_subs 2023-09-30T01:35:18.651 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:18.626+0000 7f3c2e4ca700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.88:3302/0 2023-09-30T01:35:18.651 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:18.626+0000 7f3c2e4ca700 1 -- 172.21.15.88:0/3625700547 --> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c28137510 con 0x7f3c2809d5f0 2023-09-30T01:35:18.651 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:18.630+0000 7f3c0ffff700 1 -- 172.21.15.88:0/3625700547 <== mon.4 v2:172.21.15.88:3302/0 4 ==== mgrmap(e 5) v1 ==== 40144+0+0 (secure 0 0 0) 0x7f3c1c011600 con 0x7f3c2809d5f0 2023-09-30T01:35:18.652 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:18.630+0000 7f3c2e4ca700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:35:18.652 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:18.630+0000 7f3c2e4ca700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:35:18.652 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:18.630+0000 7f3c2e4ca700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.88:3302/0 2023-09-30T01:35:18.652 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:18.630+0000 7f3c2e4ca700 1 -- 172.21.15.88:0/3625700547 --> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c28071960 con 0x7f3c2809d5f0 2023-09-30T01:35:18.652 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:18.630+0000 7f3c0ffff700 1 --2- 172.21.15.88:0/3625700547 >> v2:172.21.15.107:6812/29926 conn(0x7f3c14033390 0x7f3c14035840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:35:18.652 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:18.630+0000 7f3c0ffff700 1 -- 172.21.15.88:0/3625700547 <== mon.4 v2:172.21.15.88:3302/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 3502+0+0 (secure 0 0 0) 0x7f3c1c0432f0 con 0x7f3c2809d5f0 2023-09-30T01:35:18.652 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:18.630+0000 7f3c27fff700 1 --2- 172.21.15.88:0/3625700547 >> v2:172.21.15.107:6812/29926 conn(0x7f3c14033390 0x7f3c14035840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:35:18.652 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:18.630+0000 7f3c27fff700 10 monclient: get_auth_request con 0x7f3c14033390 auth_method 0 2023-09-30T01:35:18.652 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:18.630+0000 7f3c27fff700 1 --2- 172.21.15.88:0/3625700547 >> v2:172.21.15.107:6812/29926 conn(0x7f3c14033390 0x7f3c14035840 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f3c1800f570 tx=0x7f3c1800b040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:35:18.653 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:18.638+0000 7f3c0ffff700 1 -- 172.21.15.88:0/3625700547 <== mon.4 v2:172.21.15.88:3302/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f3c1c0118b0 con 0x7f3c2809d5f0 2023-09-30T01:35:18.653 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:18.638+0000 7f3c0ffff700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:35:18.653 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:18.638+0000 7f3c0ffff700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:35:18.836 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:18.834+0000 7f3c2e4ca700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T01:35:18.836 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:18.834+0000 7f3c2e4ca700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:35:18.836 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:18.834+0000 7f3c2e4ca700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.88:3302/0 2023-09-30T01:35:18.836 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:18.834+0000 7f3c2e4ca700 1 -- 172.21.15.88:0/3625700547 --> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f3c28064b80 con 0x7f3c2809d5f0 2023-09-30T01:35:18.836 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:18.834+0000 7f3c0ffff700 1 -- 172.21.15.88:0/3625700547 <== mon.4 v2:172.21.15.88:3302/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f3c1c021300 con 0x7f3c2809d5f0 2023-09-30T01:35:18.837 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:18.834+0000 7f3c0ffff700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:35:18.837 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:18.834+0000 7f3c0ffff700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:35:18.837 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-09-30T01:35:18.837 INFO:teuthology.orchestra.run.smithi088.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:35:18.839 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:18.834+0000 7f3c2e4ca700 1 -- 172.21.15.88:0/3625700547 >> v2:172.21.15.107:6812/29926 conn(0x7f3c14033390 msgr2=0x7f3c14035840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:35:18.839 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:18.834+0000 7f3c2e4ca700 1 --2- 172.21.15.88:0/3625700547 >> v2:172.21.15.107:6812/29926 conn(0x7f3c14033390 0x7f3c14035840 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f3c1800f570 tx=0x7f3c1800b040 comp rx=0 tx=0).stop 2023-09-30T01:35:18.839 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:18.834+0000 7f3c2e4ca700 10 monclient: shutdown 2023-09-30T01:35:18.839 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:18.834+0000 7f3c2e4ca700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:35:18.839 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:18.834+0000 7f3c2e4ca700 1 -- 172.21.15.88:0/3625700547 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f3c2809d5f0 msgr2=0x7f3c28097f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:35:18.839 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:18.834+0000 7f3c2e4ca700 1 --2- 172.21.15.88:0/3625700547 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f3c2809d5f0 0x7f3c28097f10 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f3c1c00c860 tx=0x7f3c1c008040 comp rx=0 tx=0).stop 2023-09-30T01:35:18.839 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:18.834+0000 7f3c2e4ca700 1 -- 172.21.15.88:0/3625700547 shutdown_connections 2023-09-30T01:35:18.839 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:18.834+0000 7f3c2e4ca700 1 --2- 172.21.15.88:0/3625700547 >> v2:172.21.15.107:6812/29926 conn(0x7f3c14033390 0x7f3c14035840 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:35:18.839 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:18.834+0000 7f3c2e4ca700 1 --2- 172.21.15.88:0/3625700547 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f3c2809df00 0x7f3c2809bdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:35:18.839 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:18.834+0000 7f3c2e4ca700 1 --2- 172.21.15.88:0/3625700547 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f3c2809d5f0 0x7f3c28097f10 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:35:18.840 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:18.834+0000 7f3c2e4ca700 1 --2- 172.21.15.88:0/3625700547 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7f3c2806e870 0x7f3c281368c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:35:18.840 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:18.834+0000 7f3c2e4ca700 1 -- 172.21.15.88:0/3625700547 >> 172.21.15.88:0/3625700547 conn(0x7f3c280a01d0 msgr2=0x7f3c2806d0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:35:18.840 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:18.834+0000 7f3c2e4ca700 1 -- 172.21.15.88:0/3625700547 shutdown_connections 2023-09-30T01:35:18.840 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:18.834+0000 7f3c2e4ca700 1 -- 172.21.15.88:0/3625700547 wait complete. 2023-09-30T01:35:21.850 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:35:21.962 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:21.958+0000 7f849a889700 10 monclient: get_monmap_and_config 2023-09-30T01:35:21.962 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:21.958+0000 7f849a889700 10 monclient: build_initial_monmap 2023-09-30T01:35:21.962 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:21.958+0000 7f849a889700 10 monclient: monmap: 2023-09-30T01:35:21.962 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2023-09-30T01:35:21.962 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:35:21.963 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:35:21.962411+0000 2023-09-30T01:35:21.963 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:35:21.962411+0000 2023-09-30T01:35:21.963 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2023-09-30T01:35:21.963 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:35:21.963 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a-legacy 2023-09-30T01:35:21.963 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.107:6789/0 mon.noname-f-legacy 2023-09-30T01:35:21.963 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.noname-b 2023-09-30T01:35:21.963 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.noname-g 2023-09-30T01:35:21.963 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.noname-c 2023-09-30T01:35:21.963 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.noname-h 2023-09-30T01:35:21.963 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.noname-d 2023-09-30T01:35:21.963 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.noname-i 2023-09-30T01:35:21.964 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.noname-e 2023-09-30T01:35:21.964 INFO:teuthology.orchestra.run.smithi088.stderr:9: v2:172.21.15.88:3300/0 mon.noname-a 2023-09-30T01:35:21.964 INFO:teuthology.orchestra.run.smithi088.stderr:10: v2:172.21.15.107:3300/0 mon.noname-f 2023-09-30T01:35:21.964 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:35:21.972 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:21.958+0000 7f849a889700 1 Processor -- start 2023-09-30T01:35:21.972 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:21.958+0000 7f849a889700 1 -- start start 2023-09-30T01:35:21.972 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:21.958+0000 7f849a889700 10 monclient: init 2023-09-30T01:35:21.972 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:21.958+0000 7f849a889700 10 monclient: _reopen_session rank -1 2023-09-30T01:35:21.972 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:21.958+0000 7f849a889700 10 monclient: _add_conns ranks=[0,9,10,4,5,1,6,8,7,2,3] 2023-09-30T01:35:21.972 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:21.958+0000 7f849a889700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f84940976f0 addr v1:172.21.15.88:6789/0 2023-09-30T01:35:21.972 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:21.958+0000 7f849a889700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f8494098de0 0x7f849409f490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:35:21.973 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:21.958+0000 7f849a889700 10 monclient(hunting): picked mon.noname-a con 0x7f8494098de0 addr v2:172.21.15.88:3300/0 2023-09-30T01:35:21.973 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:21.958+0000 7f849a889700 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f84941357b0 0x7f8494137b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:35:21.973 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:21.958+0000 7f849a889700 10 monclient(hunting): picked mon.noname-f con 0x7f84941357b0 addr v2:172.21.15.107:3300/0 2023-09-30T01:35:21.973 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:21.958+0000 7f849a889700 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f849404bb20 con 0x7f84940976f0 2023-09-30T01:35:21.973 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:21.958+0000 7f849a889700 10 monclient(hunting): start opening mon connection 2023-09-30T01:35:21.973 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:21.958+0000 7f849a889700 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8494098a30 con 0x7f8494098de0 2023-09-30T01:35:21.973 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:21.958+0000 7f849a889700 10 monclient(hunting): start opening mon connection 2023-09-30T01:35:21.973 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:21.958+0000 7f849a889700 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8494097ac0 con 0x7f84941357b0 2023-09-30T01:35:21.973 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:21.958+0000 7f849a889700 10 monclient(hunting): _renew_subs 2023-09-30T01:35:21.973 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:21.958+0000 7f849a889700 10 monclient(hunting): authenticate will time out at 993.999856s 2023-09-30T01:35:21.973 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:21.962+0000 7f8499086700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f8494098de0 0x7f849409f490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:35:21.973 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:21.962+0000 7f8499086700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f8494098de0 0x7f849409f490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:48372/0 (socket says 172.21.15.88:48372) 2023-09-30T01:35:21.974 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:21.962+0000 7f8499086700 1 -- 172.21.15.88:0/3865656198 learned_addr learned my addr 172.21.15.88:0/3865656198 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-09-30T01:35:21.974 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:21.962+0000 7f8499086700 10 monclient(hunting): get_auth_request con 0x7f8494098de0 auth_method 0 2023-09-30T01:35:21.974 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:21.962+0000 7f8499086700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:35:21.974 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:21.962+0000 7f8499086700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:35:21.974 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:21.962+0000 7f8499086700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:35:21.974 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:21.962+0000 7f8499086700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:35:21.974 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:21.962+0000 7f8499086700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:35:21.974 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:21.962+0000 7f8499086700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:35:21.974 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:21.962+0000 7f849a088700 1 --2- 172.21.15.88:0/3865656198 >> v2:172.21.15.107:3300/0 conn(0x7f84941357b0 0x7f8494137b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:35:21.974 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:21.962+0000 7f849a088700 10 monclient(hunting): get_auth_request con 0x7f84941357b0 auth_method 0 2023-09-30T01:35:21.974 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:21.962+0000 7f849a088700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:35:21.975 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:21.962+0000 7f849a088700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:35:21.975 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:21.962+0000 7f849a088700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:35:21.975 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:21.962+0000 7f8499086700 10 monclient(hunting): handle_auth_done global_id 4788 payload 274 2023-09-30T01:35:21.975 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:21.962+0000 7f8499086700 1 -- 172.21.15.88:0/3865656198 >> v2:172.21.15.107:3300/0 conn(0x7f84941357b0 msgr2=0x7f8494137b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:35:21.975 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:21.962+0000 7f8499086700 1 --2- 172.21.15.88:0/3865656198 >> v2:172.21.15.107:3300/0 conn(0x7f84941357b0 0x7f8494137b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:35:21.975 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:21.962+0000 7f8499086700 1 -- 172.21.15.88:0/3865656198 >> v1:172.21.15.88:6789/0 conn(0x7f84940976f0 legacy=0x7f849409ed80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:35:21.975 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:21.962+0000 7f8499086700 10 monclient: _finish_hunting 0 2023-09-30T01:35:21.975 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:21.962+0000 7f8499086700 1 monclient: found mon.noname-a 2023-09-30T01:35:21.975 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:21.962+0000 7f8499086700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:35:21.975 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:21.962+0000 7f8499086700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.88:3300/0 2023-09-30T01:35:21.975 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:21.962+0000 7f8499086700 1 -- 172.21.15.88:0/3865656198 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8494098120 con 0x7f8494098de0 2023-09-30T01:35:21.975 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:21.962+0000 7f8499086700 1 --2- 172.21.15.88:0/3865656198 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8494098de0 0x7f849409f490 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f849406ab40 tx=0x7f8490009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=713b936a37ab62a server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:35:21.976 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:21.962+0000 7f8498885700 1 -- 172.21.15.88:0/3865656198 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f849000e070 con 0x7f8494098de0 2023-09-30T01:35:21.976 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:21.962+0000 7f8498885700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:35:21.976 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:21.962+0000 7f8498885700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T01:35:21.976 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:21.962+0000 7f8498885700 10 monclient: dump: 2023-09-30T01:35:21.976 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:35:21.976 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:35:21.976 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:35:21.976 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:35:21.976 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:35:21.977 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:35:21.977 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:35:21.977 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:35:21.977 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:35:21.977 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:35:21.977 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:35:21.977 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:35:21.977 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:35:21.977 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:35:21.977 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:35:21.977 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:35:21.977 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:21.962+0000 7f8498885700 10 monclient: _finish_auth 0 2023-09-30T01:35:21.977 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:21.962+0000 7f8498885700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:35:21.978 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:21.962+0000 7f849a889700 5 monclient: authenticate success, global_id 4788 2023-09-30T01:35:21.978 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:21.962+0000 7f849a889700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:35:21.978 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:21.962+0000 7f8498885700 1 -- 172.21.15.88:0/3865656198 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8490007400 con 0x7f8494098de0 2023-09-30T01:35:21.978 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:21.962+0000 7f8498885700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:35:21.978 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:21.962+0000 7f849a889700 10 monclient: get_monmap_and_config success 2023-09-30T01:35:21.978 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:21.962+0000 7f849a889700 4 set_mon_vals no callback set 2023-09-30T01:35:21.978 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:21.962+0000 7f8498885700 1 -- 172.21.15.88:0/3865656198 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f84900092e0 con 0x7f8494098de0 2023-09-30T01:35:21.978 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:21.962+0000 7f8498885700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:35:21.978 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:21.962+0000 7f8498885700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:35:21.978 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:21.962+0000 7f8498885700 10 monclient: dump: 2023-09-30T01:35:21.978 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:35:21.978 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:35:21.979 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:35:21.979 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:35:21.979 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:35:21.979 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:35:21.979 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:35:21.979 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:35:21.980 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:35:21.980 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:35:21.980 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:35:21.980 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:35:21.980 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:35:21.980 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:35:21.980 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:35:21.980 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:35:21.980 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:21.962+0000 7f849a889700 10 monclient: shutdown 2023-09-30T01:35:21.980 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:21.962+0000 7f849a889700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:35:21.980 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:21.962+0000 7f849a889700 1 -- 172.21.15.88:0/3865656198 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8494098de0 msgr2=0x7f849409f490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:35:21.981 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:21.962+0000 7f849a889700 1 --2- 172.21.15.88:0/3865656198 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8494098de0 0x7f849409f490 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f849406ab40 tx=0x7f8490009840 comp rx=0 tx=0).stop 2023-09-30T01:35:21.981 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:21.962+0000 7f849a889700 1 -- 172.21.15.88:0/3865656198 shutdown_connections 2023-09-30T01:35:21.981 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:21.962+0000 7f849a889700 1 --2- 172.21.15.88:0/3865656198 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8494098de0 0x7f849409f490 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:35:21.981 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:21.962+0000 7f849a889700 1 --2- 172.21.15.88:0/3865656198 >> v2:172.21.15.107:3300/0 conn(0x7f84941357b0 0x7f8494137b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:35:21.981 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:21.962+0000 7f849a889700 1 -- 172.21.15.88:0/3865656198 >> 172.21.15.88:0/3865656198 conn(0x7f84940a0910 msgr2=0x7f84940a33c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:35:21.981 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:21.962+0000 7f849a889700 1 -- 172.21.15.88:0/3865656198 shutdown_connections 2023-09-30T01:35:21.981 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:21.962+0000 7f849a889700 1 -- 172.21.15.88:0/3865656198 wait complete. 2023-09-30T01:35:21.981 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:21.962+0000 7f849a889700 10 monclient: build_initial_monmap 2023-09-30T01:35:21.982 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:21.962+0000 7f849a889700 10 monclient: monmap: 2023-09-30T01:35:21.982 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2023-09-30T01:35:21.982 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:35:21.982 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 0.000000 2023-09-30T01:35:21.982 INFO:teuthology.orchestra.run.smithi088.stderr:created 0.000000 2023-09-30T01:35:21.982 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2023-09-30T01:35:21.982 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 1 2023-09-30T01:35:21.982 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.noname-a 2023-09-30T01:35:21.983 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.noname-f 2023-09-30T01:35:21.983 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.noname-b 2023-09-30T01:35:21.983 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.noname-g 2023-09-30T01:35:21.983 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.noname-c 2023-09-30T01:35:21.983 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.noname-h 2023-09-30T01:35:21.983 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.noname-d 2023-09-30T01:35:21.983 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.noname-i 2023-09-30T01:35:21.983 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.noname-e 2023-09-30T01:35:21.983 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:35:21.983 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:21.962+0000 7f849a889700 1 Processor -- start 2023-09-30T01:35:21.984 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:21.962+0000 7f849a889700 1 -- start start 2023-09-30T01:35:21.984 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:21.962+0000 7f849a889700 10 monclient: init 2023-09-30T01:35:21.984 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:21.966+0000 7f849a889700 10 monclient: _reopen_session rank -1 2023-09-30T01:35:21.984 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:21.966+0000 7f849a889700 10 monclient: _add_conns ranks=[0,5,2,1,4,7,3,6,8] 2023-09-30T01:35:21.984 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:21.966+0000 7f849a889700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f84940976f0 0x7f84941345b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:35:21.984 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:21.966+0000 7f849a889700 10 monclient(hunting): picked mon.noname-a con 0x7f84940976f0 addr [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] 2023-09-30T01:35:21.984 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:21.966+0000 7f849a889700 1 --2- >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7f8494098de0 0x7f8494134af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:35:21.984 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:21.966+0000 7f849a889700 10 monclient(hunting): picked mon.noname-h con 0x7f8494098de0 addr [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] 2023-09-30T01:35:21.984 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:21.966+0000 7f849a889700 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f84941357b0 0x7f8494135030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:35:21.984 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:21.966+0000 7f849a889700 10 monclient(hunting): picked mon.noname-b con 0x7f84941357b0 addr [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] 2023-09-30T01:35:21.984 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:21.966+0000 7f849a889700 10 monclient(hunting): start opening mon connection 2023-09-30T01:35:21.985 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:21.966+0000 7f849a889700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8494094720 con 0x7f84940976f0 2023-09-30T01:35:21.985 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:21.966+0000 7f849a889700 10 monclient(hunting): start opening mon connection 2023-09-30T01:35:21.985 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:21.966+0000 7f849a889700 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8494094440 con 0x7f84941357b0 2023-09-30T01:35:21.986 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:21.966+0000 7f849a889700 10 monclient(hunting): start opening mon connection 2023-09-30T01:35:21.986 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:21.966+0000 7f849a889700 1 -- --> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] -- mon_getmap magic: 0 v1 -- 0x7f8494094a00 con 0x7f8494098de0 2023-09-30T01:35:21.986 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:21.966+0000 7f849a889700 10 monclient(hunting): _renew_subs 2023-09-30T01:35:21.986 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:21.966+0000 7f849a889700 10 monclient(hunting): authenticate will time out at 994.004276s 2023-09-30T01:35:21.986 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:21.966+0000 7f8499887700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f84940976f0 0x7f84941345b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:35:21.986 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:21.966+0000 7f849a088700 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f84941357b0 0x7f8494135030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:35:21.986 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:21.966+0000 7f849a088700 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f84941357b0 0x7f8494135030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3301/0 says I am v2:172.21.15.88:37972/0 (socket says 172.21.15.88:37972) 2023-09-30T01:35:21.986 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:21.966+0000 7f849a088700 1 -- 172.21.15.88:0/3689808971 learned_addr learned my addr 172.21.15.88:0/3689808971 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-09-30T01:35:21.986 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:21.966+0000 7f849a088700 10 monclient(hunting): get_auth_request con 0x7f84941357b0 auth_method 0 2023-09-30T01:35:21.986 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:21.966+0000 7f849a088700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:35:21.986 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:21.966+0000 7f849a088700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:35:21.987 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:21.966+0000 7f849a088700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:35:21.987 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:21.966+0000 7f8499887700 10 monclient(hunting): get_auth_request con 0x7f84940976f0 auth_method 0 2023-09-30T01:35:21.987 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:21.966+0000 7f8499887700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:35:21.987 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:21.966+0000 7f8499887700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:35:21.987 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:21.966+0000 7f8499887700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:35:21.987 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:21.966+0000 7f849a088700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:35:21.987 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:21.966+0000 7f849a088700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:35:21.987 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:21.966+0000 7f849a088700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:35:21.987 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:21.966+0000 7f8499887700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:35:21.987 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:21.966+0000 7f8499887700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:35:21.987 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:21.966+0000 7f8499887700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:35:21.987 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:21.966+0000 7f8499086700 1 --2- 172.21.15.88:0/3689808971 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7f8494098de0 0x7f8494134af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:35:21.988 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:21.966+0000 7f8499086700 10 monclient(hunting): get_auth_request con 0x7f8494098de0 auth_method 0 2023-09-30T01:35:21.988 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:21.966+0000 7f8499086700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:35:21.988 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:21.966+0000 7f8499086700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:35:21.988 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:21.966+0000 7f8499086700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:35:21.988 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:21.966+0000 7f849a088700 10 monclient(hunting): handle_auth_done global_id 4574 payload 824 2023-09-30T01:35:21.988 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:21.966+0000 7f849a088700 1 -- 172.21.15.88:0/3689808971 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7f8494098de0 msgr2=0x7f8494134af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:35:21.988 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:21.966+0000 7f849a088700 1 --2- 172.21.15.88:0/3689808971 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7f8494098de0 0x7f8494134af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:35:21.988 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:21.966+0000 7f849a088700 1 -- 172.21.15.88:0/3689808971 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f84940976f0 msgr2=0x7f84941345b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:35:21.988 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:21.966+0000 7f849a088700 1 --2- 172.21.15.88:0/3689808971 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f84940976f0 0x7f84941345b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:35:21.988 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:21.966+0000 7f849a088700 10 monclient: _finish_hunting 0 2023-09-30T01:35:21.988 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:21.966+0000 7f849a088700 1 monclient: found mon.noname-b 2023-09-30T01:35:21.988 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:21.966+0000 7f849a088700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:35:21.989 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:21.966+0000 7f849a088700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.88:3301/0 2023-09-30T01:35:21.989 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:21.966+0000 7f849a088700 1 -- 172.21.15.88:0/3689808971 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8490009ae0 con 0x7f84941357b0 2023-09-30T01:35:21.989 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:21.966+0000 7f849a088700 1 --2- 172.21.15.88:0/3689808971 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f84941357b0 0x7f8494135030 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f848c00d860 tx=0x7f848c008040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:35:21.989 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:21.966+0000 7f8489ffb700 1 -- 172.21.15.88:0/3689808971 <== mon.2 v2:172.21.15.88:3301/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f848c012a70 con 0x7f84941357b0 2023-09-30T01:35:21.989 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:21.966+0000 7f8489ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:35:21.989 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:21.966+0000 7f8489ffb700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T01:35:21.989 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:21.966+0000 7f8489ffb700 10 monclient: dump: 2023-09-30T01:35:21.989 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:35:21.989 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:35:21.989 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:35:21.989 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:35:21.989 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:35:21.990 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:35:21.990 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:35:21.990 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:35:21.990 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:35:21.990 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:35:21.990 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:35:21.990 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:35:21.990 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:35:21.990 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:35:21.990 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:35:21.990 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:35:21.991 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:21.966+0000 7f8489ffb700 10 monclient: _finish_auth 0 2023-09-30T01:35:21.991 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:21.966+0000 7f8489ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:35:21.991 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:21.966+0000 7f849a889700 5 monclient: authenticate success, global_id 4574 2023-09-30T01:35:21.991 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:21.966+0000 7f8489ffb700 1 -- 172.21.15.88:0/3689808971 <== mon.2 v2:172.21.15.88:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f848c012c30 con 0x7f84941357b0 2023-09-30T01:35:21.991 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:21.966+0000 7f849a889700 10 monclient: _renew_subs 2023-09-30T01:35:21.991 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:21.966+0000 7f849a889700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.88:3301/0 2023-09-30T01:35:21.991 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:21.966+0000 7f849a889700 1 -- 172.21.15.88:0/3689808971 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84940934e0 con 0x7f84941357b0 2023-09-30T01:35:21.991 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:21.966+0000 7f8489ffb700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:35:21.991 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:21.966+0000 7f8489ffb700 1 -- 172.21.15.88:0/3689808971 <== mon.2 v2:172.21.15.88:3301/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f848c012da0 con 0x7f84941357b0 2023-09-30T01:35:21.991 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:21.966+0000 7f8489ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:35:21.991 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:21.966+0000 7f8489ffb700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T01:35:21.991 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:21.966+0000 7f848b7fe700 4 set_mon_vals no callback set 2023-09-30T01:35:21.992 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:21.966+0000 7f8489ffb700 10 monclient: dump: 2023-09-30T01:35:21.992 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:35:21.992 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:35:21.992 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:35:21.992 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:35:21.992 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:35:21.992 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:35:21.992 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:35:21.992 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:35:21.992 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:35:21.992 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:35:21.992 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:35:21.993 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:35:21.993 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:35:21.993 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:35:21.993 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:35:21.993 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:35:21.993 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:21.966+0000 7f849a889700 10 monclient: _renew_subs 2023-09-30T01:35:21.993 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:21.966+0000 7f849a889700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.88:3301/0 2023-09-30T01:35:21.993 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:21.966+0000 7f849a889700 1 -- 172.21.15.88:0/3689808971 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8494096fb0 con 0x7f84941357b0 2023-09-30T01:35:21.993 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:21.966+0000 7f849a889700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:35:21.993 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:21.966+0000 7f849a889700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:35:21.993 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:21.966+0000 7f849a889700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.88:3301/0 2023-09-30T01:35:21.994 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:21.966+0000 7f849a889700 1 -- 172.21.15.88:0/3689808971 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8494094bd0 con 0x7f84941357b0 2023-09-30T01:35:21.994 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:21.966+0000 7f8489ffb700 1 -- 172.21.15.88:0/3689808971 <== mon.2 v2:172.21.15.88:3301/0 4 ==== mgrmap(e 5) v1 ==== 40144+0+0 (secure 0 0 0) 0x7f848c0125f0 con 0x7f84941357b0 2023-09-30T01:35:21.994 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:21.966+0000 7f8489ffb700 1 --2- 172.21.15.88:0/3689808971 >> v2:172.21.15.107:6812/29926 conn(0x7f8478033360 0x7f8478035810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:35:21.994 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:21.966+0000 7f8489ffb700 1 -- 172.21.15.88:0/3689808971 <== mon.2 v2:172.21.15.88:3301/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 3502+0+0 (secure 0 0 0) 0x7f848c043a00 con 0x7f84941357b0 2023-09-30T01:35:21.994 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:21.966+0000 7f8499887700 1 --2- 172.21.15.88:0/3689808971 >> v2:172.21.15.107:6812/29926 conn(0x7f8478033360 0x7f8478035810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:35:21.994 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:21.970+0000 7f8499887700 10 monclient: get_auth_request con 0x7f8478033360 auth_method 0 2023-09-30T01:35:21.994 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:21.970+0000 7f8499887700 1 --2- 172.21.15.88:0/3689808971 >> v2:172.21.15.107:6812/29926 conn(0x7f8478033360 0x7f8478035810 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f8484000c00 tx=0x7f8484007e00 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:35:21.995 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:21.978+0000 7f8489ffb700 1 -- 172.21.15.88:0/3689808971 <== mon.2 v2:172.21.15.88:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f8494094bd0 con 0x7f84941357b0 2023-09-30T01:35:21.995 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:21.978+0000 7f8489ffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:35:21.995 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:21.978+0000 7f8489ffb700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:35:22.175 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:22.174+0000 7f849a889700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T01:35:22.176 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:22.174+0000 7f849a889700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:35:22.176 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:22.174+0000 7f849a889700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.88:3301/0 2023-09-30T01:35:22.176 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:22.174+0000 7f849a889700 1 -- 172.21.15.88:0/3689808971 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f8494094bd0 con 0x7f84941357b0 2023-09-30T01:35:22.176 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:22.174+0000 7f8489ffb700 1 -- 172.21.15.88:0/3689808971 <== mon.2 v2:172.21.15.88:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f8494094bd0 con 0x7f84941357b0 2023-09-30T01:35:22.176 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:22.174+0000 7f8489ffb700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:35:22.176 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:22.174+0000 7f8489ffb700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:35:22.177 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-09-30T01:35:22.177 INFO:teuthology.orchestra.run.smithi088.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:35:22.179 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:22.174+0000 7f849a889700 1 -- 172.21.15.88:0/3689808971 >> v2:172.21.15.107:6812/29926 conn(0x7f8478033360 msgr2=0x7f8478035810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:35:22.179 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:22.174+0000 7f849a889700 1 --2- 172.21.15.88:0/3689808971 >> v2:172.21.15.107:6812/29926 conn(0x7f8478033360 0x7f8478035810 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f8484000c00 tx=0x7f8484007e00 comp rx=0 tx=0).stop 2023-09-30T01:35:22.179 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:22.174+0000 7f849a889700 10 monclient: shutdown 2023-09-30T01:35:22.179 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:22.174+0000 7f849a889700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:35:22.179 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:22.174+0000 7f849a889700 1 -- 172.21.15.88:0/3689808971 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f84941357b0 msgr2=0x7f8494135030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:35:22.179 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:22.174+0000 7f849a889700 1 --2- 172.21.15.88:0/3689808971 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f84941357b0 0x7f8494135030 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f848c00d860 tx=0x7f848c008040 comp rx=0 tx=0).stop 2023-09-30T01:35:22.179 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:22.174+0000 7f849a889700 1 -- 172.21.15.88:0/3689808971 shutdown_connections 2023-09-30T01:35:22.179 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:22.174+0000 7f849a889700 1 --2- 172.21.15.88:0/3689808971 >> v2:172.21.15.107:6812/29926 conn(0x7f8478033360 0x7f8478035810 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:35:22.179 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:22.174+0000 7f849a889700 1 --2- 172.21.15.88:0/3689808971 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f84941357b0 0x7f8494135030 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:35:22.179 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:22.174+0000 7f849a889700 1 --2- 172.21.15.88:0/3689808971 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7f8494098de0 0x7f8494134af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:35:22.179 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:22.174+0000 7f849a889700 1 --2- 172.21.15.88:0/3689808971 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f84940976f0 0x7f84941345b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:35:22.180 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:22.174+0000 7f849a889700 1 -- 172.21.15.88:0/3689808971 >> 172.21.15.88:0/3689808971 conn(0x7f84940a0910 msgr2=0x7f849409ec00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:35:22.180 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:22.174+0000 7f849a889700 1 -- 172.21.15.88:0/3689808971 shutdown_connections 2023-09-30T01:35:22.180 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:22.174+0000 7f849a889700 1 -- 172.21.15.88:0/3689808971 wait complete. 2023-09-30T01:35:25.193 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:35:25.305 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:25.302+0000 7f36b6b06700 10 monclient: get_monmap_and_config 2023-09-30T01:35:25.305 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:25.302+0000 7f36b6b06700 10 monclient: build_initial_monmap 2023-09-30T01:35:25.305 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:25.302+0000 7f36b6b06700 10 monclient: monmap: 2023-09-30T01:35:25.306 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2023-09-30T01:35:25.306 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:35:25.306 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:35:25.305132+0000 2023-09-30T01:35:25.306 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:35:25.305132+0000 2023-09-30T01:35:25.306 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2023-09-30T01:35:25.306 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:35:25.306 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a-legacy 2023-09-30T01:35:25.306 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.107:6789/0 mon.noname-f-legacy 2023-09-30T01:35:25.306 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.noname-b 2023-09-30T01:35:25.306 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.noname-g 2023-09-30T01:35:25.306 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.noname-c 2023-09-30T01:35:25.307 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.noname-h 2023-09-30T01:35:25.307 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.noname-d 2023-09-30T01:35:25.307 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.noname-i 2023-09-30T01:35:25.307 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.noname-e 2023-09-30T01:35:25.307 INFO:teuthology.orchestra.run.smithi088.stderr:9: v2:172.21.15.88:3300/0 mon.noname-a 2023-09-30T01:35:25.307 INFO:teuthology.orchestra.run.smithi088.stderr:10: v2:172.21.15.107:3300/0 mon.noname-f 2023-09-30T01:35:25.307 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:35:25.307 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:25.302+0000 7f36b6b06700 1 Processor -- start 2023-09-30T01:35:25.307 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:25.302+0000 7f36b6b06700 1 -- start start 2023-09-30T01:35:25.307 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:25.302+0000 7f36b6b06700 10 monclient: init 2023-09-30T01:35:25.311 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:25.302+0000 7f36b6b06700 10 monclient: _reopen_session rank -1 2023-09-30T01:35:25.311 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:25.302+0000 7f36b6b06700 10 monclient: _add_conns ranks=[2,4,3,0,1,6,8,10,9,5,7] 2023-09-30T01:35:25.311 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:25.302+0000 7f36b6b06700 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f36b00a3380 0x7f36b00a48b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:35:25.311 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:25.302+0000 7f36b6b06700 10 monclient(hunting): picked mon.noname-b con 0x7f36b00a3380 addr [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] 2023-09-30T01:35:25.311 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:25.302+0000 7f36b6b06700 1 --2- >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f36b009ffd0 0x7f36b00a03a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:35:25.311 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:25.302+0000 7f36b6b06700 10 monclient(hunting): picked mon.noname-c con 0x7f36b009ffd0 addr [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] 2023-09-30T01:35:25.311 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:25.302+0000 7f36b6b06700 1 --2- >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7f36b01333b0 0x7f36b0133800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:35:25.312 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:25.302+0000 7f36b6b06700 10 monclient(hunting): picked mon.noname-g con 0x7f36b01333b0 addr [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] 2023-09-30T01:35:25.312 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:25.302+0000 7f36b6b06700 10 monclient(hunting): start opening mon connection 2023-09-30T01:35:25.312 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:25.302+0000 7f36b6b06700 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f36b00a08e0 con 0x7f36b00a3380 2023-09-30T01:35:25.312 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:25.302+0000 7f36b6b06700 10 monclient(hunting): start opening mon connection 2023-09-30T01:35:25.312 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:25.302+0000 7f36b6b06700 1 -- --> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f36b0098480 con 0x7f36b01333b0 2023-09-30T01:35:25.312 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:25.302+0000 7f36b6b06700 10 monclient(hunting): start opening mon connection 2023-09-30T01:35:25.312 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:25.302+0000 7f36b6b06700 1 -- --> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] -- mon_getmap magic: 0 v1 -- 0x7f36b00a3750 con 0x7f36b009ffd0 2023-09-30T01:35:25.312 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:25.302+0000 7f36b6b06700 10 monclient(hunting): _renew_subs 2023-09-30T01:35:25.312 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:25.302+0000 7f36b6b06700 10 monclient(hunting): authenticate will time out at 997.342641s 2023-09-30T01:35:25.312 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:25.302+0000 7f36b5303700 1 --2- >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f36b009ffd0 0x7f36b00a03a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:35:25.312 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:25.302+0000 7f36b5b04700 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f36b00a3380 0x7f36b00a48b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:35:25.313 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:25.302+0000 7f36b5303700 1 --2- >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f36b009ffd0 0x7f36b00a03a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3302/0 says I am v2:172.21.15.88:43816/0 (socket says 172.21.15.88:43816) 2023-09-30T01:35:25.313 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:25.302+0000 7f36b5303700 1 -- 172.21.15.88:0/2809941207 learned_addr learned my addr 172.21.15.88:0/2809941207 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-09-30T01:35:25.313 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:25.302+0000 7f36b5303700 10 monclient(hunting): get_auth_request con 0x7f36b009ffd0 auth_method 0 2023-09-30T01:35:25.313 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:25.302+0000 7f36b5303700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:35:25.313 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:25.302+0000 7f36b5303700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:35:25.313 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:25.302+0000 7f36b5303700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:35:25.313 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:25.302+0000 7f36b5b04700 10 monclient(hunting): get_auth_request con 0x7f36b00a3380 auth_method 0 2023-09-30T01:35:25.313 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:25.302+0000 7f36b5b04700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:35:25.314 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:25.302+0000 7f36b5b04700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:35:25.314 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:25.302+0000 7f36b5b04700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:35:25.314 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:25.302+0000 7f36b5303700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:35:25.314 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:25.302+0000 7f36b5303700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:35:25.314 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:25.302+0000 7f36b5303700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:35:25.314 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:25.302+0000 7f36b6305700 1 --2- 172.21.15.88:0/2809941207 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7f36b01333b0 0x7f36b0133800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:35:25.314 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:25.302+0000 7f36b5b04700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:35:25.315 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:25.302+0000 7f36b5b04700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:35:25.315 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:25.302+0000 7f36b5b04700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:35:25.315 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:25.302+0000 7f36b6305700 10 monclient(hunting): get_auth_request con 0x7f36b01333b0 auth_method 0 2023-09-30T01:35:25.315 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:25.302+0000 7f36b6305700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:35:25.315 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:25.302+0000 7f36b6305700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:35:25.315 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:25.302+0000 7f36b6305700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:35:25.315 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:25.302+0000 7f36b5303700 10 monclient(hunting): handle_auth_done global_id 4603 payload 274 2023-09-30T01:35:25.315 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:25.302+0000 7f36b5303700 1 -- 172.21.15.88:0/2809941207 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7f36b01333b0 msgr2=0x7f36b0133800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:35:25.315 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:25.302+0000 7f36b5303700 1 --2- 172.21.15.88:0/2809941207 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7f36b01333b0 0x7f36b0133800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:35:25.316 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:25.302+0000 7f36b5303700 1 -- 172.21.15.88:0/2809941207 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f36b00a3380 msgr2=0x7f36b00a48b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:35:25.316 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:25.302+0000 7f36b5303700 1 --2- 172.21.15.88:0/2809941207 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f36b00a3380 0x7f36b00a48b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:35:25.316 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:25.302+0000 7f36b5303700 10 monclient: _finish_hunting 0 2023-09-30T01:35:25.316 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:25.302+0000 7f36b5303700 1 monclient: found mon.noname-c 2023-09-30T01:35:25.316 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:25.302+0000 7f36b5303700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:35:25.316 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:25.302+0000 7f36b5303700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.88:3302/0 2023-09-30T01:35:25.316 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:25.302+0000 7f36b5303700 1 -- 172.21.15.88:0/2809941207 --> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36b004bb20 con 0x7f36b009ffd0 2023-09-30T01:35:25.316 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:25.302+0000 7f36b6305700 1 --2- 172.21.15.88:0/2809941207 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7f36b01333b0 0x7f36b0133800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-30T01:35:25.316 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:25.302+0000 7f36b5303700 1 --2- 172.21.15.88:0/2809941207 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f36b009ffd0 0x7f36b00a03a0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f36ac0099e0 tx=0x7f36ac009250 comp rx=0 tx=0).ready entity=mon.4 client_cookie=237f2e1d575cd317 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:35:25.316 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:25.306+0000 7f36b4b02700 1 -- 172.21.15.88:0/2809941207 <== mon.4 v2:172.21.15.88:3302/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f36ac00bc80 con 0x7f36b009ffd0 2023-09-30T01:35:25.316 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:25.306+0000 7f36b4b02700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:35:25.316 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:25.306+0000 7f36b4b02700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T01:35:25.317 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:25.306+0000 7f36b4b02700 10 monclient: dump: 2023-09-30T01:35:25.317 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:35:25.317 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:35:25.317 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:35:25.317 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:35:25.317 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:35:25.317 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:35:25.317 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:35:25.317 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:35:25.317 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:35:25.317 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:35:25.317 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:35:25.318 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:35:25.318 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:35:25.318 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:35:25.318 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:35:25.318 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:35:25.318 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:25.306+0000 7f36b4b02700 10 monclient: _finish_auth 0 2023-09-30T01:35:25.318 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:25.306+0000 7f36b4b02700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:35:25.318 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:25.306+0000 7f36b4b02700 1 -- 172.21.15.88:0/2809941207 <== mon.4 v2:172.21.15.88:3302/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f36ac00be00 con 0x7f36b009ffd0 2023-09-30T01:35:25.318 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:25.306+0000 7f36b4b02700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:35:25.318 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:25.306+0000 7f36b4b02700 1 -- 172.21.15.88:0/2809941207 <== mon.4 v2:172.21.15.88:3302/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f36ac0118a0 con 0x7f36b009ffd0 2023-09-30T01:35:25.318 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:25.306+0000 7f36b6b06700 5 monclient: authenticate success, global_id 4603 2023-09-30T01:35:25.318 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:25.306+0000 7f36b6b06700 10 monclient: get_monmap_and_config success 2023-09-30T01:35:25.318 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:25.306+0000 7f36b6b06700 4 set_mon_vals no callback set 2023-09-30T01:35:25.319 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:25.306+0000 7f36b4b02700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:35:25.319 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:25.306+0000 7f36b4b02700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T01:35:25.319 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:25.306+0000 7f36b4b02700 10 monclient: dump: 2023-09-30T01:35:25.319 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:35:25.319 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:35:25.319 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:35:25.319 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:35:25.319 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:35:25.319 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:35:25.319 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:35:25.319 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:35:25.319 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:35:25.320 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:35:25.320 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:35:25.320 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:35:25.320 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:35:25.320 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:35:25.320 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:35:25.320 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:35:25.320 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:25.306+0000 7f36b6b06700 10 monclient: shutdown 2023-09-30T01:35:25.320 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:25.306+0000 7f36b6b06700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:35:25.320 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:25.306+0000 7f36b6b06700 1 -- 172.21.15.88:0/2809941207 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f36b009ffd0 msgr2=0x7f36b00a03a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:35:25.320 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:25.306+0000 7f36b6b06700 1 --2- 172.21.15.88:0/2809941207 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f36b009ffd0 0x7f36b00a03a0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f36ac0099e0 tx=0x7f36ac009250 comp rx=0 tx=0).stop 2023-09-30T01:35:25.320 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:25.306+0000 7f36b6b06700 1 -- 172.21.15.88:0/2809941207 shutdown_connections 2023-09-30T01:35:25.320 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:25.306+0000 7f36b6b06700 1 --2- 172.21.15.88:0/2809941207 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7f36b01333b0 0x7f36b0133800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:35:25.321 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:25.306+0000 7f36b6b06700 1 --2- 172.21.15.88:0/2809941207 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f36b009ffd0 0x7f36b00a03a0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:35:25.321 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:25.306+0000 7f36b6b06700 1 --2- 172.21.15.88:0/2809941207 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f36b00a3380 0x7f36b00a48b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:35:25.321 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:25.306+0000 7f36b6b06700 1 -- 172.21.15.88:0/2809941207 >> 172.21.15.88:0/2809941207 conn(0x7f36b006e5e0 msgr2=0x7f36b006e9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:35:25.321 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:25.306+0000 7f36b6b06700 1 -- 172.21.15.88:0/2809941207 shutdown_connections 2023-09-30T01:35:25.321 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:25.306+0000 7f36b6b06700 1 -- 172.21.15.88:0/2809941207 wait complete. 2023-09-30T01:35:25.321 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:25.306+0000 7f36b6b06700 10 monclient: build_initial_monmap 2023-09-30T01:35:25.321 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:25.306+0000 7f36b6b06700 10 monclient: monmap: 2023-09-30T01:35:25.321 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2023-09-30T01:35:25.321 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:35:25.321 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 0.000000 2023-09-30T01:35:25.321 INFO:teuthology.orchestra.run.smithi088.stderr:created 0.000000 2023-09-30T01:35:25.321 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2023-09-30T01:35:25.321 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 1 2023-09-30T01:35:25.322 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.noname-a 2023-09-30T01:35:25.322 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.noname-f 2023-09-30T01:35:25.322 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.noname-b 2023-09-30T01:35:25.322 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.noname-g 2023-09-30T01:35:25.322 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.noname-c 2023-09-30T01:35:25.322 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.noname-h 2023-09-30T01:35:25.322 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.noname-d 2023-09-30T01:35:25.322 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.noname-i 2023-09-30T01:35:25.322 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.noname-e 2023-09-30T01:35:25.322 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:35:25.322 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:25.306+0000 7f36b6b06700 1 Processor -- start 2023-09-30T01:35:25.322 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:25.306+0000 7f36b6b06700 1 -- start start 2023-09-30T01:35:25.323 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:25.306+0000 7f36b6b06700 10 monclient: init 2023-09-30T01:35:25.327 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:25.310+0000 7f36b6b06700 10 monclient: _reopen_session rank -1 2023-09-30T01:35:25.327 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:25.310+0000 7f36b6b06700 10 monclient: _add_conns ranks=[7,4,2,5,0,3,6,8,1] 2023-09-30T01:35:25.327 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:25.310+0000 7f36b6b06700 1 --2- >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7f36b009ffd0 0x7f36b009f510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:35:25.327 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:25.310+0000 7f36b6b06700 10 monclient(hunting): picked mon.noname-i con 0x7f36b009ffd0 addr [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] 2023-09-30T01:35:25.327 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:25.310+0000 7f36b6b06700 1 --2- >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f36b00a3380 0x7f36b011cf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:35:25.328 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:25.310+0000 7f36b6b06700 10 monclient(hunting): picked mon.noname-c con 0x7f36b00a3380 addr [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] 2023-09-30T01:35:25.328 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:25.310+0000 7f36b6b06700 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f36b01333b0 0x7f36b011d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:35:25.328 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:25.310+0000 7f36b6b06700 10 monclient(hunting): picked mon.noname-b con 0x7f36b01333b0 addr [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] 2023-09-30T01:35:25.328 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:25.310+0000 7f36b6b06700 10 monclient(hunting): start opening mon connection 2023-09-30T01:35:25.328 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:25.310+0000 7f36b6b06700 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f36b00a0ac0 con 0x7f36b01333b0 2023-09-30T01:35:25.328 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:25.310+0000 7f36b6b06700 10 monclient(hunting): start opening mon connection 2023-09-30T01:35:25.328 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:25.310+0000 7f36b6b06700 1 -- --> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] -- mon_getmap magic: 0 v1 -- 0x7f36b0097ce0 con 0x7f36b00a3380 2023-09-30T01:35:25.328 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:25.310+0000 7f36b6b06700 10 monclient(hunting): start opening mon connection 2023-09-30T01:35:25.328 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:25.310+0000 7f36b6b06700 1 -- --> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] -- mon_getmap magic: 0 v1 -- 0x7f36b011ff70 con 0x7f36b009ffd0 2023-09-30T01:35:25.328 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:25.310+0000 7f36b6b06700 10 monclient(hunting): _renew_subs 2023-09-30T01:35:25.328 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:25.310+0000 7f36b6b06700 10 monclient(hunting): authenticate will time out at 997.349164s 2023-09-30T01:35:25.329 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:25.310+0000 7f36b5303700 1 --2- >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f36b00a3380 0x7f36b011cf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:35:25.329 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:25.310+0000 7f36b5303700 1 --2- >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f36b00a3380 0x7f36b011cf60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3302/0 says I am v2:172.21.15.88:43826/0 (socket says 172.21.15.88:43826) 2023-09-30T01:35:25.329 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:25.310+0000 7f36b5303700 1 -- 172.21.15.88:0/1043378658 learned_addr learned my addr 172.21.15.88:0/1043378658 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-09-30T01:35:25.329 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:25.310+0000 7f36b5303700 10 monclient(hunting): get_auth_request con 0x7f36b00a3380 auth_method 0 2023-09-30T01:35:25.329 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:25.310+0000 7f36b5303700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:35:25.329 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:25.310+0000 7f36b5303700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:35:25.329 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:25.310+0000 7f36b5303700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:35:25.329 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:25.310+0000 7f36b6305700 1 --2- 172.21.15.88:0/1043378658 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f36b01333b0 0x7f36b011d4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:35:25.329 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:25.310+0000 7f36b5b04700 1 --2- 172.21.15.88:0/1043378658 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7f36b009ffd0 0x7f36b009f510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:35:25.329 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:25.310+0000 7f36b6305700 10 monclient(hunting): get_auth_request con 0x7f36b01333b0 auth_method 0 2023-09-30T01:35:25.329 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:25.310+0000 7f36b6305700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:35:25.329 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:25.310+0000 7f36b6305700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:35:25.329 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:25.310+0000 7f36b6305700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:35:25.330 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:25.310+0000 7f36b5b04700 10 monclient(hunting): get_auth_request con 0x7f36b009ffd0 auth_method 0 2023-09-30T01:35:25.330 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:25.310+0000 7f36b5b04700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:35:25.330 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:25.310+0000 7f36b5b04700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:35:25.330 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:25.310+0000 7f36b5b04700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:35:25.330 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:25.310+0000 7f36b5303700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:35:25.330 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:25.310+0000 7f36b5303700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:35:25.330 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:25.310+0000 7f36b5303700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:35:25.330 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:25.310+0000 7f36b6305700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:35:25.330 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:25.310+0000 7f36b6305700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:35:25.330 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:25.310+0000 7f36b6305700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:35:25.330 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:25.310+0000 7f36b5b04700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:35:25.330 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:25.310+0000 7f36b5b04700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:35:25.331 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:25.310+0000 7f36b5b04700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:35:25.331 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:25.310+0000 7f36b6305700 10 monclient(hunting): handle_auth_done global_id 4592 payload 824 2023-09-30T01:35:25.331 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:25.310+0000 7f36b6305700 1 -- 172.21.15.88:0/1043378658 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7f36b009ffd0 msgr2=0x7f36b009f510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:35:25.331 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:25.310+0000 7f36b6305700 1 --2- 172.21.15.88:0/1043378658 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7f36b009ffd0 0x7f36b009f510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:35:25.331 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:25.310+0000 7f36b6305700 1 -- 172.21.15.88:0/1043378658 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f36b00a3380 msgr2=0x7f36b011cf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:35:25.331 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:25.310+0000 7f36b6305700 1 --2- 172.21.15.88:0/1043378658 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f36b00a3380 0x7f36b011cf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:35:25.331 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:25.310+0000 7f36b6305700 10 monclient: _finish_hunting 0 2023-09-30T01:35:25.331 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:25.310+0000 7f36b6305700 1 monclient: found mon.noname-b 2023-09-30T01:35:25.331 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:25.310+0000 7f36b6305700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:35:25.331 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:25.310+0000 7f36b6305700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.88:3301/0 2023-09-30T01:35:25.331 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:25.310+0000 7f36b6305700 1 -- 172.21.15.88:0/1043378658 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36ac0094f0 con 0x7f36b01333b0 2023-09-30T01:35:25.331 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:25.310+0000 7f36b5303700 1 --2- 172.21.15.88:0/1043378658 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f36b00a3380 0x7f36b011cf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:35:25.331 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:25.310+0000 7f36b6305700 1 --2- 172.21.15.88:0/1043378658 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f36b01333b0 0x7f36b011d4a0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f36a8007f30 tx=0x7f36a8009910 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:35:25.332 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:25.310+0000 7f36a5ffb700 1 -- 172.21.15.88:0/1043378658 <== mon.2 v2:172.21.15.88:3301/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f36a8015400 con 0x7f36b01333b0 2023-09-30T01:35:25.332 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:25.310+0000 7f36a5ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:35:25.332 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:25.310+0000 7f36a5ffb700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T01:35:25.332 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:25.310+0000 7f36a5ffb700 10 monclient: dump: 2023-09-30T01:35:25.332 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:35:25.332 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:35:25.332 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:35:25.332 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:35:25.332 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:35:25.332 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:35:25.332 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:35:25.332 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:35:25.333 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:35:25.333 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:35:25.333 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:35:25.333 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:35:25.333 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:35:25.333 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:35:25.333 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:35:25.333 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:35:25.333 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:25.310+0000 7f36a5ffb700 10 monclient: _finish_auth 0 2023-09-30T01:35:25.333 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:25.310+0000 7f36a5ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:35:25.334 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:25.310+0000 7f36b6b06700 5 monclient: authenticate success, global_id 4592 2023-09-30T01:35:25.334 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:25.310+0000 7f36a5ffb700 1 -- 172.21.15.88:0/1043378658 <== mon.2 v2:172.21.15.88:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f36a800f040 con 0x7f36b01333b0 2023-09-30T01:35:25.334 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:25.310+0000 7f36b6b06700 10 monclient: _renew_subs 2023-09-30T01:35:25.334 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:25.310+0000 7f36b6b06700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.88:3301/0 2023-09-30T01:35:25.334 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:25.310+0000 7f36b6b06700 1 -- 172.21.15.88:0/1043378658 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f36b012fc90 con 0x7f36b01333b0 2023-09-30T01:35:25.334 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:25.310+0000 7f36a5ffb700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:35:25.334 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:25.310+0000 7f36a5ffb700 1 -- 172.21.15.88:0/1043378658 <== mon.2 v2:172.21.15.88:3301/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f36a8015a20 con 0x7f36b01333b0 2023-09-30T01:35:25.334 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:25.310+0000 7f36a5ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:35:25.334 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:25.310+0000 7f36a7fff700 4 set_mon_vals no callback set 2023-09-30T01:35:25.334 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:25.310+0000 7f36a5ffb700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T01:35:25.334 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:25.310+0000 7f36a5ffb700 10 monclient: dump: 2023-09-30T01:35:25.334 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:35:25.335 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:35:25.335 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:35:25.335 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:35:25.335 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:35:25.335 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:35:25.335 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:35:25.335 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:35:25.335 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:35:25.335 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:35:25.335 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:35:25.335 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:35:25.335 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:35:25.336 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:35:25.336 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:35:25.336 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:35:25.336 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:25.310+0000 7f36b6b06700 10 monclient: _renew_subs 2023-09-30T01:35:25.336 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:25.310+0000 7f36b6b06700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.88:3301/0 2023-09-30T01:35:25.336 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:25.310+0000 7f36b6b06700 1 -- 172.21.15.88:0/1043378658 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f36b011e0d0 con 0x7f36b01333b0 2023-09-30T01:35:25.336 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:25.314+0000 7f36a5ffb700 1 -- 172.21.15.88:0/1043378658 <== mon.2 v2:172.21.15.88:3301/0 4 ==== mgrmap(e 5) v1 ==== 40144+0+0 (secure 0 0 0) 0x7f36a80155a0 con 0x7f36b01333b0 2023-09-30T01:35:25.336 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:25.314+0000 7f36b6b06700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:35:25.336 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:25.314+0000 7f36b6b06700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:35:25.336 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:25.314+0000 7f36b6b06700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.88:3301/0 2023-09-30T01:35:25.336 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:25.314+0000 7f36b6b06700 1 -- 172.21.15.88:0/1043378658 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f36b00a31f0 con 0x7f36b01333b0 2023-09-30T01:35:25.336 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:25.314+0000 7f36a5ffb700 1 --2- 172.21.15.88:0/1043378658 >> v2:172.21.15.107:6812/29926 conn(0x7f3694033390 0x7f3694035840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:35:25.336 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:25.314+0000 7f36a5ffb700 1 -- 172.21.15.88:0/1043378658 <== mon.2 v2:172.21.15.88:3301/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 3502+0+0 (secure 0 0 0) 0x7f36a80438d0 con 0x7f36b01333b0 2023-09-30T01:35:25.337 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:25.314+0000 7f36b5b04700 1 --2- 172.21.15.88:0/1043378658 >> v2:172.21.15.107:6812/29926 conn(0x7f3694033390 0x7f3694035840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:35:25.337 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:25.314+0000 7f36b5b04700 10 monclient: get_auth_request con 0x7f3694033390 auth_method 0 2023-09-30T01:35:25.337 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:25.314+0000 7f36b5b04700 1 --2- 172.21.15.88:0/1043378658 >> v2:172.21.15.107:6812/29926 conn(0x7f3694033390 0x7f3694035840 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f36a0009db0 tx=0x7f36a00091a0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:35:25.337 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:25.318+0000 7f36a5ffb700 1 -- 172.21.15.88:0/1043378658 <== mon.2 v2:172.21.15.88:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f36a801e400 con 0x7f36b01333b0 2023-09-30T01:35:25.337 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:25.318+0000 7f36a5ffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:35:25.337 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:25.318+0000 7f36a5ffb700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:35:25.521 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:25.518+0000 7f36b6b06700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T01:35:25.522 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:25.518+0000 7f36b6b06700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:35:25.522 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:25.518+0000 7f36b6b06700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.88:3301/0 2023-09-30T01:35:25.522 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:25.518+0000 7f36b6b06700 1 -- 172.21.15.88:0/1043378658 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f36b0120120 con 0x7f36b01333b0 2023-09-30T01:35:25.522 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:25.518+0000 7f36a5ffb700 1 -- 172.21.15.88:0/1043378658 <== mon.2 v2:172.21.15.88:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f36a8013300 con 0x7f36b01333b0 2023-09-30T01:35:25.522 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:25.518+0000 7f36a5ffb700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:35:25.523 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:25.518+0000 7f36a5ffb700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:35:25.523 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-09-30T01:35:25.523 INFO:teuthology.orchestra.run.smithi088.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:35:25.525 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:25.522+0000 7f36b6b06700 1 -- 172.21.15.88:0/1043378658 >> v2:172.21.15.107:6812/29926 conn(0x7f3694033390 msgr2=0x7f3694035840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:35:25.525 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:25.522+0000 7f36b6b06700 1 --2- 172.21.15.88:0/1043378658 >> v2:172.21.15.107:6812/29926 conn(0x7f3694033390 0x7f3694035840 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f36a0009db0 tx=0x7f36a00091a0 comp rx=0 tx=0).stop 2023-09-30T01:35:25.525 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:25.522+0000 7f36b6b06700 10 monclient: shutdown 2023-09-30T01:35:25.525 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:25.522+0000 7f36b6b06700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:35:25.525 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:25.522+0000 7f36b6b06700 1 -- 172.21.15.88:0/1043378658 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f36b01333b0 msgr2=0x7f36b011d4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:35:25.525 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:25.522+0000 7f36b6b06700 1 --2- 172.21.15.88:0/1043378658 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f36b01333b0 0x7f36b011d4a0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f36a8007f30 tx=0x7f36a8009910 comp rx=0 tx=0).stop 2023-09-30T01:35:25.525 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:25.522+0000 7f36b6b06700 1 -- 172.21.15.88:0/1043378658 shutdown_connections 2023-09-30T01:35:25.525 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:25.522+0000 7f36b6b06700 1 --2- 172.21.15.88:0/1043378658 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f36b01333b0 0x7f36b011d4a0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:35:25.525 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:25.522+0000 7f36b6b06700 1 --2- 172.21.15.88:0/1043378658 >> v2:172.21.15.107:6812/29926 conn(0x7f3694033390 0x7f3694035840 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:35:25.526 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:25.522+0000 7f36b6b06700 1 --2- 172.21.15.88:0/1043378658 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f36b00a3380 0x7f36b011cf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:35:25.526 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:25.522+0000 7f36b6b06700 1 --2- 172.21.15.88:0/1043378658 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7f36b009ffd0 0x7f36b009f510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:35:25.526 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:25.522+0000 7f36b6b06700 1 -- 172.21.15.88:0/1043378658 >> 172.21.15.88:0/1043378658 conn(0x7f36b006e5e0 msgr2=0x7f36b0131750 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:35:25.526 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:25.522+0000 7f36b6b06700 1 -- 172.21.15.88:0/1043378658 shutdown_connections 2023-09-30T01:35:25.526 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:25.522+0000 7f36b6b06700 1 -- 172.21.15.88:0/1043378658 wait complete. 2023-09-30T01:35:28.536 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:35:28.639 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:28.634+0000 7f0fd6547700 10 monclient: get_monmap_and_config 2023-09-30T01:35:28.639 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:28.634+0000 7f0fd6547700 10 monclient: build_initial_monmap 2023-09-30T01:35:28.639 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:28.634+0000 7f0fd6547700 10 monclient: monmap: 2023-09-30T01:35:28.639 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2023-09-30T01:35:28.639 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:35:28.640 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:35:28.639266+0000 2023-09-30T01:35:28.640 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:35:28.639266+0000 2023-09-30T01:35:28.640 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2023-09-30T01:35:28.640 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:35:28.640 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a-legacy 2023-09-30T01:35:28.640 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.107:6789/0 mon.noname-f-legacy 2023-09-30T01:35:28.640 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.noname-b 2023-09-30T01:35:28.641 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.noname-g 2023-09-30T01:35:28.641 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.noname-c 2023-09-30T01:35:28.641 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.noname-h 2023-09-30T01:35:28.641 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.noname-d 2023-09-30T01:35:28.641 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.noname-i 2023-09-30T01:35:28.641 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.noname-e 2023-09-30T01:35:28.641 INFO:teuthology.orchestra.run.smithi088.stderr:9: v2:172.21.15.88:3300/0 mon.noname-a 2023-09-30T01:35:28.641 INFO:teuthology.orchestra.run.smithi088.stderr:10: v2:172.21.15.107:3300/0 mon.noname-f 2023-09-30T01:35:28.641 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:35:28.651 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:28.638+0000 7f0fd6547700 1 Processor -- start 2023-09-30T01:35:28.651 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:28.638+0000 7f0fd6547700 1 -- start start 2023-09-30T01:35:28.651 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:28.638+0000 7f0fd6547700 10 monclient: init 2023-09-30T01:35:28.651 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:28.638+0000 7f0fd6547700 10 monclient: _reopen_session rank -1 2023-09-30T01:35:28.651 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:28.638+0000 7f0fd6547700 10 monclient: _add_conns ranks=[0,5,6,1,9,2,10,8,7,4,3] 2023-09-30T01:35:28.651 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:28.638+0000 7f0fd6547700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f0fd009be50 addr v1:172.21.15.88:6789/0 2023-09-30T01:35:28.651 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:28.638+0000 7f0fd6547700 1 --2- >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7f0fd009c220 0x7f0fd009c610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:35:28.652 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:28.638+0000 7f0fd6547700 10 monclient(hunting): picked mon.noname-h con 0x7f0fd009c220 addr [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] 2023-09-30T01:35:28.652 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:28.638+0000 7f0fd6547700 1 --2- >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7f0fd0138860 0x7f0fd013ac40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:35:28.652 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:28.638+0000 7f0fd6547700 10 monclient(hunting): picked mon.noname-d con 0x7f0fd0138860 addr [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] 2023-09-30T01:35:28.652 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:28.638+0000 7f0fd6547700 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0fd004bb20 con 0x7f0fd009be50 2023-09-30T01:35:28.652 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:28.638+0000 7f0fd6547700 10 monclient(hunting): start opening mon connection 2023-09-30T01:35:28.652 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:28.638+0000 7f0fd6547700 1 -- --> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] -- mon_getmap magic: 0 v1 -- 0x7f0fd009d7c0 con 0x7f0fd009c220 2023-09-30T01:35:28.652 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:28.638+0000 7f0fd6547700 10 monclient(hunting): start opening mon connection 2023-09-30T01:35:28.652 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:28.638+0000 7f0fd6547700 1 -- --> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] -- mon_getmap magic: 0 v1 -- 0x7f0fd006b050 con 0x7f0fd0138860 2023-09-30T01:35:28.653 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:28.638+0000 7f0fd6547700 10 monclient(hunting): _renew_subs 2023-09-30T01:35:28.653 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:28.638+0000 7f0fd6547700 10 monclient(hunting): authenticate will time out at 1000.678506s 2023-09-30T01:35:28.653 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:28.638+0000 7f0fd5545700 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f0fd009be50 0x7f0fd006a940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:32898/0 (socket says 172.21.15.88:32898) 2023-09-30T01:35:28.653 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:28.638+0000 7f0fd5545700 1 -- 172.21.15.88:0/389748402 learned_addr learned my addr 172.21.15.88:0/389748402 (peer_addr_for_me v1:172.21.15.88:0/0) 2023-09-30T01:35:28.653 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:28.638+0000 7f0fd5d46700 1 --2- 172.21.15.88:0/389748402 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7f0fd0138860 0x7f0fd013ac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:35:28.653 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:28.638+0000 7f0fd5d46700 10 monclient(hunting): get_auth_request con 0x7f0fd0138860 auth_method 0 2023-09-30T01:35:28.653 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:28.638+0000 7f0fd5d46700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:35:28.653 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:28.638+0000 7f0fd5d46700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:35:28.654 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:28.638+0000 7f0fd5d46700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:35:28.654 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:28.638+0000 7f0fd4d44700 1 --2- 172.21.15.88:0/389748402 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7f0fd009c220 0x7f0fd009c610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:35:28.654 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:28.638+0000 7f0fd4d44700 10 monclient(hunting): get_auth_request con 0x7f0fd009c220 auth_method 0 2023-09-30T01:35:28.654 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:28.638+0000 7f0fd4d44700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:35:28.654 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:28.638+0000 7f0fd4d44700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:35:28.654 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:28.638+0000 7f0fd4d44700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:35:28.654 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:28.638+0000 7f0fd5d46700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:35:28.654 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:28.638+0000 7f0fd5d46700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:35:28.655 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:28.638+0000 7f0fd5d46700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:35:28.655 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:28.638+0000 7f0fd4d44700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:35:28.655 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:28.638+0000 7f0fd4d44700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:35:28.655 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:28.638+0000 7f0fd4d44700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:35:28.655 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:28.638+0000 7f0fd5d46700 10 monclient(hunting): handle_auth_done global_id 4497 payload 274 2023-09-30T01:35:28.655 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:28.638+0000 7f0fc7fff700 1 -- 172.21.15.88:0/389748402 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 852202949 0 0) 0x7f0fd004bb20 con 0x7f0fd009be50 2023-09-30T01:35:28.655 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:28.638+0000 7f0fd5d46700 1 -- 172.21.15.88:0/389748402 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7f0fd009c220 msgr2=0x7f0fd009c610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:35:28.655 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:28.638+0000 7f0fd5d46700 1 --2- 172.21.15.88:0/389748402 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7f0fd009c220 0x7f0fd009c610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:35:28.656 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:28.638+0000 7f0fd5d46700 1 -- 172.21.15.88:0/389748402 >> v1:172.21.15.88:6789/0 conn(0x7f0fd009be50 legacy=0x7f0fd006a940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:35:28.656 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:28.638+0000 7f0fd5d46700 10 monclient: _finish_hunting 0 2023-09-30T01:35:28.656 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:28.638+0000 7f0fd5d46700 1 monclient: found mon.noname-d 2023-09-30T01:35:28.656 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:28.638+0000 7f0fd5d46700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:35:28.656 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:28.638+0000 7f0fd5d46700 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.88:3303/0 2023-09-30T01:35:28.656 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:28.638+0000 7f0fd5d46700 1 -- 172.21.15.88:0/389748402 --> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0fd009ede0 con 0x7f0fd0138860 2023-09-30T01:35:28.656 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:28.638+0000 7f0fc7fff700 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2023-09-30T01:35:28.657 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:28.638+0000 7f0fd5d46700 1 --2- 172.21.15.88:0/389748402 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7f0fd0138860 0x7f0fd013ac40 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f0fc8009e90 tx=0x7f0fc8009780 comp rx=0 tx=0).ready entity=mon.6 client_cookie=4e27f9910c44adae server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:35:28.657 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:28.638+0000 7f0fc7fff700 1 -- 172.21.15.88:0/389748402 <== mon.6 v2:172.21.15.88:3303/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f0fc800e070 con 0x7f0fd0138860 2023-09-30T01:35:28.657 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:28.638+0000 7f0fc7fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:35:28.657 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:28.638+0000 7f0fc7fff700 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2023-09-30T01:35:28.657 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:28.638+0000 7f0fc7fff700 10 monclient: dump: 2023-09-30T01:35:28.657 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:35:28.657 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:35:28.657 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:35:28.657 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:35:28.658 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:35:28.658 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:35:28.658 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:35:28.658 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:35:28.658 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:35:28.658 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:35:28.658 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:35:28.658 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:35:28.659 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:35:28.659 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:35:28.659 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:35:28.659 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:35:28.659 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:28.642+0000 7f0fc7fff700 10 monclient: _finish_auth 0 2023-09-30T01:35:28.659 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:28.642+0000 7f0fc7fff700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:35:28.659 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:28.642+0000 7f0fd6547700 5 monclient: authenticate success, global_id 4497 2023-09-30T01:35:28.659 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:28.642+0000 7f0fc7fff700 1 -- 172.21.15.88:0/389748402 <== mon.6 v2:172.21.15.88:3303/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0fc800bc30 con 0x7f0fd0138860 2023-09-30T01:35:28.660 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:28.642+0000 7f0fd6547700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:35:28.660 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:28.642+0000 7f0fc7fff700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:35:28.660 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:28.642+0000 7f0fd6547700 10 monclient: get_monmap_and_config success 2023-09-30T01:35:28.660 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:28.642+0000 7f0fd6547700 4 set_mon_vals no callback set 2023-09-30T01:35:28.660 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:28.642+0000 7f0fc7fff700 1 -- 172.21.15.88:0/389748402 <== mon.6 v2:172.21.15.88:3303/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f0fc800bda0 con 0x7f0fd0138860 2023-09-30T01:35:28.660 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:28.642+0000 7f0fc7fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:35:28.660 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:28.642+0000 7f0fc7fff700 10 monclient: got monmap 1 from mon.d (according to old e1) 2023-09-30T01:35:28.661 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:28.642+0000 7f0fc7fff700 10 monclient: dump: 2023-09-30T01:35:28.661 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:35:28.661 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:35:28.661 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:35:28.661 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:35:28.661 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:35:28.661 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:35:28.661 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:35:28.661 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:35:28.662 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:35:28.662 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:35:28.662 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:35:28.662 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:35:28.662 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:35:28.662 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:35:28.662 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:35:28.662 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:35:28.663 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:28.642+0000 7f0fd6547700 10 monclient: shutdown 2023-09-30T01:35:28.663 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:28.642+0000 7f0fd6547700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:35:28.663 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:28.642+0000 7f0fd6547700 1 -- 172.21.15.88:0/389748402 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7f0fd0138860 msgr2=0x7f0fd013ac40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:35:28.663 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:28.642+0000 7f0fd6547700 1 --2- 172.21.15.88:0/389748402 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7f0fd0138860 0x7f0fd013ac40 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f0fc8009e90 tx=0x7f0fc8009780 comp rx=0 tx=0).stop 2023-09-30T01:35:28.663 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:28.642+0000 7f0fd6547700 1 -- 172.21.15.88:0/389748402 shutdown_connections 2023-09-30T01:35:28.663 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:28.642+0000 7f0fd6547700 1 --2- 172.21.15.88:0/389748402 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7f0fd0138860 0x7f0fd013ac40 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:35:28.663 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:28.642+0000 7f0fd6547700 1 --2- 172.21.15.88:0/389748402 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7f0fd009c220 0x7f0fd009c610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:35:28.663 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:28.642+0000 7f0fd6547700 1 -- 172.21.15.88:0/389748402 >> 172.21.15.88:0/389748402 conn(0x7f0fd0063610 msgr2=0x7f0fd00a2b90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:35:28.664 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:28.642+0000 7f0fd6547700 1 -- 172.21.15.88:0/389748402 shutdown_connections 2023-09-30T01:35:28.664 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:28.642+0000 7f0fd6547700 1 -- 172.21.15.88:0/389748402 wait complete. 2023-09-30T01:35:28.664 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:28.642+0000 7f0fd6547700 10 monclient: build_initial_monmap 2023-09-30T01:35:28.664 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:28.642+0000 7f0fd6547700 10 monclient: monmap: 2023-09-30T01:35:28.664 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2023-09-30T01:35:28.664 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:35:28.664 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 0.000000 2023-09-30T01:35:28.664 INFO:teuthology.orchestra.run.smithi088.stderr:created 0.000000 2023-09-30T01:35:28.664 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2023-09-30T01:35:28.665 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 1 2023-09-30T01:35:28.665 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.noname-a 2023-09-30T01:35:28.665 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.noname-f 2023-09-30T01:35:28.665 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.noname-b 2023-09-30T01:35:28.665 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.noname-g 2023-09-30T01:35:28.665 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.noname-c 2023-09-30T01:35:28.665 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.noname-h 2023-09-30T01:35:28.665 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.noname-d 2023-09-30T01:35:28.665 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.noname-i 2023-09-30T01:35:28.666 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.noname-e 2023-09-30T01:35:28.666 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:35:28.666 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:28.642+0000 7f0fd6547700 1 Processor -- start 2023-09-30T01:35:28.666 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:28.642+0000 7f0fd6547700 1 -- start start 2023-09-30T01:35:28.666 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:28.642+0000 7f0fd6547700 10 monclient: init 2023-09-30T01:35:28.666 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:28.642+0000 7f0fd6547700 10 monclient: _reopen_session rank -1 2023-09-30T01:35:28.666 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:28.642+0000 7f0fd6547700 10 monclient: _add_conns ranks=[8,5,0,4,1,2,3,6,7] 2023-09-30T01:35:28.666 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:28.642+0000 7f0fd6547700 1 --2- >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f0fd009be50 0x7f0fd013ed00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:35:28.666 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:28.642+0000 7f0fd6547700 10 monclient(hunting): picked mon.noname-e con 0x7f0fd009be50 addr [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] 2023-09-30T01:35:28.667 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:28.642+0000 7f0fd6547700 1 --2- >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7f0fd009c220 0x7f0fd0137f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:35:28.667 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:28.642+0000 7f0fd6547700 10 monclient(hunting): picked mon.noname-h con 0x7f0fd009c220 addr [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] 2023-09-30T01:35:28.667 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:28.642+0000 7f0fd6547700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0fd0138860 0x7f0fd0142210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:35:28.667 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:28.642+0000 7f0fd6547700 10 monclient(hunting): picked mon.noname-a con 0x7f0fd0138860 addr [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] 2023-09-30T01:35:28.667 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:28.642+0000 7f0fd6547700 10 monclient(hunting): start opening mon connection 2023-09-30T01:35:28.667 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:28.642+0000 7f0fd6547700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fd01235e0 con 0x7f0fd0138860 2023-09-30T01:35:28.667 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:28.642+0000 7f0fd6547700 10 monclient(hunting): start opening mon connection 2023-09-30T01:35:28.667 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:28.642+0000 7f0fd6547700 1 -- --> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] -- mon_getmap magic: 0 v1 -- 0x7f0fd01215f0 con 0x7f0fd009c220 2023-09-30T01:35:28.668 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:28.642+0000 7f0fd6547700 10 monclient(hunting): start opening mon connection 2023-09-30T01:35:28.668 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:28.642+0000 7f0fd6547700 1 -- --> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] -- mon_getmap magic: 0 v1 -- 0x7f0fd0123020 con 0x7f0fd009be50 2023-09-30T01:35:28.668 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:28.642+0000 7f0fd6547700 10 monclient(hunting): _renew_subs 2023-09-30T01:35:28.668 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:28.642+0000 7f0fd6547700 10 monclient(hunting): authenticate will time out at 1000.683317s 2023-09-30T01:35:28.668 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:28.642+0000 7f0fd5545700 1 --2- >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f0fd009be50 0x7f0fd013ed00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:35:28.668 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:28.642+0000 7f0fd5d46700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0fd0138860 0x7f0fd0142210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:35:28.668 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:28.642+0000 7f0fd5545700 1 --2- >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f0fd009be50 0x7f0fd013ed00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3304/0 says I am v2:172.21.15.88:39656/0 (socket says 172.21.15.88:39656) 2023-09-30T01:35:28.668 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:28.642+0000 7f0fd4d44700 1 --2- >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7f0fd009c220 0x7f0fd0137f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:35:28.669 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:28.642+0000 7f0fd5d46700 10 monclient(hunting): get_auth_request con 0x7f0fd0138860 auth_method 0 2023-09-30T01:35:28.669 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:28.642+0000 7f0fd5d46700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:35:28.669 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:28.642+0000 7f0fd5d46700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:35:28.669 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:28.642+0000 7f0fd5545700 1 -- 172.21.15.88:0/670350157 learned_addr learned my addr 172.21.15.88:0/670350157 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-09-30T01:35:28.669 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:28.642+0000 7f0fd5d46700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:35:28.669 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:28.642+0000 7f0fd5545700 10 monclient(hunting): get_auth_request con 0x7f0fd009be50 auth_method 0 2023-09-30T01:35:28.669 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:28.642+0000 7f0fd5545700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:35:28.669 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:28.642+0000 7f0fd5545700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:35:28.669 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:28.642+0000 7f0fd5545700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:35:28.670 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:28.642+0000 7f0fd4d44700 10 monclient(hunting): get_auth_request con 0x7f0fd009c220 auth_method 0 2023-09-30T01:35:28.670 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:28.642+0000 7f0fd4d44700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:35:28.670 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:28.642+0000 7f0fd4d44700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:35:28.670 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:28.642+0000 7f0fd4d44700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:35:28.670 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:28.642+0000 7f0fd5d46700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:35:28.670 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:28.642+0000 7f0fd5d46700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:35:28.670 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:28.642+0000 7f0fd5d46700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:35:28.671 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:28.642+0000 7f0fd4d44700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:35:28.671 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:28.642+0000 7f0fd4d44700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:35:28.671 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:28.642+0000 7f0fd4d44700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:35:28.671 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:28.642+0000 7f0fd5545700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:35:28.671 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:28.642+0000 7f0fd5545700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:35:28.671 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:28.642+0000 7f0fd5545700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:35:28.671 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:28.646+0000 7f0fd5d46700 10 monclient(hunting): handle_auth_done global_id 4815 payload 824 2023-09-30T01:35:28.671 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:28.646+0000 7f0fd5d46700 1 -- 172.21.15.88:0/670350157 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f0fd009be50 msgr2=0x7f0fd013ed00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:35:28.671 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:28.646+0000 7f0fd5d46700 1 --2- 172.21.15.88:0/670350157 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f0fd009be50 0x7f0fd013ed00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:35:28.672 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:28.646+0000 7f0fd5d46700 1 -- 172.21.15.88:0/670350157 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7f0fd009c220 msgr2=0x7f0fd0137f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:35:28.672 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:28.646+0000 7f0fd5d46700 1 --2- 172.21.15.88:0/670350157 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7f0fd009c220 0x7f0fd0137f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:35:28.672 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:28.646+0000 7f0fd5d46700 10 monclient: _finish_hunting 0 2023-09-30T01:35:28.672 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:28.646+0000 7f0fd5d46700 1 monclient: found mon.noname-a 2023-09-30T01:35:28.672 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:28.646+0000 7f0fd5d46700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:35:28.672 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:28.646+0000 7f0fd5d46700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.88:3300/0 2023-09-30T01:35:28.672 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:28.646+0000 7f0fd5d46700 1 -- 172.21.15.88:0/670350157 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0fc8009a20 con 0x7f0fd0138860 2023-09-30T01:35:28.672 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:28.646+0000 7f0fd4d44700 1 --2- 172.21.15.88:0/670350157 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7f0fd009c220 0x7f0fd0137f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:35:28.673 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:28.646+0000 7f0fd5d46700 1 --2- 172.21.15.88:0/670350157 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0fd0138860 0x7f0fd0142210 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f0fd00675d0 tx=0x7f0fc8019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:35:28.673 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:28.646+0000 7f0fc57fa700 1 -- 172.21.15.88:0/670350157 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f0fc801d410 con 0x7f0fd0138860 2023-09-30T01:35:28.673 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:28.646+0000 7f0fc57fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:35:28.673 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:28.646+0000 7f0fc57fa700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T01:35:28.673 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:28.646+0000 7f0fc57fa700 10 monclient: dump: 2023-09-30T01:35:28.673 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:35:28.673 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:35:28.673 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:35:28.674 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:35:28.674 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:35:28.674 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:35:28.674 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:35:28.674 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:35:28.674 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:35:28.674 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:35:28.675 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:35:28.675 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:35:28.675 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:35:28.675 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:35:28.675 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:35:28.675 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:35:28.675 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:28.646+0000 7f0fc57fa700 10 monclient: _finish_auth 0 2023-09-30T01:35:28.675 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:28.646+0000 7f0fc57fa700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:35:28.676 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:28.646+0000 7f0fd6547700 5 monclient: authenticate success, global_id 4815 2023-09-30T01:35:28.676 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:28.646+0000 7f0fc57fa700 1 -- 172.21.15.88:0/670350157 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0fc8021070 con 0x7f0fd0138860 2023-09-30T01:35:28.676 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:28.646+0000 7f0fd6547700 10 monclient: _renew_subs 2023-09-30T01:35:28.676 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:28.646+0000 7f0fd6547700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.88:3300/0 2023-09-30T01:35:28.676 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:28.646+0000 7f0fd6547700 1 -- 172.21.15.88:0/670350157 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0fd0123300 con 0x7f0fd0138860 2023-09-30T01:35:28.676 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:28.646+0000 7f0fc57fa700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:35:28.677 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:28.646+0000 7f0fc57fa700 1 -- 172.21.15.88:0/670350157 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f0fc801da30 con 0x7f0fd0138860 2023-09-30T01:35:28.677 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:28.646+0000 7f0fc57fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:35:28.677 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:28.646+0000 7f0fc57fa700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:35:28.677 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:28.646+0000 7f0fc57fa700 10 monclient: dump: 2023-09-30T01:35:28.677 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:35:28.677 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:35:28.678 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:35:28.678 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:35:28.678 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:35:28.678 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:35:28.678 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:35:28.678 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:35:28.679 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:35:28.679 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:35:28.679 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:35:28.679 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:35:28.679 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:35:28.679 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:35:28.679 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:35:28.680 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:35:28.680 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:28.646+0000 7f0fc77fe700 4 set_mon_vals no callback set 2023-09-30T01:35:28.680 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:28.646+0000 7f0fd6547700 10 monclient: _renew_subs 2023-09-30T01:35:28.680 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:28.646+0000 7f0fd6547700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.88:3300/0 2023-09-30T01:35:28.680 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:28.646+0000 7f0fd6547700 1 -- 172.21.15.88:0/670350157 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0fd013f710 con 0x7f0fd0138860 2023-09-30T01:35:28.680 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:28.646+0000 7f0fd6547700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:35:28.681 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:28.646+0000 7f0fd6547700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:35:28.681 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:28.646+0000 7f0fd6547700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.88:3300/0 2023-09-30T01:35:28.681 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:28.646+0000 7f0fd6547700 1 -- 172.21.15.88:0/670350157 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0fd009e950 con 0x7f0fd0138860 2023-09-30T01:35:28.681 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:28.646+0000 7f0fc57fa700 1 -- 172.21.15.88:0/670350157 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 40144+0+0 (secure 0 0 0) 0x7f0fc8023030 con 0x7f0fd0138860 2023-09-30T01:35:28.681 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:28.646+0000 7f0fc57fa700 1 --2- 172.21.15.88:0/670350157 >> v2:172.21.15.107:6812/29926 conn(0x7f0fb4033360 0x7f0fb4035810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:35:28.681 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:28.646+0000 7f0fc57fa700 1 -- 172.21.15.88:0/670350157 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 3502+0+0 (secure 0 0 0) 0x7f0fc8045190 con 0x7f0fd0138860 2023-09-30T01:35:28.681 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:28.646+0000 7f0fd5545700 1 --2- 172.21.15.88:0/670350157 >> v2:172.21.15.107:6812/29926 conn(0x7f0fb4033360 0x7f0fb4035810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:35:28.681 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:28.646+0000 7f0fd5545700 10 monclient: get_auth_request con 0x7f0fb4033360 auth_method 0 2023-09-30T01:35:28.682 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:28.650+0000 7f0fd5545700 1 --2- 172.21.15.88:0/670350157 >> v2:172.21.15.107:6812/29926 conn(0x7f0fb4033360 0x7f0fb4035810 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f0fcc007160 tx=0x7f0fcc000fe0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:35:28.682 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:28.654+0000 7f0fc57fa700 1 -- 172.21.15.88:0/670350157 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f0fd009e950 con 0x7f0fd0138860 2023-09-30T01:35:28.682 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:28.654+0000 7f0fc57fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:35:28.682 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:28.654+0000 7f0fc57fa700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:35:28.851 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:28.846+0000 7f0fd6547700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T01:35:28.851 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:28.846+0000 7f0fd6547700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:35:28.851 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:28.846+0000 7f0fd6547700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.88:3300/0 2023-09-30T01:35:28.851 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:28.846+0000 7f0fd6547700 1 -- 172.21.15.88:0/670350157 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f0fd009e950 con 0x7f0fd0138860 2023-09-30T01:35:28.852 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:28.850+0000 7f0fc57fa700 1 -- 172.21.15.88:0/670350157 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f0fd009e950 con 0x7f0fd0138860 2023-09-30T01:35:28.852 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:28.850+0000 7f0fc57fa700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:35:28.852 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:28.850+0000 7f0fc57fa700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:35:28.852 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-09-30T01:35:28.852 INFO:teuthology.orchestra.run.smithi088.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:35:28.854 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:28.850+0000 7f0fd6547700 1 -- 172.21.15.88:0/670350157 >> v2:172.21.15.107:6812/29926 conn(0x7f0fb4033360 msgr2=0x7f0fb4035810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:35:28.854 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:28.850+0000 7f0fd6547700 1 --2- 172.21.15.88:0/670350157 >> v2:172.21.15.107:6812/29926 conn(0x7f0fb4033360 0x7f0fb4035810 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f0fcc007160 tx=0x7f0fcc000fe0 comp rx=0 tx=0).stop 2023-09-30T01:35:28.855 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:28.850+0000 7f0fd6547700 10 monclient: shutdown 2023-09-30T01:35:28.855 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:28.850+0000 7f0fd6547700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:35:28.855 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:28.850+0000 7f0fd6547700 1 -- 172.21.15.88:0/670350157 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0fd0138860 msgr2=0x7f0fd0142210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:35:28.855 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:28.850+0000 7f0fd6547700 1 --2- 172.21.15.88:0/670350157 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0fd0138860 0x7f0fd0142210 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f0fd00675d0 tx=0x7f0fc8019000 comp rx=0 tx=0).stop 2023-09-30T01:35:28.855 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:28.850+0000 7f0fd6547700 1 -- 172.21.15.88:0/670350157 shutdown_connections 2023-09-30T01:35:28.855 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:28.850+0000 7f0fd6547700 1 --2- 172.21.15.88:0/670350157 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0fd0138860 0x7f0fd0142210 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:35:28.855 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:28.850+0000 7f0fd6547700 1 --2- 172.21.15.88:0/670350157 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7f0fd009c220 0x7f0fd0137f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:35:28.855 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:28.850+0000 7f0fd6547700 1 --2- 172.21.15.88:0/670350157 >> v2:172.21.15.107:6812/29926 conn(0x7f0fb4033360 0x7f0fb4035810 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:35:28.855 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:28.850+0000 7f0fd6547700 1 --2- 172.21.15.88:0/670350157 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f0fd009be50 0x7f0fd013ed00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:35:28.855 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:28.850+0000 7f0fd6547700 1 -- 172.21.15.88:0/670350157 >> 172.21.15.88:0/670350157 conn(0x7f0fd0063610 msgr2=0x7f0fd006b720 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:35:28.856 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:28.850+0000 7f0fd6547700 1 -- 172.21.15.88:0/670350157 shutdown_connections 2023-09-30T01:35:28.856 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:28.850+0000 7f0fd6547700 1 -- 172.21.15.88:0/670350157 wait complete. 2023-09-30T01:35:31.867 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:35:31.980 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:31.978+0000 7f338f544700 10 monclient: get_monmap_and_config 2023-09-30T01:35:31.980 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:31.978+0000 7f338f544700 10 monclient: build_initial_monmap 2023-09-30T01:35:31.980 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:31.978+0000 7f338f544700 10 monclient: monmap: 2023-09-30T01:35:31.980 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2023-09-30T01:35:31.980 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:35:31.980 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:35:31.980254+0000 2023-09-30T01:35:31.981 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:35:31.980254+0000 2023-09-30T01:35:31.981 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2023-09-30T01:35:31.981 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:35:31.981 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a-legacy 2023-09-30T01:35:31.981 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.107:6789/0 mon.noname-f-legacy 2023-09-30T01:35:31.981 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.noname-b 2023-09-30T01:35:31.981 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.noname-g 2023-09-30T01:35:31.981 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.noname-c 2023-09-30T01:35:31.981 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.noname-h 2023-09-30T01:35:31.981 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.noname-d 2023-09-30T01:35:31.981 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.noname-i 2023-09-30T01:35:31.982 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.noname-e 2023-09-30T01:35:31.982 INFO:teuthology.orchestra.run.smithi088.stderr:9: v2:172.21.15.88:3300/0 mon.noname-a 2023-09-30T01:35:31.982 INFO:teuthology.orchestra.run.smithi088.stderr:10: v2:172.21.15.107:3300/0 mon.noname-f 2023-09-30T01:35:31.982 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:35:31.986 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:31.978+0000 7f338f544700 1 Processor -- start 2023-09-30T01:35:31.986 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:31.978+0000 7f338f544700 1 -- start start 2023-09-30T01:35:31.987 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:31.978+0000 7f338f544700 10 monclient: init 2023-09-30T01:35:31.987 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:31.978+0000 7f338f544700 10 monclient: _reopen_session rank -1 2023-09-30T01:35:31.987 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:31.978+0000 7f338f544700 10 monclient: _add_conns ranks=[5,7,10,8,1,6,4,0,3,2,9] 2023-09-30T01:35:31.987 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:31.978+0000 7f338f544700 1 --2- >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7f33880a3380 0x7f33880a48b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:35:31.987 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:31.978+0000 7f338f544700 10 monclient(hunting): picked mon.noname-h con 0x7f33880a3380 addr [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] 2023-09-30T01:35:31.987 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:31.978+0000 7f338f544700 1 --2- >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7f338809ffd0 0x7f33880a03a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:35:31.987 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:31.978+0000 7f338f544700 10 monclient(hunting): picked mon.noname-i con 0x7f338809ffd0 addr [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] 2023-09-30T01:35:31.987 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:31.978+0000 7f338f544700 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f33881333b0 0x7f3388133800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:35:31.987 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:31.978+0000 7f338f544700 10 monclient(hunting): picked mon.noname-f con 0x7f33881333b0 addr v2:172.21.15.107:3300/0 2023-09-30T01:35:31.987 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:31.978+0000 7f338f544700 10 monclient(hunting): start opening mon connection 2023-09-30T01:35:31.987 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:31.978+0000 7f338f544700 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3388097ab0 con 0x7f33881333b0 2023-09-30T01:35:31.988 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:31.978+0000 7f338f544700 10 monclient(hunting): start opening mon connection 2023-09-30T01:35:31.988 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:31.978+0000 7f338f544700 1 -- --> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] -- mon_getmap magic: 0 v1 -- 0x7f33880a08e0 con 0x7f33880a3380 2023-09-30T01:35:31.988 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:31.978+0000 7f338f544700 10 monclient(hunting): start opening mon connection 2023-09-30T01:35:31.988 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:31.978+0000 7f338f544700 1 -- --> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] -- mon_getmap magic: 0 v1 -- 0x7f3388098480 con 0x7f338809ffd0 2023-09-30T01:35:31.988 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:31.978+0000 7f338f544700 10 monclient(hunting): _renew_subs 2023-09-30T01:35:31.988 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:31.978+0000 7f338f544700 10 monclient(hunting): authenticate will time out at 1004.018632s 2023-09-30T01:35:31.988 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:31.978+0000 7f338e542700 1 --2- >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7f33880a3380 0x7f33880a48b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:35:31.988 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:31.978+0000 7f338dd41700 1 --2- >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7f338809ffd0 0x7f33880a03a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:35:31.988 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:31.978+0000 7f338ed43700 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f33881333b0 0x7f3388133800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:35:31.988 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:31.978+0000 7f338e542700 1 --2- >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7f33880a3380 0x7f33880a48b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3302/0 says I am v2:172.21.15.88:37574/0 (socket says 172.21.15.88:37574) 2023-09-30T01:35:31.988 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:31.978+0000 7f338e542700 1 -- 172.21.15.88:0/873279613 learned_addr learned my addr 172.21.15.88:0/873279613 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-09-30T01:35:31.989 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:31.978+0000 7f338e542700 10 monclient(hunting): get_auth_request con 0x7f33880a3380 auth_method 0 2023-09-30T01:35:31.989 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:31.978+0000 7f338e542700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:35:31.989 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:31.978+0000 7f338e542700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:35:31.989 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:31.978+0000 7f338e542700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:35:31.989 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:31.978+0000 7f338ed43700 10 monclient(hunting): get_auth_request con 0x7f33881333b0 auth_method 0 2023-09-30T01:35:31.989 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:31.978+0000 7f338ed43700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:35:31.989 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:31.978+0000 7f338ed43700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:35:31.989 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:31.978+0000 7f338ed43700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:35:31.989 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:31.978+0000 7f338dd41700 10 monclient(hunting): get_auth_request con 0x7f338809ffd0 auth_method 0 2023-09-30T01:35:31.989 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:31.978+0000 7f338dd41700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:35:31.989 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:31.978+0000 7f338dd41700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:35:31.990 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:31.978+0000 7f338dd41700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:35:31.990 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:31.978+0000 7f338e542700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:35:31.990 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:31.978+0000 7f338e542700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:35:31.990 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:31.978+0000 7f338e542700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:35:31.990 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:31.978+0000 7f338ed43700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:35:31.990 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:31.978+0000 7f338ed43700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:35:31.990 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:31.978+0000 7f338ed43700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:35:31.990 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:31.978+0000 7f338dd41700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:35:31.990 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:31.978+0000 7f338dd41700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:35:31.990 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:31.978+0000 7f338dd41700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:35:31.990 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:31.978+0000 7f338e542700 10 monclient(hunting): handle_auth_done global_id 4523 payload 274 2023-09-30T01:35:31.990 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:31.978+0000 7f338e542700 1 -- 172.21.15.88:0/873279613 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7f338809ffd0 msgr2=0x7f33880a03a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:35:31.991 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:31.978+0000 7f338e542700 1 --2- 172.21.15.88:0/873279613 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7f338809ffd0 0x7f33880a03a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:35:31.991 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:31.978+0000 7f338e542700 1 -- 172.21.15.88:0/873279613 >> v2:172.21.15.107:3300/0 conn(0x7f33881333b0 msgr2=0x7f3388133800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:35:31.991 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:31.978+0000 7f338e542700 1 --2- 172.21.15.88:0/873279613 >> v2:172.21.15.107:3300/0 conn(0x7f33881333b0 0x7f3388133800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:35:31.991 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:31.978+0000 7f338e542700 10 monclient: _finish_hunting 0 2023-09-30T01:35:31.991 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:31.978+0000 7f338e542700 1 monclient: found mon.noname-h 2023-09-30T01:35:31.991 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:31.978+0000 7f338e542700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:35:31.991 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:31.978+0000 7f338e542700 10 monclient: _send_mon_message to mon.noname-h at v2:172.21.15.107:3302/0 2023-09-30T01:35:31.991 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:31.978+0000 7f338e542700 1 -- 172.21.15.88:0/873279613 --> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f338804bb20 con 0x7f33880a3380 2023-09-30T01:35:31.991 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:31.982+0000 7f338e542700 1 --2- 172.21.15.88:0/873279613 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7f33880a3380 0x7f33880a48b0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f33840099b0 tx=0x7f3384009250 comp rx=0 tx=0).ready entity=mon.5 client_cookie=7c3a1e23a63aea14 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:35:31.991 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:31.982+0000 7f338d540700 1 -- 172.21.15.88:0/873279613 <== mon.5 v2:172.21.15.107:3302/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f338400bb00 con 0x7f33880a3380 2023-09-30T01:35:31.991 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:31.982+0000 7f338d540700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:35:31.992 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:31.982+0000 7f338d540700 10 monclient: got monmap 1 from mon.noname-h (according to old e1) 2023-09-30T01:35:31.992 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:31.982+0000 7f338d540700 10 monclient: dump: 2023-09-30T01:35:31.992 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:35:31.992 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:35:31.992 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:35:31.992 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:35:31.992 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:35:31.992 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:35:31.992 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:35:31.992 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:35:31.992 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:35:31.993 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:35:31.993 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:35:31.993 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:35:31.993 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:35:31.993 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:35:31.993 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:35:31.993 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:35:31.993 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:31.982+0000 7f338d540700 10 monclient: _finish_auth 0 2023-09-30T01:35:31.993 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:31.982+0000 7f338d540700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:35:31.993 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:31.982+0000 7f338d540700 1 -- 172.21.15.88:0/873279613 <== mon.5 v2:172.21.15.107:3302/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f338400bc60 con 0x7f33880a3380 2023-09-30T01:35:31.993 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:31.982+0000 7f338f544700 5 monclient: authenticate success, global_id 4523 2023-09-30T01:35:31.993 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:31.982+0000 7f338f544700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:35:31.994 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:31.982+0000 7f338d540700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:35:31.994 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:31.982+0000 7f338f544700 10 monclient: get_monmap_and_config success 2023-09-30T01:35:31.994 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:31.982+0000 7f338f544700 4 set_mon_vals no callback set 2023-09-30T01:35:31.994 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:31.982+0000 7f338d540700 1 -- 172.21.15.88:0/873279613 <== mon.5 v2:172.21.15.107:3302/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f338400bdd0 con 0x7f33880a3380 2023-09-30T01:35:31.994 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:31.982+0000 7f338d540700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:35:31.994 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:31.982+0000 7f338d540700 10 monclient: got monmap 1 from mon.h (according to old e1) 2023-09-30T01:35:31.994 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:31.982+0000 7f338d540700 10 monclient: dump: 2023-09-30T01:35:31.994 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:35:31.994 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:35:31.994 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:35:31.994 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:35:31.995 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:35:31.995 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:35:31.995 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:35:31.995 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:35:31.995 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:35:31.995 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:35:31.995 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:35:31.995 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:35:31.995 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:35:31.995 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:35:31.995 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:35:31.995 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:35:31.996 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:31.982+0000 7f338f544700 10 monclient: shutdown 2023-09-30T01:35:31.996 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:31.982+0000 7f338f544700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:35:31.996 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:31.982+0000 7f338f544700 1 -- 172.21.15.88:0/873279613 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7f33880a3380 msgr2=0x7f33880a48b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:35:31.996 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:31.982+0000 7f338f544700 1 --2- 172.21.15.88:0/873279613 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7f33880a3380 0x7f33880a48b0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f33840099b0 tx=0x7f3384009250 comp rx=0 tx=0).stop 2023-09-30T01:35:31.996 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:31.982+0000 7f338f544700 1 -- 172.21.15.88:0/873279613 shutdown_connections 2023-09-30T01:35:31.996 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:31.982+0000 7f338f544700 1 --2- 172.21.15.88:0/873279613 >> v2:172.21.15.107:3300/0 conn(0x7f33881333b0 0x7f3388133800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:35:31.996 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:31.982+0000 7f338f544700 1 --2- 172.21.15.88:0/873279613 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7f338809ffd0 0x7f33880a03a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:35:31.996 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:31.982+0000 7f338f544700 1 --2- 172.21.15.88:0/873279613 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7f33880a3380 0x7f33880a48b0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:35:31.996 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:31.982+0000 7f338f544700 1 -- 172.21.15.88:0/873279613 >> 172.21.15.88:0/873279613 conn(0x7f338806e5e0 msgr2=0x7f338806e9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:35:31.996 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:31.982+0000 7f338f544700 1 -- 172.21.15.88:0/873279613 shutdown_connections 2023-09-30T01:35:31.996 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:31.982+0000 7f338f544700 1 -- 172.21.15.88:0/873279613 wait complete. 2023-09-30T01:35:31.997 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:31.982+0000 7f338f544700 10 monclient: build_initial_monmap 2023-09-30T01:35:31.997 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:31.982+0000 7f338f544700 10 monclient: monmap: 2023-09-30T01:35:31.997 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2023-09-30T01:35:31.997 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:35:31.997 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 0.000000 2023-09-30T01:35:31.997 INFO:teuthology.orchestra.run.smithi088.stderr:created 0.000000 2023-09-30T01:35:31.997 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2023-09-30T01:35:31.997 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 1 2023-09-30T01:35:31.997 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.noname-a 2023-09-30T01:35:31.997 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.noname-f 2023-09-30T01:35:31.997 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.noname-b 2023-09-30T01:35:31.997 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.noname-g 2023-09-30T01:35:31.998 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.noname-c 2023-09-30T01:35:31.998 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.noname-h 2023-09-30T01:35:31.998 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.noname-d 2023-09-30T01:35:31.998 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.noname-i 2023-09-30T01:35:31.998 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.noname-e 2023-09-30T01:35:31.998 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:35:31.998 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:31.982+0000 7f338f544700 1 Processor -- start 2023-09-30T01:35:31.998 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:31.982+0000 7f338f544700 1 -- start start 2023-09-30T01:35:31.998 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:31.982+0000 7f338f544700 10 monclient: init 2023-09-30T01:35:32.002 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:31.986+0000 7f338f544700 10 monclient: _reopen_session rank -1 2023-09-30T01:35:32.002 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:31.986+0000 7f338f544700 10 monclient: _add_conns ranks=[4,1,2,5,8,0,6,7,3] 2023-09-30T01:35:32.003 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:31.986+0000 7f338f544700 1 --2- >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f338809ffd0 0x7f3388137440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:35:32.003 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:31.986+0000 7f338f544700 10 monclient(hunting): picked mon.noname-c con 0x7f338809ffd0 addr [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] 2023-09-30T01:35:32.003 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:31.986+0000 7f338f544700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f33880a3380 0x7f3388131f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:35:32.003 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:31.986+0000 7f338f544700 10 monclient(hunting): picked mon.noname-f con 0x7f33880a3380 addr [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] 2023-09-30T01:35:32.003 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:31.986+0000 7f338f544700 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f33881333b0 0x7f338809e660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:35:32.003 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:31.986+0000 7f338f544700 10 monclient(hunting): picked mon.noname-b con 0x7f33881333b0 addr [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] 2023-09-30T01:35:32.003 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:31.986+0000 7f338f544700 10 monclient(hunting): start opening mon connection 2023-09-30T01:35:32.003 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:31.986+0000 7f338f544700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3388142f40 con 0x7f33880a3380 2023-09-30T01:35:32.003 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:31.986+0000 7f338f544700 10 monclient(hunting): start opening mon connection 2023-09-30T01:35:32.003 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:31.986+0000 7f338f544700 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f338811f100 con 0x7f33881333b0 2023-09-30T01:35:32.004 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:31.986+0000 7f338f544700 10 monclient(hunting): start opening mon connection 2023-09-30T01:35:32.004 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:31.986+0000 7f338f544700 1 -- --> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] -- mon_getmap magic: 0 v1 -- 0x7f33880a0ac0 con 0x7f338809ffd0 2023-09-30T01:35:32.004 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:31.986+0000 7f338f544700 10 monclient(hunting): _renew_subs 2023-09-30T01:35:32.004 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:31.986+0000 7f338f544700 10 monclient(hunting): authenticate will time out at 1004.024430s 2023-09-30T01:35:32.004 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:31.986+0000 7f338e542700 1 --2- >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f338809ffd0 0x7f3388137440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:35:32.004 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:31.986+0000 7f338e542700 1 --2- >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f338809ffd0 0x7f3388137440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3302/0 says I am v2:172.21.15.88:44518/0 (socket says 172.21.15.88:44518) 2023-09-30T01:35:32.004 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:31.986+0000 7f338e542700 1 -- 172.21.15.88:0/3417964196 learned_addr learned my addr 172.21.15.88:0/3417964196 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-09-30T01:35:32.004 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:31.986+0000 7f338e542700 10 monclient(hunting): get_auth_request con 0x7f338809ffd0 auth_method 0 2023-09-30T01:35:32.004 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:31.986+0000 7f338e542700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:35:32.004 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:31.986+0000 7f338e542700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:35:32.004 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:31.986+0000 7f338e542700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:35:32.004 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:31.986+0000 7f338e542700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:35:32.005 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:31.986+0000 7f338e542700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:35:32.005 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:31.986+0000 7f338ed43700 1 --2- 172.21.15.88:0/3417964196 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f33881333b0 0x7f338809e660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:35:32.005 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:31.986+0000 7f338e542700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:35:32.005 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:31.986+0000 7f338ed43700 10 monclient(hunting): get_auth_request con 0x7f33881333b0 auth_method 0 2023-09-30T01:35:32.005 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:31.986+0000 7f338dd41700 1 --2- 172.21.15.88:0/3417964196 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f33880a3380 0x7f3388131f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:35:32.005 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:31.986+0000 7f338ed43700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:35:32.005 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:31.986+0000 7f338ed43700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:35:32.005 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:31.986+0000 7f338ed43700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:35:32.005 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:31.986+0000 7f338dd41700 10 monclient(hunting): get_auth_request con 0x7f33880a3380 auth_method 0 2023-09-30T01:35:32.005 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:31.986+0000 7f338dd41700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:35:32.005 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:31.986+0000 7f338dd41700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:35:32.006 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:31.986+0000 7f338dd41700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:35:32.006 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:31.986+0000 7f338ed43700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:35:32.006 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:31.986+0000 7f338ed43700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:35:32.006 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:31.986+0000 7f338ed43700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:35:32.006 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:31.986+0000 7f338e542700 10 monclient(hunting): handle_auth_done global_id 4621 payload 824 2023-09-30T01:35:32.006 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:31.986+0000 7f338e542700 1 -- 172.21.15.88:0/3417964196 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f33881333b0 msgr2=0x7f338809e660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:35:32.006 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:31.986+0000 7f338e542700 1 --2- 172.21.15.88:0/3417964196 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f33881333b0 0x7f338809e660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:35:32.006 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:31.986+0000 7f338e542700 1 -- 172.21.15.88:0/3417964196 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f33880a3380 msgr2=0x7f3388131f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:35:32.006 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:31.986+0000 7f338e542700 1 --2- 172.21.15.88:0/3417964196 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f33880a3380 0x7f3388131f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:35:32.006 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:31.986+0000 7f338e542700 10 monclient: _finish_hunting 0 2023-09-30T01:35:32.006 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:31.986+0000 7f338e542700 1 monclient: found mon.noname-c 2023-09-30T01:35:32.006 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:31.986+0000 7f338e542700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:35:32.007 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:31.986+0000 7f338e542700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.88:3302/0 2023-09-30T01:35:32.007 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:31.986+0000 7f338e542700 1 -- 172.21.15.88:0/3417964196 --> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33840094f0 con 0x7f338809ffd0 2023-09-30T01:35:32.007 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:31.986+0000 7f338dd41700 1 --2- 172.21.15.88:0/3417964196 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f33880a3380 0x7f3388131f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-30T01:35:32.007 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:31.986+0000 7f338e542700 1 --2- 172.21.15.88:0/3417964196 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f338809ffd0 0x7f3388137440 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f3384009880 tx=0x7f3384017000 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:35:32.007 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:31.986+0000 7f337e7fc700 1 -- 172.21.15.88:0/3417964196 <== mon.4 v2:172.21.15.88:3302/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f3384006ab0 con 0x7f338809ffd0 2023-09-30T01:35:32.007 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:31.986+0000 7f337e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:35:32.007 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:31.986+0000 7f337e7fc700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T01:35:32.007 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:31.986+0000 7f337e7fc700 10 monclient: dump: 2023-09-30T01:35:32.007 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:35:32.007 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:35:32.007 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:35:32.007 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:35:32.008 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:35:32.008 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:35:32.008 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:35:32.008 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:35:32.008 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:35:32.008 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:35:32.008 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:35:32.008 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:35:32.008 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:35:32.008 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:35:32.008 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:35:32.008 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:35:32.008 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:31.986+0000 7f337e7fc700 10 monclient: _finish_auth 0 2023-09-30T01:35:32.009 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:31.986+0000 7f337e7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:35:32.009 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:31.986+0000 7f337e7fc700 1 -- 172.21.15.88:0/3417964196 <== mon.4 v2:172.21.15.88:3302/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f338401cbd0 con 0x7f338809ffd0 2023-09-30T01:35:32.009 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:31.986+0000 7f338f544700 5 monclient: authenticate success, global_id 4621 2023-09-30T01:35:32.009 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:31.986+0000 7f337e7fc700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:35:32.009 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:31.986+0000 7f338f544700 10 monclient: _renew_subs 2023-09-30T01:35:32.009 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:31.986+0000 7f337e7fc700 1 -- 172.21.15.88:0/3417964196 <== mon.4 v2:172.21.15.88:3302/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f338401ad50 con 0x7f338809ffd0 2023-09-30T01:35:32.009 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:31.986+0000 7f338f544700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.88:3302/0 2023-09-30T01:35:32.009 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:31.986+0000 7f338f544700 1 -- 172.21.15.88:0/3417964196 --> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3388097ce0 con 0x7f338809ffd0 2023-09-30T01:35:32.009 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:31.986+0000 7f338cd3f700 4 set_mon_vals no callback set 2023-09-30T01:35:32.009 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:31.986+0000 7f337e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:35:32.009 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:31.986+0000 7f337e7fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T01:35:32.009 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:31.986+0000 7f337e7fc700 10 monclient: dump: 2023-09-30T01:35:32.010 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:35:32.010 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:35:32.010 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:35:32.010 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:35:32.010 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:35:32.010 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:35:32.010 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:35:32.010 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:35:32.010 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:35:32.010 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:35:32.010 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:35:32.010 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:35:32.011 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:35:32.011 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:35:32.011 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:35:32.011 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:35:32.011 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:31.986+0000 7f338f544700 10 monclient: _renew_subs 2023-09-30T01:35:32.011 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:31.986+0000 7f338f544700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.88:3302/0 2023-09-30T01:35:32.011 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:31.986+0000 7f338f544700 1 -- 172.21.15.88:0/3417964196 --> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f338811d280 con 0x7f338809ffd0 2023-09-30T01:35:32.011 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:31.986+0000 7f337e7fc700 1 -- 172.21.15.88:0/3417964196 <== mon.4 v2:172.21.15.88:3302/0 4 ==== mgrmap(e 5) v1 ==== 40144+0+0 (secure 0 0 0) 0x7f338401cd40 con 0x7f338809ffd0 2023-09-30T01:35:32.011 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:31.986+0000 7f338f544700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:35:32.011 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:31.986+0000 7f338f544700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:35:32.011 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:31.986+0000 7f338f544700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.88:3302/0 2023-09-30T01:35:32.011 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:31.986+0000 7f338f544700 1 -- 172.21.15.88:0/3417964196 --> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f33880710d0 con 0x7f338809ffd0 2023-09-30T01:35:32.012 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:31.986+0000 7f337e7fc700 1 --2- 172.21.15.88:0/3417964196 >> v2:172.21.15.107:6812/29926 conn(0x7f336c033360 0x7f336c035810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:35:32.012 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:31.986+0000 7f337e7fc700 1 -- 172.21.15.88:0/3417964196 <== mon.4 v2:172.21.15.88:3302/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 3502+0+0 (secure 0 0 0) 0x7f3384045260 con 0x7f338809ffd0 2023-09-30T01:35:32.012 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:31.990+0000 7f338dd41700 1 --2- 172.21.15.88:0/3417964196 >> v2:172.21.15.107:6812/29926 conn(0x7f336c033360 0x7f336c035810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:35:32.012 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:31.990+0000 7f338dd41700 10 monclient: get_auth_request con 0x7f336c033360 auth_method 0 2023-09-30T01:35:32.012 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:31.990+0000 7f338dd41700 1 --2- 172.21.15.88:0/3417964196 >> v2:172.21.15.107:6812/29926 conn(0x7f336c033360 0x7f336c035810 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f338811cfd0 tx=0x7f3378008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:35:32.012 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:31.994+0000 7f337e7fc700 1 -- 172.21.15.88:0/3417964196 <== mon.4 v2:172.21.15.88:3302/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f338402d980 con 0x7f338809ffd0 2023-09-30T01:35:32.012 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:31.994+0000 7f337e7fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:35:32.012 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:31.994+0000 7f337e7fc700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:35:32.198 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:32.194+0000 7f338f544700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T01:35:32.198 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:32.194+0000 7f338f544700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:35:32.198 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:32.194+0000 7f338f544700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.88:3302/0 2023-09-30T01:35:32.198 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:32.194+0000 7f338f544700 1 -- 172.21.15.88:0/3417964196 --> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f338811dee0 con 0x7f338809ffd0 2023-09-30T01:35:32.199 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:32.194+0000 7f337e7fc700 1 -- 172.21.15.88:0/3417964196 <== mon.4 v2:172.21.15.88:3302/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f3384015080 con 0x7f338809ffd0 2023-09-30T01:35:32.199 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:32.194+0000 7f337e7fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:35:32.199 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:32.194+0000 7f337e7fc700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:35:32.199 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-09-30T01:35:32.199 INFO:teuthology.orchestra.run.smithi088.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:35:32.201 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:32.198+0000 7f338f544700 1 -- 172.21.15.88:0/3417964196 >> v2:172.21.15.107:6812/29926 conn(0x7f336c033360 msgr2=0x7f336c035810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:35:32.201 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:32.198+0000 7f338f544700 1 --2- 172.21.15.88:0/3417964196 >> v2:172.21.15.107:6812/29926 conn(0x7f336c033360 0x7f336c035810 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f338811cfd0 tx=0x7f3378008040 comp rx=0 tx=0).stop 2023-09-30T01:35:32.201 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:32.198+0000 7f338f544700 10 monclient: shutdown 2023-09-30T01:35:32.201 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:32.198+0000 7f338f544700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:35:32.201 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:32.198+0000 7f338f544700 1 -- 172.21.15.88:0/3417964196 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f338809ffd0 msgr2=0x7f3388137440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:35:32.201 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:32.198+0000 7f338f544700 1 --2- 172.21.15.88:0/3417964196 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f338809ffd0 0x7f3388137440 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f3384009880 tx=0x7f3384017000 comp rx=0 tx=0).stop 2023-09-30T01:35:32.201 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:32.198+0000 7f338f544700 1 -- 172.21.15.88:0/3417964196 shutdown_connections 2023-09-30T01:35:32.202 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:32.198+0000 7f338f544700 1 --2- 172.21.15.88:0/3417964196 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f33881333b0 0x7f338809e660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:35:32.202 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:32.198+0000 7f338f544700 1 --2- 172.21.15.88:0/3417964196 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f33880a3380 0x7f3388131f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:35:32.202 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:32.198+0000 7f338f544700 1 --2- 172.21.15.88:0/3417964196 >> v2:172.21.15.107:6812/29926 conn(0x7f336c033360 0x7f336c035810 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:35:32.202 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:32.198+0000 7f338f544700 1 --2- 172.21.15.88:0/3417964196 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f338809ffd0 0x7f3388137440 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:35:32.202 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:32.198+0000 7f338f544700 1 -- 172.21.15.88:0/3417964196 >> 172.21.15.88:0/3417964196 conn(0x7f338806e5e0 msgr2=0x7f3388135c20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:35:32.202 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:32.198+0000 7f338f544700 1 -- 172.21.15.88:0/3417964196 shutdown_connections 2023-09-30T01:35:32.202 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:32.198+0000 7f338f544700 1 -- 172.21.15.88:0/3417964196 wait complete. 2023-09-30T01:35:35.212 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:35:35.331 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:35.326+0000 7f54d5aef700 10 monclient: get_monmap_and_config 2023-09-30T01:35:35.331 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:35.326+0000 7f54d5aef700 10 monclient: build_initial_monmap 2023-09-30T01:35:35.331 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:35.326+0000 7f54d5aef700 10 monclient: monmap: 2023-09-30T01:35:35.331 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2023-09-30T01:35:35.332 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:35:35.332 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:35:35.331050+0000 2023-09-30T01:35:35.332 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:35:35.331050+0000 2023-09-30T01:35:35.332 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2023-09-30T01:35:35.332 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:35:35.332 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a-legacy 2023-09-30T01:35:35.332 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.107:6789/0 mon.noname-f-legacy 2023-09-30T01:35:35.333 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.noname-b 2023-09-30T01:35:35.333 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.noname-g 2023-09-30T01:35:35.333 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.noname-c 2023-09-30T01:35:35.333 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.noname-h 2023-09-30T01:35:35.333 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.noname-d 2023-09-30T01:35:35.333 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.noname-i 2023-09-30T01:35:35.333 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.noname-e 2023-09-30T01:35:35.334 INFO:teuthology.orchestra.run.smithi088.stderr:9: v2:172.21.15.88:3300/0 mon.noname-a 2023-09-30T01:35:35.334 INFO:teuthology.orchestra.run.smithi088.stderr:10: v2:172.21.15.107:3300/0 mon.noname-f 2023-09-30T01:35:35.334 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:35:35.345 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:35.330+0000 7f54d5aef700 1 Processor -- start 2023-09-30T01:35:35.345 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:35.330+0000 7f54d5aef700 1 -- start start 2023-09-30T01:35:35.345 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:35.330+0000 7f54d5aef700 10 monclient: init 2023-09-30T01:35:35.345 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:35.330+0000 7f54d5aef700 10 monclient: _reopen_session rank -1 2023-09-30T01:35:35.345 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:35.330+0000 7f54d5aef700 10 monclient: _add_conns ranks=[7,8,5,9,1,4,10,6,2,3,0] 2023-09-30T01:35:35.346 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:35.330+0000 7f54d5aef700 1 --2- >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7f54d00966a0 0x7f54d0096a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:35:35.346 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:35.330+0000 7f54d5aef700 10 monclient(hunting): picked mon.noname-i con 0x7f54d00966a0 addr [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] 2023-09-30T01:35:35.346 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:35.330+0000 7f54d5aef700 1 --2- >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f54d0097e40 0x7f54d0098210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:35:35.346 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:35.330+0000 7f54d5aef700 10 monclient(hunting): picked mon.noname-e con 0x7f54d0097e40 addr [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] 2023-09-30T01:35:35.346 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:35.330+0000 7f54d5aef700 1 --2- >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7f54d009dd70 0x7f54d009e1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:35:35.346 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:35.330+0000 7f54d5aef700 10 monclient(hunting): picked mon.noname-h con 0x7f54d009dd70 addr [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] 2023-09-30T01:35:35.347 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:35.330+0000 7f54d5aef700 10 monclient(hunting): start opening mon connection 2023-09-30T01:35:35.347 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:35.330+0000 7f54d5aef700 1 -- --> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] -- mon_getmap magic: 0 v1 -- 0x7f54d0063c40 con 0x7f54d009dd70 2023-09-30T01:35:35.347 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:35.330+0000 7f54d5aef700 10 monclient(hunting): start opening mon connection 2023-09-30T01:35:35.347 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:35.330+0000 7f54d5aef700 1 -- --> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] -- mon_getmap magic: 0 v1 -- 0x7f54d0097460 con 0x7f54d00966a0 2023-09-30T01:35:35.347 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:35.330+0000 7f54d5aef700 10 monclient(hunting): start opening mon connection 2023-09-30T01:35:35.347 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:35.330+0000 7f54d5aef700 1 -- --> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] -- mon_getmap magic: 0 v1 -- 0x7f54d0063950 con 0x7f54d0097e40 2023-09-30T01:35:35.347 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:35.330+0000 7f54d5aef700 10 monclient(hunting): _renew_subs 2023-09-30T01:35:35.348 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:35.330+0000 7f54d5aef700 10 monclient(hunting): authenticate will time out at 1007.369485s 2023-09-30T01:35:35.348 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:35.330+0000 7f54cffff700 1 --2- >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f54d0097e40 0x7f54d0098210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:35:35.348 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:35.330+0000 7f54cffff700 1 --2- >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f54d0097e40 0x7f54d0098210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3304/0 says I am v2:172.21.15.88:39714/0 (socket says 172.21.15.88:39714) 2023-09-30T01:35:35.348 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:35.330+0000 7f54cffff700 1 -- 172.21.15.88:0/597199276 learned_addr learned my addr 172.21.15.88:0/597199276 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-09-30T01:35:35.348 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:35.330+0000 7f54cffff700 10 monclient(hunting): get_auth_request con 0x7f54d0097e40 auth_method 0 2023-09-30T01:35:35.348 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:35.330+0000 7f54cffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:35:35.349 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:35.330+0000 7f54cffff700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:35:35.349 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:35.330+0000 7f54cffff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:35:35.349 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:35.330+0000 7f54cffff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:35:35.349 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:35.330+0000 7f54cffff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:35:35.349 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:35.330+0000 7f54cffff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:35:35.350 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:35.330+0000 7f54d52ee700 1 --2- 172.21.15.88:0/597199276 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7f54d009dd70 0x7f54d009e1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:35:35.350 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:35.330+0000 7f54d4aed700 1 --2- 172.21.15.88:0/597199276 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7f54d00966a0 0x7f54d0096a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:35:35.350 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:35.330+0000 7f54d52ee700 10 monclient(hunting): get_auth_request con 0x7f54d009dd70 auth_method 0 2023-09-30T01:35:35.350 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:35.330+0000 7f54d52ee700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:35:35.350 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:35.330+0000 7f54d52ee700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:35:35.350 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:35.330+0000 7f54d52ee700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:35:35.350 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:35.330+0000 7f54d4aed700 10 monclient(hunting): get_auth_request con 0x7f54d00966a0 auth_method 0 2023-09-30T01:35:35.351 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:35.330+0000 7f54d4aed700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:35:35.351 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:35.330+0000 7f54d4aed700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:35:35.351 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:35.330+0000 7f54d4aed700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:35:35.351 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:35.330+0000 7f54cffff700 10 monclient(hunting): handle_auth_done global_id 4661 payload 274 2023-09-30T01:35:35.351 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:35.330+0000 7f54cffff700 1 -- 172.21.15.88:0/597199276 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7f54d00966a0 msgr2=0x7f54d0096a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:35:35.351 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:35.330+0000 7f54cffff700 1 --2- 172.21.15.88:0/597199276 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7f54d00966a0 0x7f54d0096a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:35:35.351 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:35.330+0000 7f54cffff700 1 -- 172.21.15.88:0/597199276 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7f54d009dd70 msgr2=0x7f54d009e1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:35:35.351 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:35.330+0000 7f54cffff700 1 --2- 172.21.15.88:0/597199276 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7f54d009dd70 0x7f54d009e1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:35:35.351 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:35.330+0000 7f54cffff700 10 monclient: _finish_hunting 0 2023-09-30T01:35:35.352 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:35.330+0000 7f54cffff700 1 monclient: found mon.noname-e 2023-09-30T01:35:35.352 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:35.330+0000 7f54cffff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:35:35.352 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:35.330+0000 7f54cffff700 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.88:3304/0 2023-09-30T01:35:35.352 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:35.330+0000 7f54cffff700 1 -- 172.21.15.88:0/597199276 --> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54d004bb20 con 0x7f54d0097e40 2023-09-30T01:35:35.352 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:35.330+0000 7f54cffff700 1 --2- 172.21.15.88:0/597199276 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f54d0097e40 0x7f54d0098210 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f54c40099e0 tx=0x7f54c4009250 comp rx=0 tx=0).ready entity=mon.8 client_cookie=d80ae6fc075d26f0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:35:35.352 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:35.330+0000 7f54cf7fe700 1 -- 172.21.15.88:0/597199276 <== mon.8 v2:172.21.15.88:3304/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f54c400bce0 con 0x7f54d0097e40 2023-09-30T01:35:35.352 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:35.330+0000 7f54cf7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:35:35.352 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:35.330+0000 7f54cf7fe700 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2023-09-30T01:35:35.352 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:35.330+0000 7f54cf7fe700 10 monclient: dump: 2023-09-30T01:35:35.352 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:35:35.353 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:35:35.353 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:35:35.353 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:35:35.353 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:35:35.353 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:35:35.353 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:35:35.353 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:35:35.353 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:35:35.353 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:35:35.353 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:35:35.353 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:35:35.353 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:35:35.354 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:35:35.354 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:35:35.354 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:35:35.354 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:35.330+0000 7f54cf7fe700 10 monclient: _finish_auth 0 2023-09-30T01:35:35.354 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:35.330+0000 7f54cf7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:35:35.354 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:35.330+0000 7f54d5aef700 5 monclient: authenticate success, global_id 4661 2023-09-30T01:35:35.354 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:35.330+0000 7f54cf7fe700 1 -- 172.21.15.88:0/597199276 <== mon.8 v2:172.21.15.88:3304/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f54c400be60 con 0x7f54d0097e40 2023-09-30T01:35:35.354 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:35.330+0000 7f54cf7fe700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:35:35.354 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:35.330+0000 7f54d5aef700 10 monclient: get_monmap_and_config success 2023-09-30T01:35:35.354 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:35.330+0000 7f54cf7fe700 1 -- 172.21.15.88:0/597199276 <== mon.8 v2:172.21.15.88:3304/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f54c40118a0 con 0x7f54d0097e40 2023-09-30T01:35:35.354 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:35.330+0000 7f54d5aef700 4 set_mon_vals no callback set 2023-09-30T01:35:35.355 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:35.330+0000 7f54cf7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:35:35.355 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:35.330+0000 7f54cf7fe700 10 monclient: got monmap 1 from mon.e (according to old e1) 2023-09-30T01:35:35.355 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:35.330+0000 7f54cf7fe700 10 monclient: dump: 2023-09-30T01:35:35.355 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:35:35.355 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:35:35.355 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:35:35.355 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:35:35.355 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:35:35.355 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:35:35.355 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:35:35.355 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:35:35.356 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:35:35.356 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:35:35.356 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:35:35.356 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:35:35.356 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:35:35.356 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:35:35.356 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:35:35.356 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:35:35.356 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:35.330+0000 7f54d5aef700 10 monclient: shutdown 2023-09-30T01:35:35.356 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:35.330+0000 7f54d5aef700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:35:35.356 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:35.330+0000 7f54d5aef700 1 -- 172.21.15.88:0/597199276 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f54d0097e40 msgr2=0x7f54d0098210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:35:35.356 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:35.330+0000 7f54d5aef700 1 --2- 172.21.15.88:0/597199276 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f54d0097e40 0x7f54d0098210 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f54c40099e0 tx=0x7f54c4009250 comp rx=0 tx=0).stop 2023-09-30T01:35:35.357 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:35.330+0000 7f54d5aef700 1 -- 172.21.15.88:0/597199276 shutdown_connections 2023-09-30T01:35:35.357 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:35.330+0000 7f54d5aef700 1 --2- 172.21.15.88:0/597199276 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7f54d009dd70 0x7f54d009e1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:35:35.357 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:35.330+0000 7f54d5aef700 1 --2- 172.21.15.88:0/597199276 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f54d0097e40 0x7f54d0098210 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:35:35.357 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:35.330+0000 7f54d5aef700 1 --2- 172.21.15.88:0/597199276 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7f54d00966a0 0x7f54d0096a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:35:35.357 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:35.330+0000 7f54d5aef700 1 -- 172.21.15.88:0/597199276 >> 172.21.15.88:0/597199276 conn(0x7f54d009fe70 msgr2=0x7f54d00a0270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:35:35.357 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:35.330+0000 7f54d5aef700 1 -- 172.21.15.88:0/597199276 shutdown_connections 2023-09-30T01:35:35.357 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:35.330+0000 7f54d5aef700 1 -- 172.21.15.88:0/597199276 wait complete. 2023-09-30T01:35:35.357 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:35.334+0000 7f54d5aef700 10 monclient: build_initial_monmap 2023-09-30T01:35:35.357 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:35.334+0000 7f54d5aef700 10 monclient: monmap: 2023-09-30T01:35:35.357 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2023-09-30T01:35:35.357 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:35:35.358 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 0.000000 2023-09-30T01:35:35.358 INFO:teuthology.orchestra.run.smithi088.stderr:created 0.000000 2023-09-30T01:35:35.358 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2023-09-30T01:35:35.358 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 1 2023-09-30T01:35:35.358 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.noname-a 2023-09-30T01:35:35.358 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.noname-f 2023-09-30T01:35:35.358 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.noname-b 2023-09-30T01:35:35.358 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.noname-g 2023-09-30T01:35:35.358 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.noname-c 2023-09-30T01:35:35.358 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.noname-h 2023-09-30T01:35:35.358 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.noname-d 2023-09-30T01:35:35.358 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.noname-i 2023-09-30T01:35:35.359 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.noname-e 2023-09-30T01:35:35.359 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:35:35.359 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:35.334+0000 7f54d5aef700 1 Processor -- start 2023-09-30T01:35:35.359 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:35.334+0000 7f54d5aef700 1 -- start start 2023-09-30T01:35:35.359 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:35.334+0000 7f54d5aef700 10 monclient: init 2023-09-30T01:35:35.359 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:35.334+0000 7f54d5aef700 10 monclient: _reopen_session rank -1 2023-09-30T01:35:35.359 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:35.334+0000 7f54d5aef700 10 monclient: _add_conns ranks=[8,4,6,0,5,3,2,1,7] 2023-09-30T01:35:35.359 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:35.334+0000 7f54d5aef700 1 --2- >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f54d00966a0 0x7f54d0142690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:35:35.359 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:35.334+0000 7f54d5aef700 10 monclient(hunting): picked mon.noname-e con 0x7f54d00966a0 addr [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] 2023-09-30T01:35:35.359 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:35.334+0000 7f54d5aef700 1 --2- >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f54d0097e40 0x7f54d0142bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:35:35.359 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:35.334+0000 7f54d5aef700 10 monclient(hunting): picked mon.noname-c con 0x7f54d0097e40 addr [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] 2023-09-30T01:35:35.360 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:35.334+0000 7f54d5aef700 1 --2- >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7f54d009dd70 0x7f54d0143110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:35:35.360 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:35.334+0000 7f54d5aef700 10 monclient(hunting): picked mon.noname-d con 0x7f54d009dd70 addr [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] 2023-09-30T01:35:35.360 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:35.334+0000 7f54d5aef700 10 monclient(hunting): start opening mon connection 2023-09-30T01:35:35.360 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:35.334+0000 7f54d5aef700 1 -- --> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] -- mon_getmap magic: 0 v1 -- 0x7f54d00915b0 con 0x7f54d0097e40 2023-09-30T01:35:35.360 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:35.334+0000 7f54d5aef700 10 monclient(hunting): start opening mon connection 2023-09-30T01:35:35.360 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:35.334+0000 7f54d5aef700 1 -- --> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] -- mon_getmap magic: 0 v1 -- 0x7f54d0090070 con 0x7f54d009dd70 2023-09-30T01:35:35.361 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:35.334+0000 7f54d5aef700 10 monclient(hunting): start opening mon connection 2023-09-30T01:35:35.361 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:35.334+0000 7f54d5aef700 1 -- --> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] -- mon_getmap magic: 0 v1 -- 0x7f54d00906c0 con 0x7f54d00966a0 2023-09-30T01:35:35.361 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:35.334+0000 7f54d5aef700 10 monclient(hunting): _renew_subs 2023-09-30T01:35:35.361 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:35.334+0000 7f54d5aef700 10 monclient(hunting): authenticate will time out at 1007.374093s 2023-09-30T01:35:35.361 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:35.334+0000 7f54d4aed700 1 --2- >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f54d00966a0 0x7f54d0142690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:35:35.361 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:35.334+0000 7f54d52ee700 1 --2- >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7f54d009dd70 0x7f54d0143110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:35:35.361 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:35.334+0000 7f54d52ee700 1 --2- >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7f54d009dd70 0x7f54d0143110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3303/0 says I am v2:172.21.15.88:50888/0 (socket says 172.21.15.88:50888) 2023-09-30T01:35:35.361 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:35.334+0000 7f54cffff700 1 --2- >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f54d0097e40 0x7f54d0142bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:35:35.361 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:35.334+0000 7f54d52ee700 1 -- 172.21.15.88:0/2711121114 learned_addr learned my addr 172.21.15.88:0/2711121114 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-09-30T01:35:35.361 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:35.334+0000 7f54d52ee700 10 monclient(hunting): get_auth_request con 0x7f54d009dd70 auth_method 0 2023-09-30T01:35:35.361 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:35.334+0000 7f54d52ee700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:35:35.362 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:35.334+0000 7f54d52ee700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:35:35.362 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:35.334+0000 7f54d52ee700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:35:35.362 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:35.334+0000 7f54cffff700 10 monclient(hunting): get_auth_request con 0x7f54d0097e40 auth_method 0 2023-09-30T01:35:35.362 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:35.334+0000 7f54cffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:35:35.362 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:35.334+0000 7f54cffff700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:35:35.362 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:35.334+0000 7f54cffff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:35:35.362 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:35.334+0000 7f54d4aed700 10 monclient(hunting): get_auth_request con 0x7f54d00966a0 auth_method 0 2023-09-30T01:35:35.362 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:35.334+0000 7f54d4aed700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:35:35.362 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:35.334+0000 7f54d4aed700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:35:35.362 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:35.334+0000 7f54d4aed700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:35:35.362 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:35.334+0000 7f54d52ee700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:35:35.363 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:35.334+0000 7f54d52ee700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:35:35.363 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:35.334+0000 7f54d52ee700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:35:35.363 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:35.334+0000 7f54cffff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:35:35.363 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:35.334+0000 7f54cffff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:35:35.363 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:35.334+0000 7f54cffff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:35:35.363 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:35.334+0000 7f54d4aed700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:35:35.363 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:35.334+0000 7f54d4aed700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:35:35.363 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:35.334+0000 7f54d4aed700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:35:35.363 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:35.334+0000 7f54d52ee700 10 monclient(hunting): handle_auth_done global_id 4506 payload 824 2023-09-30T01:35:35.363 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:35.334+0000 7f54d52ee700 1 -- 172.21.15.88:0/2711121114 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f54d00966a0 msgr2=0x7f54d0142690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:35:35.363 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:35.334+0000 7f54d52ee700 1 --2- 172.21.15.88:0/2711121114 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f54d00966a0 0x7f54d0142690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:35:35.363 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:35.334+0000 7f54d52ee700 1 -- 172.21.15.88:0/2711121114 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f54d0097e40 msgr2=0x7f54d0142bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:35:35.364 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:35.334+0000 7f54d52ee700 1 --2- 172.21.15.88:0/2711121114 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f54d0097e40 0x7f54d0142bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:35:35.364 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:35.334+0000 7f54d52ee700 10 monclient: _finish_hunting 0 2023-09-30T01:35:35.364 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:35.334+0000 7f54d52ee700 1 monclient: found mon.noname-d 2023-09-30T01:35:35.364 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:35.334+0000 7f54d52ee700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:35:35.364 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:35.334+0000 7f54d52ee700 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.88:3303/0 2023-09-30T01:35:35.364 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:35.334+0000 7f54d52ee700 1 -- 172.21.15.88:0/2711121114 --> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54c40094f0 con 0x7f54d009dd70 2023-09-30T01:35:35.364 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:35.334+0000 7f54d52ee700 1 --2- 172.21.15.88:0/2711121114 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7f54d009dd70 0x7f54d0143110 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f54c8007f20 tx=0x7f54c8009840 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:35:35.364 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:35.334+0000 7f54ccff9700 1 -- 172.21.15.88:0/2711121114 <== mon.6 v2:172.21.15.88:3303/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f54c80098d0 con 0x7f54d009dd70 2023-09-30T01:35:35.364 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:35.334+0000 7f54ccff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:35:35.364 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:35.334+0000 7f54ccff9700 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2023-09-30T01:35:35.364 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:35.334+0000 7f54ccff9700 10 monclient: dump: 2023-09-30T01:35:35.365 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:35:35.365 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:35:35.365 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:35:35.365 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:35:35.365 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:35:35.365 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:35:35.365 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:35:35.365 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:35:35.365 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:35:35.365 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:35:35.365 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:35:35.366 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:35:35.366 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:35:35.366 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:35:35.366 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:35:35.366 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:35:35.366 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:35.334+0000 7f54ccff9700 10 monclient: _finish_auth 0 2023-09-30T01:35:35.366 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:35.334+0000 7f54ccff9700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:35:35.366 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:35.334+0000 7f54ccff9700 1 -- 172.21.15.88:0/2711121114 <== mon.6 v2:172.21.15.88:3303/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f54c8014420 con 0x7f54d009dd70 2023-09-30T01:35:35.366 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:35.334+0000 7f54d5aef700 5 monclient: authenticate success, global_id 4506 2023-09-30T01:35:35.366 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:35.334+0000 7f54d5aef700 10 monclient: _renew_subs 2023-09-30T01:35:35.367 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:35.334+0000 7f54d5aef700 10 monclient: _send_mon_message to mon.d at v2:172.21.15.88:3303/0 2023-09-30T01:35:35.367 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:35.334+0000 7f54d5aef700 1 -- 172.21.15.88:0/2711121114 --> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f54d0099830 con 0x7f54d009dd70 2023-09-30T01:35:35.367 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:35.334+0000 7f54ccff9700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:35:35.367 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:35.334+0000 7f54ccff9700 1 -- 172.21.15.88:0/2711121114 <== mon.6 v2:172.21.15.88:3303/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f54c8014a10 con 0x7f54d009dd70 2023-09-30T01:35:35.367 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:35.334+0000 7f54ccff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:35:35.367 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:35.334+0000 7f54ccff9700 10 monclient: got monmap 1 from mon.d (according to old e1) 2023-09-30T01:35:35.367 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:35.334+0000 7f54ceffd700 4 set_mon_vals no callback set 2023-09-30T01:35:35.367 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:35.334+0000 7f54ccff9700 10 monclient: dump: 2023-09-30T01:35:35.367 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:35:35.367 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:35:35.367 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:35:35.367 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:35:35.368 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:35:35.368 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:35:35.368 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:35:35.368 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:35:35.368 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:35:35.368 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:35:35.368 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:35:35.368 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:35:35.368 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:35:35.368 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:35:35.368 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:35:35.368 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:35:35.369 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:35.334+0000 7f54d5aef700 10 monclient: _renew_subs 2023-09-30T01:35:35.369 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:35.334+0000 7f54d5aef700 10 monclient: _send_mon_message to mon.d at v2:172.21.15.88:3303/0 2023-09-30T01:35:35.369 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:35.334+0000 7f54d5aef700 1 -- 172.21.15.88:0/2711121114 --> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f54d00928e0 con 0x7f54d009dd70 2023-09-30T01:35:35.369 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:35.338+0000 7f54d5aef700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:35:35.369 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:35.338+0000 7f54d5aef700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:35:35.369 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:35.338+0000 7f54d5aef700 10 monclient: _send_mon_message to mon.d at v2:172.21.15.88:3303/0 2023-09-30T01:35:35.369 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:35.338+0000 7f54d5aef700 1 -- 172.21.15.88:0/2711121114 --> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f54d0071960 con 0x7f54d009dd70 2023-09-30T01:35:35.369 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:35.338+0000 7f54ccff9700 1 -- 172.21.15.88:0/2711121114 <== mon.6 v2:172.21.15.88:3303/0 4 ==== mgrmap(e 5) v1 ==== 40144+0+0 (secure 0 0 0) 0x7f54c8014590 con 0x7f54d009dd70 2023-09-30T01:35:35.369 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:35.338+0000 7f54ccff9700 1 --2- 172.21.15.88:0/2711121114 >> v2:172.21.15.107:6812/29926 conn(0x7f54bc033360 0x7f54bc035810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:35:35.369 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:35.338+0000 7f54ccff9700 1 -- 172.21.15.88:0/2711121114 <== mon.6 v2:172.21.15.88:3303/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 3502+0+0 (secure 0 0 0) 0x7f54c8043930 con 0x7f54d009dd70 2023-09-30T01:35:35.370 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:35.338+0000 7f54d4aed700 1 --2- 172.21.15.88:0/2711121114 >> v2:172.21.15.107:6812/29926 conn(0x7f54bc033360 0x7f54bc035810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:35:35.370 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:35.338+0000 7f54d4aed700 10 monclient: get_auth_request con 0x7f54bc033360 auth_method 0 2023-09-30T01:35:35.370 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:35.338+0000 7f54d4aed700 1 --2- 172.21.15.88:0/2711121114 >> v2:172.21.15.107:6812/29926 conn(0x7f54bc033360 0x7f54bc035810 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f54c0006aa0 tx=0x7f54c0008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:35:35.370 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:35.346+0000 7f54ccff9700 1 -- 172.21.15.88:0/2711121114 <== mon.6 v2:172.21.15.88:3303/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f54c8014e00 con 0x7f54d009dd70 2023-09-30T01:35:35.370 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:35.346+0000 7f54ccff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:35:35.370 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:35.346+0000 7f54ccff9700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:35:35.548 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:35.546+0000 7f54d5aef700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T01:35:35.548 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:35.546+0000 7f54d5aef700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:35:35.548 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:35.546+0000 7f54d5aef700 10 monclient: _send_mon_message to mon.d at v2:172.21.15.88:3303/0 2023-09-30T01:35:35.548 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:35.546+0000 7f54d5aef700 1 -- 172.21.15.88:0/2711121114 --> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f54d008fa80 con 0x7f54d009dd70 2023-09-30T01:35:35.548 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:35.546+0000 7f54ccff9700 1 -- 172.21.15.88:0/2711121114 <== mon.6 v2:172.21.15.88:3303/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f54c801b310 con 0x7f54d009dd70 2023-09-30T01:35:35.549 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:35.546+0000 7f54ccff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:35:35.549 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:35.546+0000 7f54ccff9700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:35:35.549 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-09-30T01:35:35.549 INFO:teuthology.orchestra.run.smithi088.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:35:35.550 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:35.546+0000 7f54d5aef700 1 -- 172.21.15.88:0/2711121114 >> v2:172.21.15.107:6812/29926 conn(0x7f54bc033360 msgr2=0x7f54bc035810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:35:35.551 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:35.546+0000 7f54d5aef700 1 --2- 172.21.15.88:0/2711121114 >> v2:172.21.15.107:6812/29926 conn(0x7f54bc033360 0x7f54bc035810 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f54c0006aa0 tx=0x7f54c0008040 comp rx=0 tx=0).stop 2023-09-30T01:35:35.551 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:35.546+0000 7f54d5aef700 10 monclient: shutdown 2023-09-30T01:35:35.551 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:35.546+0000 7f54d5aef700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:35:35.551 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:35.546+0000 7f54d5aef700 1 -- 172.21.15.88:0/2711121114 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7f54d009dd70 msgr2=0x7f54d0143110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:35:35.551 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:35.546+0000 7f54d5aef700 1 --2- 172.21.15.88:0/2711121114 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7f54d009dd70 0x7f54d0143110 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f54c8007f20 tx=0x7f54c8009840 comp rx=0 tx=0).stop 2023-09-30T01:35:35.551 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:35.546+0000 7f54d5aef700 1 -- 172.21.15.88:0/2711121114 shutdown_connections 2023-09-30T01:35:35.551 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:35.546+0000 7f54d5aef700 1 --2- 172.21.15.88:0/2711121114 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7f54d009dd70 0x7f54d0143110 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:35:35.551 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:35.546+0000 7f54d5aef700 1 --2- 172.21.15.88:0/2711121114 >> v2:172.21.15.107:6812/29926 conn(0x7f54bc033360 0x7f54bc035810 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:35:35.551 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:35.546+0000 7f54d5aef700 1 --2- 172.21.15.88:0/2711121114 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f54d0097e40 0x7f54d0142bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:35:35.552 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:35.546+0000 7f54d5aef700 1 --2- 172.21.15.88:0/2711121114 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f54d00966a0 0x7f54d0142690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:35:35.552 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:35.546+0000 7f54d5aef700 1 -- 172.21.15.88:0/2711121114 >> 172.21.15.88:0/2711121114 conn(0x7f54d009fe70 msgr2=0x7f54d009e580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:35:35.552 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:35.546+0000 7f54d5aef700 1 -- 172.21.15.88:0/2711121114 shutdown_connections 2023-09-30T01:35:35.552 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:35.546+0000 7f54d5aef700 1 -- 172.21.15.88:0/2711121114 wait complete. 2023-09-30T01:35:38.561 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:35:38.668 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:38.666+0000 7f691c7b0700 10 monclient: get_monmap_and_config 2023-09-30T01:35:38.668 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:38.666+0000 7f691c7b0700 10 monclient: build_initial_monmap 2023-09-30T01:35:38.669 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:38.666+0000 7f691c7b0700 10 monclient: monmap: 2023-09-30T01:35:38.669 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2023-09-30T01:35:38.669 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:35:38.669 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:35:38.668726+0000 2023-09-30T01:35:38.669 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:35:38.668726+0000 2023-09-30T01:35:38.669 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2023-09-30T01:35:38.669 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:35:38.669 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a-legacy 2023-09-30T01:35:38.669 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.107:6789/0 mon.noname-f-legacy 2023-09-30T01:35:38.669 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.noname-b 2023-09-30T01:35:38.669 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.noname-g 2023-09-30T01:35:38.669 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.noname-c 2023-09-30T01:35:38.670 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.noname-h 2023-09-30T01:35:38.670 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.noname-d 2023-09-30T01:35:38.670 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.noname-i 2023-09-30T01:35:38.670 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.noname-e 2023-09-30T01:35:38.670 INFO:teuthology.orchestra.run.smithi088.stderr:9: v2:172.21.15.88:3300/0 mon.noname-a 2023-09-30T01:35:38.670 INFO:teuthology.orchestra.run.smithi088.stderr:10: v2:172.21.15.107:3300/0 mon.noname-f 2023-09-30T01:35:38.670 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:35:38.675 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:38.666+0000 7f691c7b0700 1 Processor -- start 2023-09-30T01:35:38.675 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:38.666+0000 7f691c7b0700 1 -- start start 2023-09-30T01:35:38.675 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:38.666+0000 7f691c7b0700 10 monclient: init 2023-09-30T01:35:38.675 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:38.666+0000 7f691c7b0700 10 monclient: _reopen_session rank -1 2023-09-30T01:35:38.675 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:38.666+0000 7f691c7b0700 10 monclient: _add_conns ranks=[10,0,3,4,7,1,6,9,5,8,2] 2023-09-30T01:35:38.675 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:38.666+0000 7f691c7b0700 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f69140a35f0 0x7f6914133330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:35:38.675 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:38.666+0000 7f691c7b0700 10 monclient(hunting): picked mon.noname-f con 0x7f69140a35f0 addr v2:172.21.15.107:3300/0 2023-09-30T01:35:38.675 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:38.666+0000 7f691c7b0700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f6914133870 addr v1:172.21.15.88:6789/0 2023-09-30T01:35:38.676 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:38.666+0000 7f691c7b0700 1 --2- >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7f6914133c80 0x7f69140a4e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:35:38.676 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:38.666+0000 7f691c7b0700 10 monclient(hunting): picked mon.noname-g con 0x7f6914133c80 addr [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] 2023-09-30T01:35:38.676 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:38.666+0000 7f691c7b0700 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f691405ecb0 con 0x7f6914133870 2023-09-30T01:35:38.676 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:38.666+0000 7f691c7b0700 10 monclient(hunting): start opening mon connection 2023-09-30T01:35:38.676 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:38.666+0000 7f691c7b0700 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6914097ad0 con 0x7f69140a35f0 2023-09-30T01:35:38.676 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:38.666+0000 7f691c7b0700 10 monclient(hunting): start opening mon connection 2023-09-30T01:35:38.676 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:38.666+0000 7f691c7b0700 1 -- --> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69140a0940 con 0x7f6914133c80 2023-09-30T01:35:38.676 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:38.666+0000 7f691c7b0700 10 monclient(hunting): _renew_subs 2023-09-30T01:35:38.676 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:38.666+0000 7f691c7b0700 10 monclient(hunting): authenticate will time out at 1010.706750s 2023-09-30T01:35:38.677 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:38.666+0000 7f6919d4b700 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f6914133870 0x7f69140a46f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:50192/0 (socket says 172.21.15.88:50192) 2023-09-30T01:35:38.677 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:38.666+0000 7f6919d4b700 1 -- 172.21.15.88:0/3194955150 learned_addr learned my addr 172.21.15.88:0/3194955150 (peer_addr_for_me v1:172.21.15.88:0/0) 2023-09-30T01:35:38.677 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:38.666+0000 7f691a54c700 1 --2- 172.21.15.88:0/3194955150 >> v2:172.21.15.107:3300/0 conn(0x7f69140a35f0 0x7f6914133330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:35:38.677 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:38.666+0000 7f691ad4d700 1 --2- 172.21.15.88:0/3194955150 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7f6914133c80 0x7f69140a4e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:35:38.677 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:38.666+0000 7f691ad4d700 10 monclient(hunting): get_auth_request con 0x7f6914133c80 auth_method 0 2023-09-30T01:35:38.677 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:38.666+0000 7f691ad4d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:35:38.677 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:38.666+0000 7f691ad4d700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:35:38.677 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:38.666+0000 7f691ad4d700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:35:38.677 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:38.666+0000 7f691a54c700 10 monclient(hunting): get_auth_request con 0x7f69140a35f0 auth_method 0 2023-09-30T01:35:38.677 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:38.666+0000 7f691a54c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:35:38.678 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:38.666+0000 7f691a54c700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:35:38.678 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:38.666+0000 7f691a54c700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:35:38.678 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:38.666+0000 7f691ad4d700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:35:38.678 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:38.666+0000 7f691ad4d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:35:38.678 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:38.666+0000 7f691ad4d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:35:38.678 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:38.666+0000 7f691a54c700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:35:38.678 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:38.666+0000 7f691a54c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:35:38.678 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:38.666+0000 7f691a54c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:35:38.678 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:38.666+0000 7f691954a700 1 -- 172.21.15.88:0/3194955150 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1123884153 0 0) 0x7f691405ecb0 con 0x7f6914133870 2023-09-30T01:35:38.678 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:38.666+0000 7f691954a700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:35:38.678 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:38.666+0000 7f691954a700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:35:38.679 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:38.666+0000 7f691954a700 10 monclient(hunting): my global_id is 4824 2023-09-30T01:35:38.679 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:38.666+0000 7f691954a700 1 -- 172.21.15.88:0/3194955150 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6900003570 con 0x7f6914133870 2023-09-30T01:35:38.679 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:38.666+0000 7f691ad4d700 10 monclient(hunting): handle_auth_done global_id 4512 payload 274 2023-09-30T01:35:38.679 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:38.666+0000 7f691ad4d700 1 -- 172.21.15.88:0/3194955150 >> v2:172.21.15.107:3300/0 conn(0x7f69140a35f0 msgr2=0x7f6914133330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:35:38.679 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:38.666+0000 7f691ad4d700 1 --2- 172.21.15.88:0/3194955150 >> v2:172.21.15.107:3300/0 conn(0x7f69140a35f0 0x7f6914133330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:35:38.679 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:38.666+0000 7f691ad4d700 1 -- 172.21.15.88:0/3194955150 >> v1:172.21.15.88:6789/0 conn(0x7f6914133870 legacy=0x7f69140a46f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:35:38.679 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:38.666+0000 7f691ad4d700 10 monclient: _finish_hunting 0 2023-09-30T01:35:38.679 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:38.666+0000 7f691ad4d700 1 monclient: found mon.noname-g 2023-09-30T01:35:38.679 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:38.666+0000 7f691ad4d700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:35:38.679 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:38.666+0000 7f691ad4d700 10 monclient: _send_mon_message to mon.noname-g at v2:172.21.15.107:3301/0 2023-09-30T01:35:38.680 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:38.666+0000 7f691ad4d700 1 -- 172.21.15.88:0/3194955150 --> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69140a39c0 con 0x7f6914133c80 2023-09-30T01:35:38.680 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:38.670+0000 7f691ad4d700 1 --2- 172.21.15.88:0/3194955150 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7f6914133c80 0x7f69140a4e00 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f690c009e60 tx=0x7f690c009780 comp rx=0 tx=0).ready entity=mon.3 client_cookie=eb896954e6a2b54f server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:35:38.680 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:38.670+0000 7f691954a700 1 -- 172.21.15.88:0/3194955150 <== mon.3 v2:172.21.15.107:3301/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f690c00e070 con 0x7f6914133c80 2023-09-30T01:35:38.680 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:38.670+0000 7f691954a700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:35:38.680 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:38.670+0000 7f691954a700 10 monclient: got monmap 1 from mon.noname-g (according to old e1) 2023-09-30T01:35:38.680 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:38.670+0000 7f691954a700 10 monclient: dump: 2023-09-30T01:35:38.680 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:35:38.680 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:35:38.680 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:35:38.680 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:35:38.680 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:35:38.680 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:35:38.681 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:35:38.681 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:35:38.681 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:35:38.681 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:35:38.681 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:35:38.681 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:35:38.681 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:35:38.681 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:35:38.681 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:35:38.682 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:35:38.682 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:38.670+0000 7f691954a700 10 monclient: _finish_auth 0 2023-09-30T01:35:38.682 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:38.670+0000 7f691954a700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:35:38.682 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:38.670+0000 7f691c7b0700 5 monclient: authenticate success, global_id 4512 2023-09-30T01:35:38.682 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:38.670+0000 7f691c7b0700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:35:38.682 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:38.670+0000 7f691954a700 1 -- 172.21.15.88:0/3194955150 <== mon.3 v2:172.21.15.107:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f690c00bab0 con 0x7f6914133c80 2023-09-30T01:35:38.682 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:38.670+0000 7f691954a700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:35:38.683 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:38.670+0000 7f691c7b0700 10 monclient: get_monmap_and_config success 2023-09-30T01:35:38.683 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:38.670+0000 7f691c7b0700 4 set_mon_vals no callback set 2023-09-30T01:35:38.683 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:38.670+0000 7f691954a700 1 -- 172.21.15.88:0/3194955150 <== mon.3 v2:172.21.15.107:3301/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f690c00bc20 con 0x7f6914133c80 2023-09-30T01:35:38.683 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:38.670+0000 7f691954a700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:35:38.683 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:38.670+0000 7f691954a700 10 monclient: got monmap 1 from mon.g (according to old e1) 2023-09-30T01:35:38.683 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:38.670+0000 7f691954a700 10 monclient: dump: 2023-09-30T01:35:38.683 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:35:38.683 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:35:38.683 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:35:38.683 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:35:38.684 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:35:38.684 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:35:38.684 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:35:38.684 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:35:38.684 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:35:38.684 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:35:38.684 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:35:38.685 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:35:38.685 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:35:38.685 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:35:38.685 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:35:38.685 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:35:38.685 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:38.670+0000 7f691c7b0700 10 monclient: shutdown 2023-09-30T01:35:38.685 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:38.670+0000 7f691c7b0700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:35:38.686 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:38.670+0000 7f691c7b0700 1 -- 172.21.15.88:0/3194955150 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7f6914133c80 msgr2=0x7f69140a4e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:35:38.686 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:38.670+0000 7f691c7b0700 1 --2- 172.21.15.88:0/3194955150 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7f6914133c80 0x7f69140a4e00 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f690c009e60 tx=0x7f690c009780 comp rx=0 tx=0).stop 2023-09-30T01:35:38.686 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:38.670+0000 7f691c7b0700 1 -- 172.21.15.88:0/3194955150 shutdown_connections 2023-09-30T01:35:38.686 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:38.670+0000 7f691c7b0700 1 --2- 172.21.15.88:0/3194955150 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7f6914133c80 0x7f69140a4e00 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:35:38.686 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:38.670+0000 7f691c7b0700 1 --2- 172.21.15.88:0/3194955150 >> v2:172.21.15.107:3300/0 conn(0x7f69140a35f0 0x7f6914133330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:35:38.686 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:38.670+0000 7f691c7b0700 1 -- 172.21.15.88:0/3194955150 >> 172.21.15.88:0/3194955150 conn(0x7f691406e5e0 msgr2=0x7f691406e9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:35:38.686 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:38.670+0000 7f691c7b0700 1 -- 172.21.15.88:0/3194955150 shutdown_connections 2023-09-30T01:35:38.687 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:38.670+0000 7f691c7b0700 1 -- 172.21.15.88:0/3194955150 wait complete. 2023-09-30T01:35:38.687 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:38.670+0000 7f691c7b0700 10 monclient: build_initial_monmap 2023-09-30T01:35:38.687 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:38.670+0000 7f691c7b0700 10 monclient: monmap: 2023-09-30T01:35:38.687 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2023-09-30T01:35:38.687 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:35:38.687 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 0.000000 2023-09-30T01:35:38.687 INFO:teuthology.orchestra.run.smithi088.stderr:created 0.000000 2023-09-30T01:35:38.688 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2023-09-30T01:35:38.688 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 1 2023-09-30T01:35:38.688 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.noname-a 2023-09-30T01:35:38.688 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.noname-f 2023-09-30T01:35:38.688 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.noname-b 2023-09-30T01:35:38.688 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.noname-g 2023-09-30T01:35:38.688 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.noname-c 2023-09-30T01:35:38.688 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.noname-h 2023-09-30T01:35:38.689 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.noname-d 2023-09-30T01:35:38.689 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.noname-i 2023-09-30T01:35:38.689 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.noname-e 2023-09-30T01:35:38.689 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:35:38.697 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:38.670+0000 7f691c7b0700 1 Processor -- start 2023-09-30T01:35:38.697 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:38.674+0000 7f691c7b0700 1 -- start start 2023-09-30T01:35:38.697 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:38.674+0000 7f691c7b0700 10 monclient: init 2023-09-30T01:35:38.697 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:38.674+0000 7f691c7b0700 10 monclient: _reopen_session rank -1 2023-09-30T01:35:38.697 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:38.674+0000 7f691c7b0700 10 monclient: _add_conns ranks=[4,0,5,3,1,8,7,6,2] 2023-09-30T01:35:38.698 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:38.674+0000 7f691c7b0700 1 --2- >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f69140a35f0 0x7f69141306d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:35:38.698 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:38.674+0000 7f691c7b0700 10 monclient(hunting): picked mon.noname-c con 0x7f69140a35f0 addr [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] 2023-09-30T01:35:38.698 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:38.674+0000 7f691c7b0700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6914133870 0x7f691411d1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:35:38.698 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:38.674+0000 7f691c7b0700 10 monclient(hunting): picked mon.noname-a con 0x7f6914133870 addr [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] 2023-09-30T01:35:38.698 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:38.674+0000 7f691c7b0700 1 --2- >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7f6914133c80 0x7f691411d700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:35:38.698 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:38.674+0000 7f691c7b0700 10 monclient(hunting): picked mon.noname-h con 0x7f6914133c80 addr [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] 2023-09-30T01:35:38.698 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:38.674+0000 7f691c7b0700 10 monclient(hunting): start opening mon connection 2023-09-30T01:35:38.698 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:38.674+0000 7f691c7b0700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f691411f520 con 0x7f6914133870 2023-09-30T01:35:38.698 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:38.674+0000 7f691c7b0700 10 monclient(hunting): start opening mon connection 2023-09-30T01:35:38.698 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:38.674+0000 7f691c7b0700 1 -- --> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] -- mon_getmap magic: 0 v1 -- 0x7f69140a0b70 con 0x7f69140a35f0 2023-09-30T01:35:38.698 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:38.674+0000 7f691c7b0700 10 monclient(hunting): start opening mon connection 2023-09-30T01:35:38.699 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:38.674+0000 7f691c7b0700 1 -- --> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] -- mon_getmap magic: 0 v1 -- 0x7f6914097d90 con 0x7f6914133c80 2023-09-30T01:35:38.699 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:38.674+0000 7f691c7b0700 10 monclient(hunting): _renew_subs 2023-09-30T01:35:38.699 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:38.674+0000 7f691c7b0700 10 monclient(hunting): authenticate will time out at 1010.713285s 2023-09-30T01:35:38.699 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:38.674+0000 7f691a54c700 1 --2- >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f69140a35f0 0x7f69141306d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:35:38.699 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:38.674+0000 7f6919d4b700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6914133870 0x7f691411d1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:35:38.699 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:38.674+0000 7f691a54c700 1 --2- >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f69140a35f0 0x7f69141306d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3302/0 says I am v2:172.21.15.88:44542/0 (socket says 172.21.15.88:44542) 2023-09-30T01:35:38.699 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:38.674+0000 7f691a54c700 1 -- 172.21.15.88:0/693596454 learned_addr learned my addr 172.21.15.88:0/693596454 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-09-30T01:35:38.699 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:38.674+0000 7f691a54c700 10 monclient(hunting): get_auth_request con 0x7f69140a35f0 auth_method 0 2023-09-30T01:35:38.699 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:38.674+0000 7f691a54c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:35:38.699 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:38.674+0000 7f691a54c700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:35:38.699 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:38.674+0000 7f691a54c700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:35:38.699 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:38.674+0000 7f6919d4b700 10 monclient(hunting): get_auth_request con 0x7f6914133870 auth_method 0 2023-09-30T01:35:38.700 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:38.674+0000 7f6919d4b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:35:38.700 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:38.674+0000 7f6919d4b700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:35:38.700 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:38.674+0000 7f6919d4b700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:35:38.700 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:38.674+0000 7f691a54c700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:35:38.700 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:38.674+0000 7f691a54c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:35:38.700 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:38.674+0000 7f691ad4d700 1 --2- 172.21.15.88:0/693596454 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7f6914133c80 0x7f691411d700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:35:38.700 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:38.674+0000 7f691a54c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:35:38.700 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:38.674+0000 7f691ad4d700 10 monclient(hunting): get_auth_request con 0x7f6914133c80 auth_method 0 2023-09-30T01:35:38.701 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:38.674+0000 7f691ad4d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:35:38.701 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:38.674+0000 7f691ad4d700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:35:38.701 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:38.674+0000 7f691ad4d700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:35:38.701 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:38.674+0000 7f6919d4b700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:35:38.701 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:38.674+0000 7f6919d4b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:35:38.701 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:38.674+0000 7f6919d4b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:35:38.701 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:38.674+0000 7f691ad4d700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:35:38.702 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:38.674+0000 7f691ad4d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:35:38.702 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:38.674+0000 7f691ad4d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:35:38.702 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:38.674+0000 7f691a54c700 10 monclient(hunting): handle_auth_done global_id 4639 payload 824 2023-09-30T01:35:38.702 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:38.674+0000 7f691a54c700 1 -- 172.21.15.88:0/693596454 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7f6914133c80 msgr2=0x7f691411d700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:35:38.702 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:38.674+0000 7f691a54c700 1 --2- 172.21.15.88:0/693596454 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7f6914133c80 0x7f691411d700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:35:38.702 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:38.674+0000 7f691a54c700 1 -- 172.21.15.88:0/693596454 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6914133870 msgr2=0x7f691411d1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:35:38.703 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:38.674+0000 7f691a54c700 1 --2- 172.21.15.88:0/693596454 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6914133870 0x7f691411d1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:35:38.703 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:38.674+0000 7f691a54c700 10 monclient: _finish_hunting 0 2023-09-30T01:35:38.703 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:38.674+0000 7f691a54c700 1 monclient: found mon.noname-c 2023-09-30T01:35:38.703 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:38.674+0000 7f691a54c700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:35:38.703 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:38.674+0000 7f691a54c700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.88:3302/0 2023-09-30T01:35:38.703 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:38.674+0000 7f691a54c700 1 -- 172.21.15.88:0/693596454 --> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f690c009a20 con 0x7f69140a35f0 2023-09-30T01:35:38.704 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:38.674+0000 7f691a54c700 1 --2- 172.21.15.88:0/693596454 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f69140a35f0 0x7f69141306d0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f691000e930 tx=0x7f69100091a0 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:35:38.704 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:38.674+0000 7f690a7fc700 1 -- 172.21.15.88:0/693596454 <== mon.4 v2:172.21.15.88:3302/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f6910013b00 con 0x7f69140a35f0 2023-09-30T01:35:38.704 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:38.674+0000 7f690a7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:35:38.704 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:38.674+0000 7f690a7fc700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T01:35:38.704 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:38.674+0000 7f690a7fc700 10 monclient: dump: 2023-09-30T01:35:38.704 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:35:38.704 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:35:38.705 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:35:38.705 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:35:38.705 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:35:38.705 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:35:38.705 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:35:38.705 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:35:38.705 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:35:38.706 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:35:38.706 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:35:38.706 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:35:38.706 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:35:38.706 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:35:38.706 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:35:38.706 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:35:38.707 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:38.674+0000 7f690a7fc700 10 monclient: _finish_auth 0 2023-09-30T01:35:38.707 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:38.674+0000 7f690a7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:35:38.707 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:38.674+0000 7f691c7b0700 5 monclient: authenticate success, global_id 4639 2023-09-30T01:35:38.707 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:38.674+0000 7f690a7fc700 1 -- 172.21.15.88:0/693596454 <== mon.4 v2:172.21.15.88:3302/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6910013ca0 con 0x7f69140a35f0 2023-09-30T01:35:38.707 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:38.674+0000 7f690a7fc700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:35:38.707 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:38.674+0000 7f691c7b0700 10 monclient: _renew_subs 2023-09-30T01:35:38.708 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:38.674+0000 7f690a7fc700 1 -- 172.21.15.88:0/693596454 <== mon.4 v2:172.21.15.88:3302/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f6910013e10 con 0x7f69140a35f0 2023-09-30T01:35:38.708 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:38.674+0000 7f691c7b0700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.88:3302/0 2023-09-30T01:35:38.708 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:38.674+0000 7f691c7b0700 1 -- 172.21.15.88:0/693596454 --> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f691411f920 con 0x7f69140a35f0 2023-09-30T01:35:38.708 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:38.674+0000 7f6918d49700 4 set_mon_vals no callback set 2023-09-30T01:35:38.708 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:38.674+0000 7f690a7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:35:38.708 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:38.674+0000 7f690a7fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T01:35:38.709 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:38.674+0000 7f690a7fc700 10 monclient: dump: 2023-09-30T01:35:38.709 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:35:38.709 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:35:38.709 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:35:38.709 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:35:38.709 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:35:38.710 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:35:38.710 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:35:38.710 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:35:38.710 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:35:38.710 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:35:38.710 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:35:38.710 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:35:38.710 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:35:38.710 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:35:38.710 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:35:38.710 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:35:38.711 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:38.674+0000 7f691c7b0700 10 monclient: _renew_subs 2023-09-30T01:35:38.711 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:38.674+0000 7f691c7b0700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.88:3302/0 2023-09-30T01:35:38.711 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:38.674+0000 7f691c7b0700 1 -- 172.21.15.88:0/693596454 --> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f691411ec80 con 0x7f69140a35f0 2023-09-30T01:35:38.711 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:38.678+0000 7f690a7fc700 1 -- 172.21.15.88:0/693596454 <== mon.4 v2:172.21.15.88:3302/0 4 ==== mgrmap(e 5) v1 ==== 40144+0+0 (secure 0 0 0) 0x7f6910013680 con 0x7f69140a35f0 2023-09-30T01:35:38.711 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:38.678+0000 7f691c7b0700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:35:38.711 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:38.678+0000 7f691c7b0700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:35:38.711 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:38.678+0000 7f691c7b0700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.88:3302/0 2023-09-30T01:35:38.711 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:38.678+0000 7f690a7fc700 1 --2- 172.21.15.88:0/693596454 >> v2:172.21.15.107:6812/29926 conn(0x7f68f8033360 0x7f68f8035810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:35:38.711 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:38.678+0000 7f691c7b0700 1 -- 172.21.15.88:0/693596454 --> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f69140710d0 con 0x7f69140a35f0 2023-09-30T01:35:38.711 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:38.678+0000 7f690a7fc700 1 -- 172.21.15.88:0/693596454 <== mon.4 v2:172.21.15.88:3302/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 3502+0+0 (secure 0 0 0) 0x7f69100436b0 con 0x7f69140a35f0 2023-09-30T01:35:38.711 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:38.678+0000 7f6919d4b700 1 --2- 172.21.15.88:0/693596454 >> v2:172.21.15.107:6812/29926 conn(0x7f68f8033360 0x7f68f8035810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:35:38.711 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:38.678+0000 7f6919d4b700 10 monclient: get_auth_request con 0x7f68f8033360 auth_method 0 2023-09-30T01:35:38.711 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:38.678+0000 7f6919d4b700 1 --2- 172.21.15.88:0/693596454 >> v2:172.21.15.107:6812/29926 conn(0x7f68f8033360 0x7f68f8035810 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f6904007fd0 tx=0x7f6904009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:35:38.712 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:38.686+0000 7f690a7fc700 1 -- 172.21.15.88:0/693596454 <== mon.4 v2:172.21.15.88:3302/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f691001c450 con 0x7f69140a35f0 2023-09-30T01:35:38.712 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:38.686+0000 7f690a7fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:35:38.712 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:38.686+0000 7f690a7fc700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:35:38.880 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:38.878+0000 7f691c7b0700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T01:35:38.880 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:38.878+0000 7f691c7b0700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:35:38.880 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:38.878+0000 7f691c7b0700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.88:3302/0 2023-09-30T01:35:38.880 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:38.878+0000 7f691c7b0700 1 -- 172.21.15.88:0/693596454 --> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f691411f920 con 0x7f69140a35f0 2023-09-30T01:35:38.880 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:38.878+0000 7f690a7fc700 1 -- 172.21.15.88:0/693596454 <== mon.4 v2:172.21.15.88:3302/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f691001a5e0 con 0x7f69140a35f0 2023-09-30T01:35:38.881 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:38.878+0000 7f690a7fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:35:38.881 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:38.878+0000 7f690a7fc700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:35:38.881 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-09-30T01:35:38.882 INFO:teuthology.orchestra.run.smithi088.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:35:38.885 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:38.882+0000 7f691c7b0700 1 -- 172.21.15.88:0/693596454 >> v2:172.21.15.107:6812/29926 conn(0x7f68f8033360 msgr2=0x7f68f8035810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:35:38.885 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:38.882+0000 7f691c7b0700 1 --2- 172.21.15.88:0/693596454 >> v2:172.21.15.107:6812/29926 conn(0x7f68f8033360 0x7f68f8035810 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f6904007fd0 tx=0x7f6904009040 comp rx=0 tx=0).stop 2023-09-30T01:35:38.885 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:38.882+0000 7f691c7b0700 10 monclient: shutdown 2023-09-30T01:35:38.885 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:38.882+0000 7f691c7b0700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:35:38.885 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:38.882+0000 7f691c7b0700 1 -- 172.21.15.88:0/693596454 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f69140a35f0 msgr2=0x7f69141306d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:35:38.885 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:38.882+0000 7f691c7b0700 1 --2- 172.21.15.88:0/693596454 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f69140a35f0 0x7f69141306d0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f691000e930 tx=0x7f69100091a0 comp rx=0 tx=0).stop 2023-09-30T01:35:38.885 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:38.882+0000 7f691c7b0700 1 -- 172.21.15.88:0/693596454 shutdown_connections 2023-09-30T01:35:38.885 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:38.882+0000 7f691c7b0700 1 --2- 172.21.15.88:0/693596454 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7f6914133c80 0x7f691411d700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:35:38.885 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:38.882+0000 7f691c7b0700 1 --2- 172.21.15.88:0/693596454 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6914133870 0x7f691411d1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:35:38.886 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:38.882+0000 7f691c7b0700 1 --2- 172.21.15.88:0/693596454 >> v2:172.21.15.107:6812/29926 conn(0x7f68f8033360 0x7f68f8035810 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:35:38.886 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:38.882+0000 7f691c7b0700 1 --2- 172.21.15.88:0/693596454 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f69140a35f0 0x7f69141306d0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:35:38.886 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:38.882+0000 7f691c7b0700 1 -- 172.21.15.88:0/693596454 >> 172.21.15.88:0/693596454 conn(0x7f691406e5e0 msgr2=0x7f6914131860 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:35:38.886 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:38.882+0000 7f691c7b0700 1 -- 172.21.15.88:0/693596454 shutdown_connections 2023-09-30T01:35:38.886 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:38.882+0000 7f691c7b0700 1 -- 172.21.15.88:0/693596454 wait complete. 2023-09-30T01:35:41.900 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:35:42.013 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:42.010+0000 7f5a393a3700 10 monclient: get_monmap_and_config 2023-09-30T01:35:42.013 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:42.010+0000 7f5a393a3700 10 monclient: build_initial_monmap 2023-09-30T01:35:42.016 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:42.010+0000 7f5a393a3700 10 monclient: monmap: 2023-09-30T01:35:42.016 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2023-09-30T01:35:42.016 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:35:42.016 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:35:42.013039+0000 2023-09-30T01:35:42.016 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:35:42.013039+0000 2023-09-30T01:35:42.016 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2023-09-30T01:35:42.017 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:35:42.017 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a-legacy 2023-09-30T01:35:42.017 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.107:6789/0 mon.noname-f-legacy 2023-09-30T01:35:42.017 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.noname-b 2023-09-30T01:35:42.017 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.noname-g 2023-09-30T01:35:42.017 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.noname-c 2023-09-30T01:35:42.017 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.noname-h 2023-09-30T01:35:42.017 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.noname-d 2023-09-30T01:35:42.017 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.noname-i 2023-09-30T01:35:42.017 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.noname-e 2023-09-30T01:35:42.017 INFO:teuthology.orchestra.run.smithi088.stderr:9: v2:172.21.15.88:3300/0 mon.noname-a 2023-09-30T01:35:42.017 INFO:teuthology.orchestra.run.smithi088.stderr:10: v2:172.21.15.107:3300/0 mon.noname-f 2023-09-30T01:35:42.017 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:35:42.018 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:42.010+0000 7f5a393a3700 1 Processor -- start 2023-09-30T01:35:42.018 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:42.010+0000 7f5a393a3700 1 -- start start 2023-09-30T01:35:42.018 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:42.010+0000 7f5a393a3700 10 monclient: init 2023-09-30T01:35:42.018 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:42.010+0000 7f5a393a3700 10 monclient: _reopen_session rank -1 2023-09-30T01:35:42.018 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:42.010+0000 7f5a393a3700 10 monclient: _add_conns ranks=[6,1,9,2,8,5,7,10,0,4,3] 2023-09-30T01:35:42.018 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:42.010+0000 7f5a393a3700 1 --2- >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7f5a341376f0 0x7f5a3406a130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:35:42.018 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:42.010+0000 7f5a393a3700 10 monclient(hunting): picked mon.noname-d con 0x7f5a341376f0 addr [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] 2023-09-30T01:35:42.018 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:42.010+0000 7f5a393a3700 10 monclient(hunting): picked mon.noname-f-legacy con 0x7f5a34137ac0 addr v1:172.21.15.107:6789/0 2023-09-30T01:35:42.018 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:42.010+0000 7f5a393a3700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f5a3413cc30 0x7f5a3413f010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:35:42.018 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:42.010+0000 7f5a393a3700 10 monclient(hunting): picked mon.noname-a con 0x7f5a3413cc30 addr v2:172.21.15.88:3300/0 2023-09-30T01:35:42.018 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:42.010+0000 7f5a393a3700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a3404bb20 con 0x7f5a34137ac0 2023-09-30T01:35:42.018 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:42.010+0000 7f5a393a3700 10 monclient(hunting): start opening mon connection 2023-09-30T01:35:42.019 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:42.010+0000 7f5a393a3700 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5a341385a0 con 0x7f5a3413cc30 2023-09-30T01:35:42.019 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:42.010+0000 7f5a393a3700 10 monclient(hunting): start opening mon connection 2023-09-30T01:35:42.019 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:42.010+0000 7f5a393a3700 1 -- --> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] -- mon_getmap magic: 0 v1 -- 0x7f5a3406c960 con 0x7f5a341376f0 2023-09-30T01:35:42.019 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:42.010+0000 7f5a393a3700 10 monclient(hunting): _renew_subs 2023-09-30T01:35:42.019 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:42.010+0000 7f5a393a3700 10 monclient(hunting): authenticate will time out at 1014.050534s 2023-09-30T01:35:42.019 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:42.010+0000 7f5a38ba2700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f5a3413cc30 0x7f5a3413f010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:35:42.019 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:42.010+0000 7f5a33fff700 1 --2- >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7f5a341376f0 0x7f5a3406a130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:35:42.019 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:42.010+0000 7f5a38ba2700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f5a3413cc30 0x7f5a3413f010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:36460/0 (socket says 172.21.15.88:36460) 2023-09-30T01:35:42.019 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:42.010+0000 7f5a38ba2700 1 -- 172.21.15.88:0/2854869776 learned_addr learned my addr 172.21.15.88:0/2854869776 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-09-30T01:35:42.019 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:42.010+0000 7f5a38ba2700 10 monclient(hunting): get_auth_request con 0x7f5a3413cc30 auth_method 0 2023-09-30T01:35:42.019 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:42.010+0000 7f5a38ba2700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:35:42.019 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:42.010+0000 7f5a38ba2700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:35:42.019 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:42.010+0000 7f5a38ba2700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:35:42.020 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:42.010+0000 7f5a33fff700 10 monclient(hunting): get_auth_request con 0x7f5a341376f0 auth_method 0 2023-09-30T01:35:42.020 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:42.010+0000 7f5a33fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:35:42.020 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:42.010+0000 7f5a33fff700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:35:42.020 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:42.010+0000 7f5a33fff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:35:42.020 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:42.010+0000 7f5a38ba2700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:35:42.020 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:42.010+0000 7f5a38ba2700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:35:42.020 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:42.010+0000 7f5a38ba2700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:35:42.020 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:42.010+0000 7f5a33fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:35:42.020 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:42.010+0000 7f5a33fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:35:42.020 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:42.010+0000 7f5a33fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:35:42.020 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:42.010+0000 7f5a38ba2700 10 monclient(hunting): handle_auth_done global_id 4842 payload 274 2023-09-30T01:35:42.020 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:42.010+0000 7f5a38ba2700 1 -- 172.21.15.88:0/2854869776 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7f5a341376f0 msgr2=0x7f5a3406a130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:35:42.021 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:42.010+0000 7f5a38ba2700 1 --2- 172.21.15.88:0/2854869776 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7f5a341376f0 0x7f5a3406a130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:35:42.021 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:42.010+0000 7f5a38ba2700 1 -- 172.21.15.88:0/2854869776 >> v1:172.21.15.107:6789/0 conn(0x7f5a34137ac0 legacy=0x7f5a34137e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:35:42.021 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:42.010+0000 7f5a38ba2700 10 monclient: _finish_hunting 0 2023-09-30T01:35:42.021 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:42.010+0000 7f5a38ba2700 1 monclient: found mon.noname-a 2023-09-30T01:35:42.021 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:42.010+0000 7f5a38ba2700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:35:42.021 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:42.010+0000 7f5a38ba2700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.88:3300/0 2023-09-30T01:35:42.021 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:42.010+0000 7f5a38ba2700 1 -- 172.21.15.88:0/2854869776 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a3406dd00 con 0x7f5a3413cc30 2023-09-30T01:35:42.021 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:42.010+0000 7f5a38ba2700 1 --2- 172.21.15.88:0/2854869776 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5a3413cc30 0x7f5a3413f010 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f5a2c0099b0 tx=0x7f5a2c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ced2d9a7ae57058e server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:35:42.021 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:42.014+0000 7f5a32ffd700 1 -- 172.21.15.88:0/2854869776 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f5a2c009ea0 con 0x7f5a3413cc30 2023-09-30T01:35:42.021 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:42.014+0000 7f5a32ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:35:42.021 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:42.014+0000 7f5a32ffd700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T01:35:42.021 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:42.014+0000 7f5a32ffd700 10 monclient: dump: 2023-09-30T01:35:42.021 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:35:42.022 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:35:42.022 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:35:42.022 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:35:42.022 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:35:42.022 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:35:42.022 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:35:42.022 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:35:42.022 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:35:42.022 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:35:42.022 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:35:42.022 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:35:42.022 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:35:42.022 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:35:42.023 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:35:42.023 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:35:42.023 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:42.014+0000 7f5a32ffd700 10 monclient: _finish_auth 0 2023-09-30T01:35:42.023 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:42.014+0000 7f5a32ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:35:42.023 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:42.014+0000 7f5a393a3700 5 monclient: authenticate success, global_id 4842 2023-09-30T01:35:42.023 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:42.014+0000 7f5a393a3700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:35:42.023 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:42.014+0000 7f5a32ffd700 1 -- 172.21.15.88:0/2854869776 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a2c00b880 con 0x7f5a3413cc30 2023-09-30T01:35:42.023 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:42.014+0000 7f5a32ffd700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:35:42.023 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:42.014+0000 7f5a393a3700 10 monclient: get_monmap_and_config success 2023-09-30T01:35:42.023 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:42.014+0000 7f5a393a3700 4 set_mon_vals no callback set 2023-09-30T01:35:42.023 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:42.014+0000 7f5a32ffd700 1 -- 172.21.15.88:0/2854869776 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f5a2c00be70 con 0x7f5a3413cc30 2023-09-30T01:35:42.024 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:42.014+0000 7f5a32ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:35:42.025 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:42.014+0000 7f5a32ffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:35:42.025 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:42.014+0000 7f5a32ffd700 10 monclient: dump: 2023-09-30T01:35:42.025 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:35:42.025 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:35:42.025 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:35:42.025 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:35:42.026 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:35:42.026 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:35:42.026 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:35:42.026 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:35:42.026 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:35:42.027 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:35:42.027 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:35:42.027 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:35:42.027 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:35:42.027 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:35:42.027 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:35:42.028 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:35:42.036 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:42.014+0000 7f5a393a3700 10 monclient: shutdown 2023-09-30T01:35:42.036 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:42.014+0000 7f5a393a3700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:35:42.037 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:42.014+0000 7f5a393a3700 1 -- 172.21.15.88:0/2854869776 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5a3413cc30 msgr2=0x7f5a3413f010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:35:42.037 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:42.014+0000 7f5a393a3700 1 --2- 172.21.15.88:0/2854869776 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5a3413cc30 0x7f5a3413f010 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f5a2c0099b0 tx=0x7f5a2c009250 comp rx=0 tx=0).stop 2023-09-30T01:35:42.037 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:42.014+0000 7f5a393a3700 1 -- 172.21.15.88:0/2854869776 shutdown_connections 2023-09-30T01:35:42.037 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:42.014+0000 7f5a393a3700 1 --2- 172.21.15.88:0/2854869776 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5a3413cc30 0x7f5a3413f010 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:35:42.037 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:42.014+0000 7f5a393a3700 1 --2- 172.21.15.88:0/2854869776 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7f5a341376f0 0x7f5a3406a130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:35:42.037 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:42.014+0000 7f5a393a3700 1 -- 172.21.15.88:0/2854869776 >> 172.21.15.88:0/2854869776 conn(0x7f5a340a66f0 msgr2=0x7f5a340a6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:35:42.037 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:42.014+0000 7f5a393a3700 1 -- 172.21.15.88:0/2854869776 shutdown_connections 2023-09-30T01:35:42.037 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:42.014+0000 7f5a393a3700 1 -- 172.21.15.88:0/2854869776 wait complete. 2023-09-30T01:35:42.037 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:42.014+0000 7f5a393a3700 10 monclient: build_initial_monmap 2023-09-30T01:35:42.038 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:42.014+0000 7f5a393a3700 10 monclient: monmap: 2023-09-30T01:35:42.038 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2023-09-30T01:35:42.038 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:35:42.038 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 0.000000 2023-09-30T01:35:42.038 INFO:teuthology.orchestra.run.smithi088.stderr:created 0.000000 2023-09-30T01:35:42.038 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2023-09-30T01:35:42.038 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 1 2023-09-30T01:35:42.038 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.noname-a 2023-09-30T01:35:42.038 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.noname-f 2023-09-30T01:35:42.038 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.noname-b 2023-09-30T01:35:42.038 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.noname-g 2023-09-30T01:35:42.039 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.noname-c 2023-09-30T01:35:42.039 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.noname-h 2023-09-30T01:35:42.039 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.noname-d 2023-09-30T01:35:42.039 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.noname-i 2023-09-30T01:35:42.039 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.noname-e 2023-09-30T01:35:42.039 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:35:42.039 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:42.014+0000 7f5a393a3700 1 Processor -- start 2023-09-30T01:35:42.039 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:42.018+0000 7f5a393a3700 1 -- start start 2023-09-30T01:35:42.039 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:42.018+0000 7f5a393a3700 10 monclient: init 2023-09-30T01:35:42.039 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:42.018+0000 7f5a393a3700 10 monclient: _reopen_session rank -1 2023-09-30T01:35:42.039 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:42.018+0000 7f5a393a3700 10 monclient: _add_conns ranks=[2,3,6,1,7,8,5,4,0] 2023-09-30T01:35:42.039 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:42.018+0000 7f5a393a3700 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f5a341376f0 0x7f5a34130eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:35:42.040 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:42.018+0000 7f5a393a3700 10 monclient(hunting): picked mon.noname-b con 0x7f5a341376f0 addr [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] 2023-09-30T01:35:42.040 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:42.018+0000 7f5a393a3700 1 --2- >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7f5a34137ac0 0x7f5a34139530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:35:42.040 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:42.018+0000 7f5a393a3700 10 monclient(hunting): picked mon.noname-g con 0x7f5a34137ac0 addr [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] 2023-09-30T01:35:42.040 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:42.018+0000 7f5a393a3700 1 --2- >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7f5a3413cc30 0x7f5a34139a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:35:42.040 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:42.018+0000 7f5a393a3700 10 monclient(hunting): picked mon.noname-d con 0x7f5a3413cc30 addr [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] 2023-09-30T01:35:42.040 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:42.018+0000 7f5a393a3700 10 monclient(hunting): start opening mon connection 2023-09-30T01:35:42.040 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:42.018+0000 7f5a393a3700 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a3406cf80 con 0x7f5a341376f0 2023-09-30T01:35:42.040 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:42.018+0000 7f5a393a3700 10 monclient(hunting): start opening mon connection 2023-09-30T01:35:42.040 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:42.018+0000 7f5a393a3700 1 -- --> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a3413c4b0 con 0x7f5a34137ac0 2023-09-30T01:35:42.040 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:42.018+0000 7f5a393a3700 10 monclient(hunting): start opening mon connection 2023-09-30T01:35:42.040 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:42.018+0000 7f5a393a3700 1 -- --> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] -- mon_getmap magic: 0 v1 -- 0x7f5a34125600 con 0x7f5a3413cc30 2023-09-30T01:35:42.040 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:42.018+0000 7f5a393a3700 10 monclient(hunting): _renew_subs 2023-09-30T01:35:42.041 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:42.018+0000 7f5a393a3700 10 monclient(hunting): authenticate will time out at 1014.057443s 2023-09-30T01:35:42.041 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:42.018+0000 7f5a33fff700 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f5a341376f0 0x7f5a34130eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:35:42.041 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:42.018+0000 7f5a33fff700 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f5a341376f0 0x7f5a34130eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3301/0 says I am v2:172.21.15.88:60866/0 (socket says 172.21.15.88:60866) 2023-09-30T01:35:42.041 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:42.018+0000 7f5a33fff700 1 -- 172.21.15.88:0/3500624022 learned_addr learned my addr 172.21.15.88:0/3500624022 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-09-30T01:35:42.041 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:42.018+0000 7f5a33fff700 10 monclient(hunting): get_auth_request con 0x7f5a341376f0 auth_method 0 2023-09-30T01:35:42.041 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:42.018+0000 7f5a33fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:35:42.041 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:42.018+0000 7f5a33fff700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:35:42.041 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:42.018+0000 7f5a33fff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:35:42.041 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:42.018+0000 7f5a38ba2700 1 --2- 172.21.15.88:0/3500624022 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7f5a3413cc30 0x7f5a34139a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:35:42.041 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:42.018+0000 7f5a38ba2700 10 monclient(hunting): get_auth_request con 0x7f5a3413cc30 auth_method 0 2023-09-30T01:35:42.041 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:42.018+0000 7f5a38ba2700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:35:42.041 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:42.018+0000 7f5a38ba2700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:35:42.042 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:42.018+0000 7f5a38ba2700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:35:42.042 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:42.018+0000 7f5a337fe700 1 --2- 172.21.15.88:0/3500624022 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7f5a34137ac0 0x7f5a34139530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:35:42.042 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:42.018+0000 7f5a337fe700 10 monclient(hunting): get_auth_request con 0x7f5a34137ac0 auth_method 0 2023-09-30T01:35:42.042 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:42.018+0000 7f5a337fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:35:42.042 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:42.018+0000 7f5a337fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:35:42.042 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:42.018+0000 7f5a337fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:35:42.042 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:42.018+0000 7f5a38ba2700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:35:42.042 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:42.018+0000 7f5a38ba2700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:35:42.042 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:42.018+0000 7f5a38ba2700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:35:42.042 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:42.018+0000 7f5a33fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:35:42.042 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:42.018+0000 7f5a33fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:35:42.043 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:42.018+0000 7f5a33fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:35:42.043 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:42.018+0000 7f5a337fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:35:42.043 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:42.018+0000 7f5a337fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:35:42.043 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:42.018+0000 7f5a337fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:35:42.043 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:42.018+0000 7f5a38ba2700 10 monclient(hunting): handle_auth_done global_id 4524 payload 824 2023-09-30T01:35:42.043 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:42.018+0000 7f5a38ba2700 1 -- 172.21.15.88:0/3500624022 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7f5a34137ac0 msgr2=0x7f5a34139530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:35:42.043 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:42.018+0000 7f5a38ba2700 1 --2- 172.21.15.88:0/3500624022 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7f5a34137ac0 0x7f5a34139530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:35:42.043 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:42.018+0000 7f5a38ba2700 1 -- 172.21.15.88:0/3500624022 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f5a341376f0 msgr2=0x7f5a34130eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:35:42.043 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:42.018+0000 7f5a38ba2700 1 --2- 172.21.15.88:0/3500624022 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f5a341376f0 0x7f5a34130eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:35:42.043 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:42.018+0000 7f5a38ba2700 10 monclient: _finish_hunting 0 2023-09-30T01:35:42.043 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:42.018+0000 7f5a38ba2700 1 monclient: found mon.noname-d 2023-09-30T01:35:42.044 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:42.018+0000 7f5a38ba2700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:35:42.044 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:42.018+0000 7f5a38ba2700 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.88:3303/0 2023-09-30T01:35:42.044 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:42.018+0000 7f5a38ba2700 1 -- 172.21.15.88:0/3500624022 --> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a2c0094f0 con 0x7f5a3413cc30 2023-09-30T01:35:42.044 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:42.018+0000 7f5a38ba2700 1 --2- 172.21.15.88:0/3500624022 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7f5a3413cc30 0x7f5a34139a70 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f5a3406e9b0 tx=0x7f5a2c017000 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:35:42.044 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:42.018+0000 7f5a1bfff700 1 -- 172.21.15.88:0/3500624022 <== mon.6 v2:172.21.15.88:3303/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f5a2c009ea0 con 0x7f5a3413cc30 2023-09-30T01:35:42.044 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:42.018+0000 7f5a1bfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:35:42.044 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:42.018+0000 7f5a1bfff700 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2023-09-30T01:35:42.044 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:42.018+0000 7f5a1bfff700 10 monclient: dump: 2023-09-30T01:35:42.044 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:35:42.044 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:35:42.044 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:35:42.044 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:35:42.045 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:35:42.045 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:35:42.045 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:35:42.045 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:35:42.045 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:35:42.045 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:35:42.045 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:35:42.045 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:35:42.045 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:35:42.045 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:35:42.045 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:35:42.046 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:35:42.046 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:42.018+0000 7f5a1bfff700 10 monclient: _finish_auth 0 2023-09-30T01:35:42.046 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:42.018+0000 7f5a1bfff700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:35:42.046 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:42.018+0000 7f5a1bfff700 1 -- 172.21.15.88:0/3500624022 <== mon.6 v2:172.21.15.88:3303/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a2c01cc70 con 0x7f5a3413cc30 2023-09-30T01:35:42.046 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:42.018+0000 7f5a393a3700 5 monclient: authenticate success, global_id 4524 2023-09-30T01:35:42.046 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:42.018+0000 7f5a393a3700 10 monclient: _renew_subs 2023-09-30T01:35:42.046 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:42.018+0000 7f5a393a3700 10 monclient: _send_mon_message to mon.d at v2:172.21.15.88:3303/0 2023-09-30T01:35:42.046 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:42.018+0000 7f5a393a3700 1 -- 172.21.15.88:0/3500624022 --> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a34125d00 con 0x7f5a3413cc30 2023-09-30T01:35:42.046 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:42.018+0000 7f5a1bfff700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:35:42.046 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:42.018+0000 7f5a1bfff700 1 -- 172.21.15.88:0/3500624022 <== mon.6 v2:172.21.15.88:3303/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f5a2c01ce20 con 0x7f5a3413cc30 2023-09-30T01:35:42.046 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:42.018+0000 7f5a1bfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:35:42.047 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:42.018+0000 7f5a1bfff700 10 monclient: got monmap 1 from mon.d (according to old e1) 2023-09-30T01:35:42.047 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:42.018+0000 7f5a1bfff700 10 monclient: dump: 2023-09-30T01:35:42.047 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:35:42.047 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:35:42.047 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:35:42.047 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:35:42.047 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:35:42.047 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:35:42.047 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:35:42.047 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:35:42.047 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:35:42.048 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:35:42.048 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:35:42.048 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:35:42.048 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:35:42.048 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:35:42.048 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:35:42.048 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:35:42.048 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:42.018+0000 7f5a393a3700 10 monclient: _renew_subs 2023-09-30T01:35:42.048 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:42.018+0000 7f5a393a3700 10 monclient: _send_mon_message to mon.d at v2:172.21.15.88:3303/0 2023-09-30T01:35:42.048 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:42.018+0000 7f5a393a3700 1 -- 172.21.15.88:0/3500624022 --> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a3413a6a0 con 0x7f5a3413cc30 2023-09-30T01:35:42.048 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:42.018+0000 7f5a327fc700 4 set_mon_vals no callback set 2023-09-30T01:35:42.049 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:42.022+0000 7f5a1bfff700 1 -- 172.21.15.88:0/3500624022 <== mon.6 v2:172.21.15.88:3303/0 4 ==== mgrmap(e 5) v1 ==== 40144+0+0 (secure 0 0 0) 0x7f5a2c02f450 con 0x7f5a3413cc30 2023-09-30T01:35:42.049 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:42.022+0000 7f5a393a3700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:35:42.049 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:42.022+0000 7f5a393a3700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:35:42.049 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:42.022+0000 7f5a393a3700 10 monclient: _send_mon_message to mon.d at v2:172.21.15.88:3303/0 2023-09-30T01:35:42.049 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:42.022+0000 7f5a1bfff700 1 --2- 172.21.15.88:0/3500624022 >> v2:172.21.15.107:6812/29926 conn(0x7f5a20033360 0x7f5a20035810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:35:42.049 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:42.022+0000 7f5a393a3700 1 -- 172.21.15.88:0/3500624022 --> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5a140050b0 con 0x7f5a3413cc30 2023-09-30T01:35:42.049 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:42.022+0000 7f5a1bfff700 1 -- 172.21.15.88:0/3500624022 <== mon.6 v2:172.21.15.88:3303/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 3502+0+0 (secure 0 0 0) 0x7f5a2c044da0 con 0x7f5a3413cc30 2023-09-30T01:35:42.049 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:42.022+0000 7f5a33fff700 1 --2- 172.21.15.88:0/3500624022 >> v2:172.21.15.107:6812/29926 conn(0x7f5a20033360 0x7f5a20035810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:35:42.049 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:42.022+0000 7f5a33fff700 10 monclient: get_auth_request con 0x7f5a20033360 auth_method 0 2023-09-30T01:35:42.049 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:42.022+0000 7f5a33fff700 1 --2- 172.21.15.88:0/3500624022 >> v2:172.21.15.107:6812/29926 conn(0x7f5a20033360 0x7f5a20035810 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f5a28009dd0 tx=0x7f5a280091a0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:35:42.049 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:42.026+0000 7f5a1bfff700 1 -- 172.21.15.88:0/3500624022 <== mon.6 v2:172.21.15.88:3303/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f5a2c02f700 con 0x7f5a3413cc30 2023-09-30T01:35:42.049 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:42.026+0000 7f5a1bfff700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:35:42.049 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:42.026+0000 7f5a1bfff700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:35:42.229 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:42.226+0000 7f5a393a3700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T01:35:42.229 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:42.226+0000 7f5a393a3700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:35:42.229 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:42.226+0000 7f5a393a3700 10 monclient: _send_mon_message to mon.d at v2:172.21.15.88:3303/0 2023-09-30T01:35:42.229 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:42.226+0000 7f5a393a3700 1 -- 172.21.15.88:0/3500624022 --> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f5a14005630 con 0x7f5a3413cc30 2023-09-30T01:35:42.230 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:42.226+0000 7f5a1bfff700 1 -- 172.21.15.88:0/3500624022 <== mon.6 v2:172.21.15.88:3303/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f5a2c015030 con 0x7f5a3413cc30 2023-09-30T01:35:42.230 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:42.226+0000 7f5a1bfff700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:35:42.230 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:42.226+0000 7f5a1bfff700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:35:42.230 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-09-30T01:35:42.231 INFO:teuthology.orchestra.run.smithi088.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:35:42.232 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:42.230+0000 7f5a393a3700 1 -- 172.21.15.88:0/3500624022 >> v2:172.21.15.107:6812/29926 conn(0x7f5a20033360 msgr2=0x7f5a20035810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:35:42.232 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:42.230+0000 7f5a393a3700 1 --2- 172.21.15.88:0/3500624022 >> v2:172.21.15.107:6812/29926 conn(0x7f5a20033360 0x7f5a20035810 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f5a28009dd0 tx=0x7f5a280091a0 comp rx=0 tx=0).stop 2023-09-30T01:35:42.232 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:42.230+0000 7f5a393a3700 10 monclient: shutdown 2023-09-30T01:35:42.233 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:42.230+0000 7f5a393a3700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:35:42.233 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:42.230+0000 7f5a393a3700 1 -- 172.21.15.88:0/3500624022 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7f5a3413cc30 msgr2=0x7f5a34139a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:35:42.233 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:42.230+0000 7f5a393a3700 1 --2- 172.21.15.88:0/3500624022 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7f5a3413cc30 0x7f5a34139a70 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f5a3406e9b0 tx=0x7f5a2c017000 comp rx=0 tx=0).stop 2023-09-30T01:35:42.233 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:42.230+0000 7f5a393a3700 1 -- 172.21.15.88:0/3500624022 shutdown_connections 2023-09-30T01:35:42.233 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:42.230+0000 7f5a393a3700 1 --2- 172.21.15.88:0/3500624022 >> v2:172.21.15.107:6812/29926 conn(0x7f5a20033360 0x7f5a20035810 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:35:42.233 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:42.230+0000 7f5a393a3700 1 --2- 172.21.15.88:0/3500624022 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7f5a3413cc30 0x7f5a34139a70 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:35:42.233 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:42.230+0000 7f5a393a3700 1 --2- 172.21.15.88:0/3500624022 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7f5a34137ac0 0x7f5a34139530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:35:42.233 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:42.230+0000 7f5a393a3700 1 --2- 172.21.15.88:0/3500624022 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f5a341376f0 0x7f5a34130eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:35:42.233 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:42.230+0000 7f5a393a3700 1 -- 172.21.15.88:0/3500624022 >> 172.21.15.88:0/3500624022 conn(0x7f5a340a66f0 msgr2=0x7f5a340a4530 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:35:42.233 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:42.230+0000 7f5a393a3700 1 -- 172.21.15.88:0/3500624022 shutdown_connections 2023-09-30T01:35:42.234 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:42.230+0000 7f5a393a3700 1 -- 172.21.15.88:0/3500624022 wait complete. 2023-09-30T01:35:45.244 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:35:45.356 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:45.354+0000 7f04a8bcc700 10 monclient: get_monmap_and_config 2023-09-30T01:35:45.356 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:45.354+0000 7f04a8bcc700 10 monclient: build_initial_monmap 2023-09-30T01:35:45.357 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:45.354+0000 7f04a8bcc700 10 monclient: monmap: 2023-09-30T01:35:45.357 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2023-09-30T01:35:45.357 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:35:45.357 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:35:45.356616+0000 2023-09-30T01:35:45.357 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:35:45.356616+0000 2023-09-30T01:35:45.357 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2023-09-30T01:35:45.357 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:35:45.357 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a-legacy 2023-09-30T01:35:45.357 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.107:6789/0 mon.noname-f-legacy 2023-09-30T01:35:45.357 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.noname-b 2023-09-30T01:35:45.357 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.noname-g 2023-09-30T01:35:45.357 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.noname-c 2023-09-30T01:35:45.358 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.noname-h 2023-09-30T01:35:45.358 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.noname-d 2023-09-30T01:35:45.358 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.noname-i 2023-09-30T01:35:45.358 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.noname-e 2023-09-30T01:35:45.358 INFO:teuthology.orchestra.run.smithi088.stderr:9: v2:172.21.15.88:3300/0 mon.noname-a 2023-09-30T01:35:45.358 INFO:teuthology.orchestra.run.smithi088.stderr:10: v2:172.21.15.107:3300/0 mon.noname-f 2023-09-30T01:35:45.358 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:35:45.361 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:45.354+0000 7f04a8bcc700 1 Processor -- start 2023-09-30T01:35:45.361 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:45.354+0000 7f04a8bcc700 1 -- start start 2023-09-30T01:35:45.362 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:45.354+0000 7f04a8bcc700 10 monclient: init 2023-09-30T01:35:45.362 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:45.354+0000 7f04a8bcc700 10 monclient: _reopen_session rank -1 2023-09-30T01:35:45.362 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:45.354+0000 7f04a8bcc700 10 monclient: _add_conns ranks=[9,1,7,2,4,0,6,10,8,3,5] 2023-09-30T01:35:45.362 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:45.354+0000 7f04a8bcc700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f04a40a33f0 0x7f04a40a15a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:35:45.362 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:45.354+0000 7f04a8bcc700 10 monclient(hunting): picked mon.noname-a con 0x7f04a40a33f0 addr v2:172.21.15.88:3300/0 2023-09-30T01:35:45.362 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:45.354+0000 7f04a8bcc700 10 monclient(hunting): picked mon.noname-f-legacy con 0x7f04a409feb0 addr v1:172.21.15.107:6789/0 2023-09-30T01:35:45.362 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:45.354+0000 7f04a8bcc700 1 --2- >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7f04a40a0280 0x7f04a4133a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:35:45.362 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:45.354+0000 7f04a8bcc700 10 monclient(hunting): picked mon.noname-i con 0x7f04a40a0280 addr [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] 2023-09-30T01:35:45.362 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:45.354+0000 7f04a8bcc700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04a405ecb0 con 0x7f04a409feb0 2023-09-30T01:35:45.362 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:45.354+0000 7f04a8bcc700 10 monclient(hunting): start opening mon connection 2023-09-30T01:35:45.362 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:45.354+0000 7f04a8bcc700 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f04a4098160 con 0x7f04a40a33f0 2023-09-30T01:35:45.363 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:45.354+0000 7f04a8bcc700 10 monclient(hunting): start opening mon connection 2023-09-30T01:35:45.363 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:45.354+0000 7f04a8bcc700 1 -- --> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] -- mon_getmap magic: 0 v1 -- 0x7f04a4098360 con 0x7f04a40a0280 2023-09-30T01:35:45.363 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:45.354+0000 7f04a8bcc700 10 monclient(hunting): _renew_subs 2023-09-30T01:35:45.363 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:45.354+0000 7f04a8bcc700 10 monclient(hunting): authenticate will time out at 1017.394135s 2023-09-30T01:35:45.363 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:45.354+0000 7f04a259c700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f04a40a33f0 0x7f04a40a15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:35:45.363 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:45.354+0000 7f04a259c700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f04a40a33f0 0x7f04a40a15a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:36468/0 (socket says 172.21.15.88:36468) 2023-09-30T01:35:45.363 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:45.354+0000 7f04a259c700 1 -- 172.21.15.88:0/4241169555 learned_addr learned my addr 172.21.15.88:0/4241169555 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-09-30T01:35:45.363 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:45.354+0000 7f04a259c700 10 monclient(hunting): get_auth_request con 0x7f04a40a33f0 auth_method 0 2023-09-30T01:35:45.363 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:45.354+0000 7f04a259c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:35:45.363 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:45.354+0000 7f04a259c700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:35:45.363 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:45.354+0000 7f04a259c700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:35:45.364 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:45.354+0000 7f04a259c700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:35:45.364 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:45.354+0000 7f04a259c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:35:45.364 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:45.354+0000 7f04a259c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:35:45.364 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:45.354+0000 7f04a2d9d700 1 --2- 172.21.15.88:0/4241169555 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7f04a40a0280 0x7f04a4133a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:35:45.364 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:45.354+0000 7f04a2d9d700 10 monclient(hunting): get_auth_request con 0x7f04a40a0280 auth_method 0 2023-09-30T01:35:45.364 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:45.354+0000 7f04a2d9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:35:45.364 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:45.354+0000 7f04a2d9d700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:35:45.364 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:45.354+0000 7f04a2d9d700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:35:45.364 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:45.354+0000 7f04a259c700 10 monclient(hunting): handle_auth_done global_id 4851 payload 274 2023-09-30T01:35:45.364 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:45.354+0000 7f04a259c700 1 -- 172.21.15.88:0/4241169555 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7f04a40a0280 msgr2=0x7f04a4133a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:35:45.364 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:45.354+0000 7f04a259c700 1 --2- 172.21.15.88:0/4241169555 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7f04a40a0280 0x7f04a4133a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:35:45.364 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:45.354+0000 7f04a259c700 1 -- 172.21.15.88:0/4241169555 >> v1:172.21.15.107:6789/0 conn(0x7f04a409feb0 legacy=0x7f04a4133310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:35:45.365 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:45.354+0000 7f04a259c700 10 monclient: _finish_hunting 0 2023-09-30T01:35:45.365 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:45.354+0000 7f04a259c700 1 monclient: found mon.noname-a 2023-09-30T01:35:45.365 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:45.354+0000 7f04a259c700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:35:45.365 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:45.354+0000 7f04a259c700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.88:3300/0 2023-09-30T01:35:45.365 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:45.354+0000 7f04a259c700 1 -- 172.21.15.88:0/4241169555 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04a40a37c0 con 0x7f04a40a33f0 2023-09-30T01:35:45.365 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:45.354+0000 7f04a259c700 1 --2- 172.21.15.88:0/4241169555 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f04a40a33f0 0x7f04a40a15a0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f04940099b0 tx=0x7f0494009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2fbaf0f8f559c73c server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:35:45.365 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:45.354+0000 7f04a1d9b700 1 -- 172.21.15.88:0/4241169555 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f049400bce0 con 0x7f04a40a33f0 2023-09-30T01:35:45.365 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:45.354+0000 7f04a1d9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:35:45.365 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:45.354+0000 7f04a1d9b700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T01:35:45.365 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:45.354+0000 7f04a1d9b700 10 monclient: dump: 2023-09-30T01:35:45.365 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:35:45.365 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:35:45.366 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:35:45.366 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:35:45.366 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:35:45.366 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:35:45.366 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:35:45.366 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:35:45.366 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:35:45.366 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:35:45.366 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:35:45.366 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:35:45.366 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:35:45.366 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:35:45.367 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:35:45.367 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:35:45.367 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:45.354+0000 7f04a1d9b700 10 monclient: _finish_auth 0 2023-09-30T01:35:45.367 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:45.354+0000 7f04a1d9b700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:35:45.367 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:45.354+0000 7f04a8bcc700 5 monclient: authenticate success, global_id 4851 2023-09-30T01:35:45.367 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:45.354+0000 7f04a8bcc700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:35:45.367 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:45.354+0000 7f04a1d9b700 1 -- 172.21.15.88:0/4241169555 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f049400be60 con 0x7f04a40a33f0 2023-09-30T01:35:45.367 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:45.354+0000 7f04a1d9b700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:35:45.367 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:45.354+0000 7f04a8bcc700 10 monclient: get_monmap_and_config success 2023-09-30T01:35:45.367 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:45.354+0000 7f04a8bcc700 4 set_mon_vals no callback set 2023-09-30T01:35:45.367 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:45.354+0000 7f04a1d9b700 1 -- 172.21.15.88:0/4241169555 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f04940118a0 con 0x7f04a40a33f0 2023-09-30T01:35:45.367 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:45.354+0000 7f04a1d9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:35:45.367 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:45.354+0000 7f04a1d9b700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:35:45.368 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:45.354+0000 7f04a1d9b700 10 monclient: dump: 2023-09-30T01:35:45.368 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:35:45.368 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:35:45.368 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:35:45.368 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:35:45.368 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:35:45.368 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:35:45.368 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:35:45.368 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:35:45.368 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:35:45.368 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:35:45.368 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:35:45.369 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:35:45.369 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:35:45.369 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:35:45.369 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:35:45.369 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:35:45.369 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:45.358+0000 7f04a8bcc700 10 monclient: shutdown 2023-09-30T01:35:45.369 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:45.358+0000 7f04a8bcc700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:35:45.369 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:45.358+0000 7f04a8bcc700 1 -- 172.21.15.88:0/4241169555 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f04a40a33f0 msgr2=0x7f04a40a15a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:35:45.369 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:45.358+0000 7f04a8bcc700 1 --2- 172.21.15.88:0/4241169555 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f04a40a33f0 0x7f04a40a15a0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f04940099b0 tx=0x7f0494009250 comp rx=0 tx=0).stop 2023-09-30T01:35:45.369 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:45.358+0000 7f04a8bcc700 1 -- 172.21.15.88:0/4241169555 shutdown_connections 2023-09-30T01:35:45.369 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:45.358+0000 7f04a8bcc700 1 --2- 172.21.15.88:0/4241169555 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7f04a40a0280 0x7f04a4133a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:35:45.369 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:45.358+0000 7f04a8bcc700 1 --2- 172.21.15.88:0/4241169555 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f04a40a33f0 0x7f04a40a15a0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:35:45.369 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:45.358+0000 7f04a8bcc700 1 -- 172.21.15.88:0/4241169555 >> 172.21.15.88:0/4241169555 conn(0x7f04a406e5e0 msgr2=0x7f04a406e9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:35:45.370 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:45.358+0000 7f04a8bcc700 1 -- 172.21.15.88:0/4241169555 shutdown_connections 2023-09-30T01:35:45.370 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:45.358+0000 7f04a8bcc700 1 -- 172.21.15.88:0/4241169555 wait complete. 2023-09-30T01:35:45.370 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:45.358+0000 7f04a8bcc700 10 monclient: build_initial_monmap 2023-09-30T01:35:45.370 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:45.358+0000 7f04a8bcc700 10 monclient: monmap: 2023-09-30T01:35:45.370 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2023-09-30T01:35:45.370 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:35:45.370 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 0.000000 2023-09-30T01:35:45.370 INFO:teuthology.orchestra.run.smithi088.stderr:created 0.000000 2023-09-30T01:35:45.370 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2023-09-30T01:35:45.370 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 1 2023-09-30T01:35:45.370 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.noname-a 2023-09-30T01:35:45.370 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.noname-f 2023-09-30T01:35:45.371 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.noname-b 2023-09-30T01:35:45.371 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.noname-g 2023-09-30T01:35:45.371 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.noname-c 2023-09-30T01:35:45.371 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.noname-h 2023-09-30T01:35:45.371 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.noname-d 2023-09-30T01:35:45.371 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.noname-i 2023-09-30T01:35:45.371 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.noname-e 2023-09-30T01:35:45.371 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:35:45.371 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:45.358+0000 7f04a8bcc700 1 Processor -- start 2023-09-30T01:35:45.371 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:45.358+0000 7f04a8bcc700 1 -- start start 2023-09-30T01:35:45.371 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:45.358+0000 7f04a8bcc700 10 monclient: init 2023-09-30T01:35:45.375 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:45.358+0000 7f04a8bcc700 10 monclient: _reopen_session rank -1 2023-09-30T01:35:45.375 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:45.358+0000 7f04a8bcc700 10 monclient: _add_conns ranks=[5,1,0,6,2,7,8,3,4] 2023-09-30T01:35:45.375 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:45.358+0000 7f04a8bcc700 1 --2- >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7f04a409feb0 0x7f04a4132fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:35:45.375 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:45.358+0000 7f04a8bcc700 10 monclient(hunting): picked mon.noname-h con 0x7f04a409feb0 addr [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] 2023-09-30T01:35:45.376 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:45.358+0000 7f04a8bcc700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f04a40a0280 0x7f04a411ea90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:35:45.376 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:45.358+0000 7f04a8bcc700 10 monclient(hunting): picked mon.noname-f con 0x7f04a40a0280 addr [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] 2023-09-30T01:35:45.376 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:45.358+0000 7f04a8bcc700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f04a40a33f0 0x7f04a411efd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:35:45.376 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:45.358+0000 7f04a8bcc700 10 monclient(hunting): picked mon.noname-a con 0x7f04a40a33f0 addr [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] 2023-09-30T01:35:45.376 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:45.358+0000 7f04a8bcc700 10 monclient(hunting): start opening mon connection 2023-09-30T01:35:45.376 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:45.358+0000 7f04a8bcc700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04a40a09c0 con 0x7f04a40a33f0 2023-09-30T01:35:45.376 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:45.358+0000 7f04a8bcc700 10 monclient(hunting): start opening mon connection 2023-09-30T01:35:45.376 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:45.358+0000 7f04a8bcc700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04a40a2ff0 con 0x7f04a40a0280 2023-09-30T01:35:45.376 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:45.358+0000 7f04a8bcc700 10 monclient(hunting): start opening mon connection 2023-09-30T01:35:45.376 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:45.358+0000 7f04a8bcc700 1 -- --> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] -- mon_getmap magic: 0 v1 -- 0x7f04a41223f0 con 0x7f04a409feb0 2023-09-30T01:35:45.376 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:45.358+0000 7f04a8bcc700 10 monclient(hunting): _renew_subs 2023-09-30T01:35:45.376 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:45.358+0000 7f04a8bcc700 10 monclient(hunting): authenticate will time out at 1017.398933s 2023-09-30T01:35:45.377 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:45.358+0000 7f04a2d9d700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f04a40a33f0 0x7f04a411efd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:35:45.377 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:45.358+0000 7f04a2d9d700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f04a40a33f0 0x7f04a411efd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:36482/0 (socket says 172.21.15.88:36482) 2023-09-30T01:35:45.377 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:45.358+0000 7f04a2d9d700 1 -- 172.21.15.88:0/2291621489 learned_addr learned my addr 172.21.15.88:0/2291621489 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-09-30T01:35:45.377 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:45.358+0000 7f04a2d9d700 10 monclient(hunting): get_auth_request con 0x7f04a40a33f0 auth_method 0 2023-09-30T01:35:45.377 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:45.358+0000 7f04a2d9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:35:45.377 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:45.358+0000 7f04a2d9d700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:35:45.377 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:45.358+0000 7f04a2d9d700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:35:45.377 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:45.358+0000 7f0499d9b700 1 --2- 172.21.15.88:0/2291621489 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f04a40a0280 0x7f04a411ea90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:35:45.377 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:45.358+0000 7f04a2d9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:35:45.377 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:45.358+0000 7f04a2d9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:35:45.377 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:45.358+0000 7f04a259c700 1 --2- 172.21.15.88:0/2291621489 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7f04a409feb0 0x7f04a4132fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:35:45.377 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:45.358+0000 7f04a2d9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:35:45.378 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:45.358+0000 7f0499d9b700 10 monclient(hunting): get_auth_request con 0x7f04a40a0280 auth_method 0 2023-09-30T01:35:45.378 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:45.358+0000 7f0499d9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:35:45.378 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:45.358+0000 7f0499d9b700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:35:45.378 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:45.358+0000 7f0499d9b700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:35:45.378 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:45.358+0000 7f04a259c700 10 monclient(hunting): get_auth_request con 0x7f04a409feb0 auth_method 0 2023-09-30T01:35:45.378 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:45.358+0000 7f04a259c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:35:45.378 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:45.358+0000 7f04a259c700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:35:45.378 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:45.358+0000 7f04a259c700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:35:45.378 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:45.358+0000 7f0499d9b700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:35:45.378 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:45.358+0000 7f0499d9b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:35:45.378 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:45.358+0000 7f0499d9b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:35:45.378 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:45.358+0000 7f04a259c700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:35:45.379 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:45.358+0000 7f04a259c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:35:45.379 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:45.358+0000 7f04a259c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:35:45.379 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:45.358+0000 7f04a2d9d700 10 monclient(hunting): handle_auth_done global_id 4860 payload 824 2023-09-30T01:35:45.379 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:45.358+0000 7f04a2d9d700 1 -- 172.21.15.88:0/2291621489 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7f04a409feb0 msgr2=0x7f04a4132fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:35:45.379 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:45.358+0000 7f04a2d9d700 1 --2- 172.21.15.88:0/2291621489 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7f04a409feb0 0x7f04a4132fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:35:45.379 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:45.358+0000 7f04a2d9d700 1 -- 172.21.15.88:0/2291621489 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f04a40a0280 msgr2=0x7f04a411ea90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:35:45.379 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:45.358+0000 7f04a2d9d700 1 --2- 172.21.15.88:0/2291621489 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f04a40a0280 0x7f04a411ea90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:35:45.379 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:45.358+0000 7f04a2d9d700 10 monclient: _finish_hunting 0 2023-09-30T01:35:45.379 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:45.358+0000 7f04a2d9d700 1 monclient: found mon.noname-a 2023-09-30T01:35:45.379 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:45.358+0000 7f04a2d9d700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:35:45.379 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:45.358+0000 7f04a2d9d700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.88:3300/0 2023-09-30T01:35:45.379 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:45.358+0000 7f04a2d9d700 1 -- 172.21.15.88:0/2291621489 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04940094f0 con 0x7f04a40a33f0 2023-09-30T01:35:45.380 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:45.362+0000 7f04a2d9d700 1 --2- 172.21.15.88:0/2291621489 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f04a40a33f0 0x7f04a411efd0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f049c009fa0 tx=0x7f049c009900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:35:45.380 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:45.362+0000 7f049affd700 1 -- 172.21.15.88:0/2291621489 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f049c009990 con 0x7f04a40a33f0 2023-09-30T01:35:45.380 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:45.362+0000 7f049affd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:35:45.380 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:45.362+0000 7f049affd700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T01:35:45.380 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:45.362+0000 7f049affd700 10 monclient: dump: 2023-09-30T01:35:45.380 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:35:45.380 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:35:45.380 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:35:45.380 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:35:45.380 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:35:45.380 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:35:45.380 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:35:45.381 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:35:45.381 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:35:45.381 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:35:45.381 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:35:45.381 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:35:45.381 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:35:45.381 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:35:45.381 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:35:45.381 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:35:45.381 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:45.362+0000 7f049affd700 10 monclient: _finish_auth 0 2023-09-30T01:35:45.381 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:45.362+0000 7f049affd700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:35:45.381 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:45.362+0000 7f04a8bcc700 5 monclient: authenticate success, global_id 4860 2023-09-30T01:35:45.382 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:45.362+0000 7f049affd700 1 -- 172.21.15.88:0/2291621489 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f049c00ebc0 con 0x7f04a40a33f0 2023-09-30T01:35:45.382 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:45.362+0000 7f04a8bcc700 10 monclient: _renew_subs 2023-09-30T01:35:45.382 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:45.362+0000 7f04a8bcc700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.88:3300/0 2023-09-30T01:35:45.382 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:45.362+0000 7f04a8bcc700 1 -- 172.21.15.88:0/2291621489 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f04a4122110 con 0x7f04a40a33f0 2023-09-30T01:35:45.382 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:45.362+0000 7f049affd700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:35:45.382 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:45.362+0000 7f04a8bcc700 10 monclient: _renew_subs 2023-09-30T01:35:45.382 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:45.362+0000 7f04a8bcc700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.88:3300/0 2023-09-30T01:35:45.382 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:45.362+0000 7f04a159a700 4 set_mon_vals no callback set 2023-09-30T01:35:45.382 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:45.362+0000 7f04a8bcc700 1 -- 172.21.15.88:0/2291621489 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f04a4121de0 con 0x7f04a40a33f0 2023-09-30T01:35:45.382 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:45.362+0000 7f049affd700 1 -- 172.21.15.88:0/2291621489 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f049c00ed30 con 0x7f04a40a33f0 2023-09-30T01:35:45.382 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:45.362+0000 7f049affd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:35:45.383 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:45.362+0000 7f049affd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:35:45.383 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:45.362+0000 7f049affd700 10 monclient: dump: 2023-09-30T01:35:45.383 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:35:45.383 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:35:45.383 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:35:45.383 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:35:45.383 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:35:45.383 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:35:45.383 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:35:45.383 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:35:45.383 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:35:45.383 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:35:45.384 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:35:45.384 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:35:45.384 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:35:45.384 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:35:45.384 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:35:45.384 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:35:45.384 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:45.362+0000 7f04a8bcc700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:35:45.384 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:45.362+0000 7f04a8bcc700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:35:45.384 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:45.362+0000 7f04a8bcc700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.88:3300/0 2023-09-30T01:35:45.384 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:45.362+0000 7f04a8bcc700 1 -- 172.21.15.88:0/2291621489 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f04a4122630 con 0x7f04a40a33f0 2023-09-30T01:35:45.384 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:45.362+0000 7f049affd700 1 -- 172.21.15.88:0/2291621489 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 40144+0+0 (secure 0 0 0) 0x7f049c01e460 con 0x7f04a40a33f0 2023-09-30T01:35:45.384 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:45.362+0000 7f049affd700 1 --2- 172.21.15.88:0/2291621489 >> v2:172.21.15.107:6812/29926 conn(0x7f0484033360 0x7f0484035810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:35:45.385 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:45.362+0000 7f049affd700 1 -- 172.21.15.88:0/2291621489 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 3502+0+0 (secure 0 0 0) 0x7f049c043a10 con 0x7f04a40a33f0 2023-09-30T01:35:45.385 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:45.362+0000 7f04a259c700 1 --2- 172.21.15.88:0/2291621489 >> v2:172.21.15.107:6812/29926 conn(0x7f0484033360 0x7f0484035810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:35:45.385 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:45.362+0000 7f04a259c700 10 monclient: get_auth_request con 0x7f0484033360 auth_method 0 2023-09-30T01:35:45.385 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:45.366+0000 7f04a259c700 1 --2- 172.21.15.88:0/2291621489 >> v2:172.21.15.107:6812/29926 conn(0x7f0484033360 0x7f0484035810 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f04940099b0 tx=0x7f0494017000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:35:45.385 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:45.370+0000 7f049affd700 1 -- 172.21.15.88:0/2291621489 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f04a4122630 con 0x7f04a40a33f0 2023-09-30T01:35:45.385 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:45.370+0000 7f049affd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:35:45.385 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:45.370+0000 7f049affd700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:35:45.573 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:45.570+0000 7f04a8bcc700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T01:35:45.573 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:45.570+0000 7f04a8bcc700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:35:45.573 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:45.570+0000 7f04a8bcc700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.88:3300/0 2023-09-30T01:35:45.573 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:45.570+0000 7f04a8bcc700 1 -- 172.21.15.88:0/2291621489 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f04a4122630 con 0x7f04a40a33f0 2023-09-30T01:35:45.574 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:45.570+0000 7f049affd700 1 -- 172.21.15.88:0/2291621489 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f04a4122630 con 0x7f04a40a33f0 2023-09-30T01:35:45.574 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:45.570+0000 7f049affd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:35:45.574 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:45.570+0000 7f049affd700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:35:45.574 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-09-30T01:35:45.574 INFO:teuthology.orchestra.run.smithi088.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:35:45.576 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:45.574+0000 7f04a8bcc700 1 -- 172.21.15.88:0/2291621489 >> v2:172.21.15.107:6812/29926 conn(0x7f0484033360 msgr2=0x7f0484035810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:35:45.576 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:45.574+0000 7f04a8bcc700 1 --2- 172.21.15.88:0/2291621489 >> v2:172.21.15.107:6812/29926 conn(0x7f0484033360 0x7f0484035810 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f04940099b0 tx=0x7f0494017000 comp rx=0 tx=0).stop 2023-09-30T01:35:45.576 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:45.574+0000 7f04a8bcc700 10 monclient: shutdown 2023-09-30T01:35:45.576 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:45.574+0000 7f04a8bcc700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:35:45.576 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:45.574+0000 7f04a8bcc700 1 -- 172.21.15.88:0/2291621489 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f04a40a33f0 msgr2=0x7f04a411efd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:35:45.576 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:45.574+0000 7f04a8bcc700 1 --2- 172.21.15.88:0/2291621489 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f04a40a33f0 0x7f04a411efd0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f049c009fa0 tx=0x7f049c009900 comp rx=0 tx=0).stop 2023-09-30T01:35:45.577 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:45.574+0000 7f04a8bcc700 1 -- 172.21.15.88:0/2291621489 shutdown_connections 2023-09-30T01:35:45.577 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:45.574+0000 7f04a8bcc700 1 --2- 172.21.15.88:0/2291621489 >> v2:172.21.15.107:6812/29926 conn(0x7f0484033360 0x7f0484035810 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:35:45.577 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:45.574+0000 7f04a8bcc700 1 --2- 172.21.15.88:0/2291621489 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f04a40a33f0 0x7f04a411efd0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:35:45.577 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:45.574+0000 7f04a8bcc700 1 --2- 172.21.15.88:0/2291621489 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f04a40a0280 0x7f04a411ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:35:45.577 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:45.574+0000 7f04a8bcc700 1 --2- 172.21.15.88:0/2291621489 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7f04a409feb0 0x7f04a4132fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:35:45.577 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:45.574+0000 7f04a8bcc700 1 -- 172.21.15.88:0/2291621489 >> 172.21.15.88:0/2291621489 conn(0x7f04a406e5e0 msgr2=0x7f04a4134ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:35:45.577 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:45.574+0000 7f04a8bcc700 1 -- 172.21.15.88:0/2291621489 shutdown_connections 2023-09-30T01:35:45.577 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:45.574+0000 7f04a8bcc700 1 -- 172.21.15.88:0/2291621489 wait complete. 2023-09-30T01:35:48.588 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:35:48.697 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:48.694+0000 7ff79adc6700 10 monclient: get_monmap_and_config 2023-09-30T01:35:48.697 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:48.694+0000 7ff79adc6700 10 monclient: build_initial_monmap 2023-09-30T01:35:48.697 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:48.694+0000 7ff79adc6700 10 monclient: monmap: 2023-09-30T01:35:48.697 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2023-09-30T01:35:48.697 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:35:48.697 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:35:48.696863+0000 2023-09-30T01:35:48.697 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:35:48.696863+0000 2023-09-30T01:35:48.697 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2023-09-30T01:35:48.697 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:35:48.698 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a-legacy 2023-09-30T01:35:48.698 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.107:6789/0 mon.noname-f-legacy 2023-09-30T01:35:48.698 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.noname-b 2023-09-30T01:35:48.698 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.noname-g 2023-09-30T01:35:48.698 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.noname-c 2023-09-30T01:35:48.698 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.noname-h 2023-09-30T01:35:48.698 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.noname-d 2023-09-30T01:35:48.698 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.noname-i 2023-09-30T01:35:48.698 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.noname-e 2023-09-30T01:35:48.698 INFO:teuthology.orchestra.run.smithi088.stderr:9: v2:172.21.15.88:3300/0 mon.noname-a 2023-09-30T01:35:48.699 INFO:teuthology.orchestra.run.smithi088.stderr:10: v2:172.21.15.107:3300/0 mon.noname-f 2023-09-30T01:35:48.699 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:35:48.708 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:48.694+0000 7ff79adc6700 1 Processor -- start 2023-09-30T01:35:48.708 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:48.694+0000 7ff79adc6700 1 -- start start 2023-09-30T01:35:48.708 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:48.694+0000 7ff79adc6700 10 monclient: init 2023-09-30T01:35:48.708 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:48.694+0000 7ff79adc6700 10 monclient: _reopen_session rank -1 2023-09-30T01:35:48.708 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:48.694+0000 7ff79adc6700 10 monclient: _add_conns ranks=[3,6,2,0,5,8,7,9,1,10,4] 2023-09-30T01:35:48.709 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:48.694+0000 7ff79adc6700 1 --2- >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7ff7940a3270 0x7ff7940a0160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:35:48.709 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:48.694+0000 7ff79adc6700 10 monclient(hunting): picked mon.noname-g con 0x7ff7940a3270 addr [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] 2023-09-30T01:35:48.709 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:48.694+0000 7ff79adc6700 1 --2- >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7ff7940a4b00 0x7ff7940a06a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:35:48.709 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:48.694+0000 7ff79adc6700 10 monclient(hunting): picked mon.noname-d con 0x7ff7940a4b00 addr [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] 2023-09-30T01:35:48.709 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:48.694+0000 7ff79adc6700 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff7941333c0 0x7ff7941337b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:35:48.709 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:48.694+0000 7ff79adc6700 10 monclient(hunting): picked mon.noname-b con 0x7ff7941333c0 addr [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] 2023-09-30T01:35:48.709 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:48.694+0000 7ff79adc6700 10 monclient(hunting): start opening mon connection 2023-09-30T01:35:48.709 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:48.694+0000 7ff79adc6700 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff7940a4ed0 con 0x7ff7941333c0 2023-09-30T01:35:48.709 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:48.694+0000 7ff79adc6700 10 monclient(hunting): start opening mon connection 2023-09-30T01:35:48.709 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:48.694+0000 7ff79adc6700 1 -- --> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff7940a6330 con 0x7ff7940a3270 2023-09-30T01:35:48.710 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:48.694+0000 7ff79adc6700 10 monclient(hunting): start opening mon connection 2023-09-30T01:35:48.710 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:48.694+0000 7ff79adc6700 1 -- --> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] -- mon_getmap magic: 0 v1 -- 0x7ff794098610 con 0x7ff7940a4b00 2023-09-30T01:35:48.710 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:48.694+0000 7ff79adc6700 10 monclient(hunting): _renew_subs 2023-09-30T01:35:48.710 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:48.694+0000 7ff79adc6700 10 monclient(hunting): authenticate will time out at 1020.735187s 2023-09-30T01:35:48.710 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:48.694+0000 7ff793fff700 1 --2- >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7ff7940a4b00 0x7ff7940a06a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:35:48.710 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:48.694+0000 7ff799363700 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff7941333c0 0x7ff7941337b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:35:48.710 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:48.694+0000 7ff793fff700 1 --2- >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7ff7940a4b00 0x7ff7940a06a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3303/0 says I am v2:172.21.15.88:48448/0 (socket says 172.21.15.88:48448) 2023-09-30T01:35:48.710 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:48.694+0000 7ff793fff700 1 -- 172.21.15.88:0/1578666242 learned_addr learned my addr 172.21.15.88:0/1578666242 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-09-30T01:35:48.710 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:48.694+0000 7ff793fff700 10 monclient(hunting): get_auth_request con 0x7ff7940a4b00 auth_method 0 2023-09-30T01:35:48.711 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:48.694+0000 7ff793fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:35:48.711 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:48.694+0000 7ff793fff700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:35:48.711 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:48.694+0000 7ff793fff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:35:48.711 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:48.694+0000 7ff799363700 10 monclient(hunting): get_auth_request con 0x7ff7941333c0 auth_method 0 2023-09-30T01:35:48.711 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:48.694+0000 7ff799363700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:35:48.711 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:48.694+0000 7ff799363700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:35:48.711 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:48.694+0000 7ff799363700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:35:48.711 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:48.694+0000 7ff798b62700 1 --2- 172.21.15.88:0/1578666242 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7ff7940a3270 0x7ff7940a0160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:35:48.711 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:48.694+0000 7ff798b62700 10 monclient(hunting): get_auth_request con 0x7ff7940a3270 auth_method 0 2023-09-30T01:35:48.711 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:48.694+0000 7ff798b62700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:35:48.711 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:48.694+0000 7ff798b62700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:35:48.712 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:48.694+0000 7ff798b62700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:35:48.712 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:48.694+0000 7ff793fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:35:48.712 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:48.694+0000 7ff793fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:35:48.712 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:48.694+0000 7ff793fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:35:48.712 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:48.694+0000 7ff799363700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:35:48.712 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:48.694+0000 7ff799363700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:35:48.712 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:48.694+0000 7ff799363700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:35:48.712 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:48.694+0000 7ff798b62700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:35:48.712 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:48.694+0000 7ff798b62700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:35:48.712 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:48.694+0000 7ff798b62700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:35:48.713 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:48.698+0000 7ff793fff700 10 monclient(hunting): handle_auth_done global_id 4533 payload 274 2023-09-30T01:35:48.713 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:48.698+0000 7ff793fff700 1 -- 172.21.15.88:0/1578666242 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7ff7940a3270 msgr2=0x7ff7940a0160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:35:48.713 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:48.698+0000 7ff793fff700 1 --2- 172.21.15.88:0/1578666242 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7ff7940a3270 0x7ff7940a0160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:35:48.713 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:48.698+0000 7ff793fff700 1 -- 172.21.15.88:0/1578666242 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff7941333c0 msgr2=0x7ff7941337b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:35:48.713 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:48.698+0000 7ff793fff700 1 --2- 172.21.15.88:0/1578666242 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff7941333c0 0x7ff7941337b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:35:48.713 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:48.698+0000 7ff793fff700 10 monclient: _finish_hunting 0 2023-09-30T01:35:48.713 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:48.698+0000 7ff793fff700 1 monclient: found mon.noname-d 2023-09-30T01:35:48.713 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:48.698+0000 7ff793fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:35:48.713 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:48.698+0000 7ff793fff700 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.88:3303/0 2023-09-30T01:35:48.713 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:48.698+0000 7ff793fff700 1 -- 172.21.15.88:0/1578666242 --> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff79405ecb0 con 0x7ff7940a4b00 2023-09-30T01:35:48.714 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:48.698+0000 7ff799363700 1 --2- 172.21.15.88:0/1578666242 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff7941333c0 0x7ff7941337b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:35:48.714 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:48.698+0000 7ff793fff700 1 --2- 172.21.15.88:0/1578666242 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7ff7940a4b00 0x7ff7940a06a0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7ff7880099e0 tx=0x7ff788009250 comp rx=0 tx=0).ready entity=mon.6 client_cookie=3bd79e7ba26e0487 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:35:48.714 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:48.698+0000 7ff7937fe700 1 -- 172.21.15.88:0/1578666242 <== mon.6 v2:172.21.15.88:3303/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7ff788011ab0 con 0x7ff7940a4b00 2023-09-30T01:35:48.714 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:48.698+0000 7ff7937fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:35:48.714 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:48.698+0000 7ff7937fe700 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2023-09-30T01:35:48.714 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:48.698+0000 7ff7937fe700 10 monclient: dump: 2023-09-30T01:35:48.714 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:35:48.714 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:35:48.714 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:35:48.714 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:35:48.715 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:35:48.715 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:35:48.715 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:35:48.715 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:35:48.715 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:35:48.715 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:35:48.715 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:35:48.715 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:35:48.715 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:35:48.715 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:35:48.715 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:35:48.716 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:35:48.716 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:48.698+0000 7ff7937fe700 10 monclient: _finish_auth 0 2023-09-30T01:35:48.716 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:48.698+0000 7ff7937fe700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:35:48.716 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:48.698+0000 7ff79adc6700 5 monclient: authenticate success, global_id 4533 2023-09-30T01:35:48.716 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:48.698+0000 7ff7937fe700 1 -- 172.21.15.88:0/1578666242 <== mon.6 v2:172.21.15.88:3303/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff788011c30 con 0x7ff7940a4b00 2023-09-30T01:35:48.716 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:48.698+0000 7ff79adc6700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:35:48.716 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:48.698+0000 7ff7937fe700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:35:48.716 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:48.698+0000 7ff79adc6700 10 monclient: get_monmap_and_config success 2023-09-30T01:35:48.716 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:48.698+0000 7ff79adc6700 4 set_mon_vals no callback set 2023-09-30T01:35:48.716 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:48.698+0000 7ff7937fe700 1 -- 172.21.15.88:0/1578666242 <== mon.6 v2:172.21.15.88:3303/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7ff788011da0 con 0x7ff7940a4b00 2023-09-30T01:35:48.716 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:48.698+0000 7ff7937fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:35:48.717 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:48.698+0000 7ff7937fe700 10 monclient: got monmap 1 from mon.d (according to old e1) 2023-09-30T01:35:48.717 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:48.698+0000 7ff7937fe700 10 monclient: dump: 2023-09-30T01:35:48.717 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:35:48.717 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:35:48.717 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:35:48.717 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:35:48.717 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:35:48.717 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:35:48.717 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:35:48.717 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:35:48.718 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:35:48.718 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:35:48.718 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:35:48.718 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:35:48.718 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:35:48.718 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:35:48.718 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:35:48.718 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:35:48.718 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:48.698+0000 7ff79adc6700 10 monclient: shutdown 2023-09-30T01:35:48.718 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:48.698+0000 7ff79adc6700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:35:48.719 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:48.698+0000 7ff79adc6700 1 -- 172.21.15.88:0/1578666242 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7ff7940a4b00 msgr2=0x7ff7940a06a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:35:48.719 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:48.698+0000 7ff79adc6700 1 --2- 172.21.15.88:0/1578666242 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7ff7940a4b00 0x7ff7940a06a0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7ff7880099e0 tx=0x7ff788009250 comp rx=0 tx=0).stop 2023-09-30T01:35:48.719 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:48.698+0000 7ff79adc6700 1 -- 172.21.15.88:0/1578666242 shutdown_connections 2023-09-30T01:35:48.719 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:48.698+0000 7ff79adc6700 1 --2- 172.21.15.88:0/1578666242 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff7941333c0 0x7ff7941337b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:35:48.719 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:48.698+0000 7ff79adc6700 1 --2- 172.21.15.88:0/1578666242 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7ff7940a4b00 0x7ff7940a06a0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:35:48.719 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:48.698+0000 7ff79adc6700 1 --2- 172.21.15.88:0/1578666242 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7ff7940a3270 0x7ff7940a0160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:35:48.719 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:48.698+0000 7ff79adc6700 1 -- 172.21.15.88:0/1578666242 >> 172.21.15.88:0/1578666242 conn(0x7ff7940a67f0 msgr2=0x7ff7940a6bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:35:48.719 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:48.698+0000 7ff79adc6700 1 -- 172.21.15.88:0/1578666242 shutdown_connections 2023-09-30T01:35:48.719 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:48.698+0000 7ff79adc6700 1 -- 172.21.15.88:0/1578666242 wait complete. 2023-09-30T01:35:48.719 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:48.698+0000 7ff79adc6700 10 monclient: build_initial_monmap 2023-09-30T01:35:48.719 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:48.698+0000 7ff79adc6700 10 monclient: monmap: 2023-09-30T01:35:48.720 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2023-09-30T01:35:48.720 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:35:48.720 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 0.000000 2023-09-30T01:35:48.720 INFO:teuthology.orchestra.run.smithi088.stderr:created 0.000000 2023-09-30T01:35:48.720 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2023-09-30T01:35:48.720 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 1 2023-09-30T01:35:48.720 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.noname-a 2023-09-30T01:35:48.720 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.noname-f 2023-09-30T01:35:48.720 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.noname-b 2023-09-30T01:35:48.720 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.noname-g 2023-09-30T01:35:48.721 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.noname-c 2023-09-30T01:35:48.721 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.noname-h 2023-09-30T01:35:48.721 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.noname-d 2023-09-30T01:35:48.721 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.noname-i 2023-09-30T01:35:48.721 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.noname-e 2023-09-30T01:35:48.721 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:35:48.721 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:48.698+0000 7ff79adc6700 1 Processor -- start 2023-09-30T01:35:48.721 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:48.698+0000 7ff79adc6700 1 -- start start 2023-09-30T01:35:48.721 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:48.698+0000 7ff79adc6700 10 monclient: init 2023-09-30T01:35:48.721 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:48.702+0000 7ff79adc6700 10 monclient: _reopen_session rank -1 2023-09-30T01:35:48.721 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:48.702+0000 7ff79adc6700 10 monclient: _add_conns ranks=[1,8,5,2,6,7,3,4,0] 2023-09-30T01:35:48.722 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:48.702+0000 7ff79adc6700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff7940a3270 0x7ff79411d9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:35:48.722 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:48.702+0000 7ff79adc6700 10 monclient(hunting): picked mon.noname-f con 0x7ff7940a3270 addr [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] 2023-09-30T01:35:48.722 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:48.702+0000 7ff79adc6700 1 --2- >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7ff7940a4b00 0x7ff79411dee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:35:48.722 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:48.702+0000 7ff79adc6700 10 monclient(hunting): picked mon.noname-e con 0x7ff7940a4b00 addr [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] 2023-09-30T01:35:48.722 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:48.702+0000 7ff79adc6700 1 --2- >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7ff7941333c0 0x7ff79411e420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:35:48.722 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:48.702+0000 7ff79adc6700 10 monclient(hunting): picked mon.noname-h con 0x7ff7941333c0 addr [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] 2023-09-30T01:35:48.722 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:48.702+0000 7ff79adc6700 10 monclient(hunting): start opening mon connection 2023-09-30T01:35:48.722 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:48.702+0000 7ff79adc6700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7941304d0 con 0x7ff7940a3270 2023-09-30T01:35:48.722 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:48.702+0000 7ff79adc6700 10 monclient(hunting): start opening mon connection 2023-09-30T01:35:48.722 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:48.702+0000 7ff79adc6700 1 -- --> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] -- mon_getmap magic: 0 v1 -- 0x7ff794097c70 con 0x7ff7941333c0 2023-09-30T01:35:48.723 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:48.702+0000 7ff79adc6700 10 monclient(hunting): start opening mon connection 2023-09-30T01:35:48.723 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:48.702+0000 7ff79adc6700 1 -- --> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] -- mon_getmap magic: 0 v1 -- 0x7ff7940a0e80 con 0x7ff7940a4b00 2023-09-30T01:35:48.723 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:48.702+0000 7ff79adc6700 10 monclient(hunting): _renew_subs 2023-09-30T01:35:48.723 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:48.702+0000 7ff79adc6700 10 monclient(hunting): authenticate will time out at 1020.740408s 2023-09-30T01:35:48.723 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:48.702+0000 7ff793fff700 1 --2- >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7ff7940a4b00 0x7ff79411dee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:35:48.723 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:48.702+0000 7ff793fff700 1 --2- >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7ff7940a4b00 0x7ff79411dee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3304/0 says I am v2:172.21.15.88:60440/0 (socket says 172.21.15.88:60440) 2023-09-30T01:35:48.723 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:48.702+0000 7ff793fff700 1 -- 172.21.15.88:0/2438671791 learned_addr learned my addr 172.21.15.88:0/2438671791 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-09-30T01:35:48.723 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:48.702+0000 7ff793fff700 10 monclient(hunting): get_auth_request con 0x7ff7940a4b00 auth_method 0 2023-09-30T01:35:48.723 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:48.702+0000 7ff793fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:35:48.724 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:48.702+0000 7ff793fff700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:35:48.724 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:48.702+0000 7ff793fff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:35:48.724 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:48.702+0000 7ff798b62700 1 --2- 172.21.15.88:0/2438671791 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff7940a3270 0x7ff79411d9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:35:48.724 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:48.702+0000 7ff798b62700 10 monclient(hunting): get_auth_request con 0x7ff7940a3270 auth_method 0 2023-09-30T01:35:48.724 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:48.702+0000 7ff799363700 1 --2- 172.21.15.88:0/2438671791 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7ff7941333c0 0x7ff79411e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:35:48.724 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:48.702+0000 7ff798b62700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:35:48.724 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:48.702+0000 7ff798b62700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:35:48.724 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:48.702+0000 7ff798b62700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:35:48.724 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:48.702+0000 7ff793fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:35:48.724 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:48.702+0000 7ff793fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:35:48.724 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:48.702+0000 7ff793fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:35:48.725 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:48.702+0000 7ff799363700 10 monclient(hunting): get_auth_request con 0x7ff7941333c0 auth_method 0 2023-09-30T01:35:48.725 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:48.702+0000 7ff799363700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:35:48.725 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:48.702+0000 7ff799363700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:35:48.725 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:48.702+0000 7ff799363700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:35:48.725 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:48.702+0000 7ff798b62700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:35:48.725 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:48.702+0000 7ff798b62700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:35:48.725 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:48.702+0000 7ff798b62700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:35:48.725 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:48.702+0000 7ff799363700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:35:48.725 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:48.702+0000 7ff799363700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:35:48.725 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:48.702+0000 7ff799363700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:35:48.726 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:48.702+0000 7ff793fff700 10 monclient(hunting): handle_auth_done global_id 4679 payload 824 2023-09-30T01:35:48.726 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:48.702+0000 7ff793fff700 1 -- 172.21.15.88:0/2438671791 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7ff7941333c0 msgr2=0x7ff79411e420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:35:48.726 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:48.702+0000 7ff793fff700 1 --2- 172.21.15.88:0/2438671791 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7ff7941333c0 0x7ff79411e420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:35:48.726 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:48.702+0000 7ff793fff700 1 -- 172.21.15.88:0/2438671791 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff7940a3270 msgr2=0x7ff79411d9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:35:48.726 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:48.702+0000 7ff793fff700 1 --2- 172.21.15.88:0/2438671791 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff7940a3270 0x7ff79411d9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:35:48.726 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:48.702+0000 7ff793fff700 10 monclient: _finish_hunting 0 2023-09-30T01:35:48.726 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:48.702+0000 7ff793fff700 1 monclient: found mon.noname-e 2023-09-30T01:35:48.726 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:48.702+0000 7ff793fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:35:48.726 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:48.702+0000 7ff793fff700 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.88:3304/0 2023-09-30T01:35:48.726 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:48.702+0000 7ff793fff700 1 -- 172.21.15.88:0/2438671791 --> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff78c009a20 con 0x7ff7940a4b00 2023-09-30T01:35:48.727 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:48.702+0000 7ff793fff700 1 --2- 172.21.15.88:0/2438671791 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7ff7940a4b00 0x7ff79411dee0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7ff788006fd0 tx=0x7ff788017000 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:35:48.727 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:48.702+0000 7ff790ff9700 1 -- 172.21.15.88:0/2438671791 <== mon.8 v2:172.21.15.88:3304/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7ff78801cd70 con 0x7ff7940a4b00 2023-09-30T01:35:48.727 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:48.702+0000 7ff790ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:35:48.727 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:48.702+0000 7ff790ff9700 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2023-09-30T01:35:48.727 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:48.702+0000 7ff790ff9700 10 monclient: dump: 2023-09-30T01:35:48.727 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:35:48.727 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:35:48.727 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:35:48.727 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:35:48.727 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:35:48.727 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:35:48.728 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:35:48.728 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:35:48.728 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:35:48.728 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:35:48.728 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:35:48.728 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:35:48.728 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:35:48.728 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:35:48.728 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:35:48.728 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:35:48.729 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:48.702+0000 7ff790ff9700 10 monclient: _finish_auth 0 2023-09-30T01:35:48.729 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:48.702+0000 7ff790ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:35:48.729 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:48.702+0000 7ff79adc6700 5 monclient: authenticate success, global_id 4679 2023-09-30T01:35:48.729 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:48.702+0000 7ff790ff9700 1 -- 172.21.15.88:0/2438671791 <== mon.8 v2:172.21.15.88:3304/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff78801a450 con 0x7ff7940a4b00 2023-09-30T01:35:48.729 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:48.702+0000 7ff790ff9700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:35:48.729 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:48.702+0000 7ff79adc6700 10 monclient: _renew_subs 2023-09-30T01:35:48.729 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:48.702+0000 7ff790ff9700 1 -- 172.21.15.88:0/2438671791 <== mon.8 v2:172.21.15.88:3304/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7ff78801aa40 con 0x7ff7940a4b00 2023-09-30T01:35:48.729 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:48.702+0000 7ff79adc6700 10 monclient: _send_mon_message to mon.e at v2:172.21.15.88:3304/0 2023-09-30T01:35:48.729 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:48.702+0000 7ff79adc6700 1 -- 172.21.15.88:0/2438671791 --> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff7880094f0 con 0x7ff7940a4b00 2023-09-30T01:35:48.729 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:48.702+0000 7ff790ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:35:48.729 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:48.702+0000 7ff792ffd700 4 set_mon_vals no callback set 2023-09-30T01:35:48.730 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:48.702+0000 7ff790ff9700 10 monclient: got monmap 1 from mon.e (according to old e1) 2023-09-30T01:35:48.730 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:48.702+0000 7ff790ff9700 10 monclient: dump: 2023-09-30T01:35:48.730 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:35:48.730 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:35:48.730 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:35:48.730 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:35:48.730 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:35:48.730 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:35:48.730 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:35:48.730 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:35:48.731 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:35:48.731 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:35:48.731 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:35:48.731 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:35:48.731 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:35:48.731 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:35:48.731 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:35:48.731 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:35:48.731 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:48.702+0000 7ff79adc6700 10 monclient: _renew_subs 2023-09-30T01:35:48.731 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:48.702+0000 7ff79adc6700 10 monclient: _send_mon_message to mon.e at v2:172.21.15.88:3304/0 2023-09-30T01:35:48.732 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:48.702+0000 7ff79adc6700 1 -- 172.21.15.88:0/2438671791 --> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff794120ea0 con 0x7ff7940a4b00 2023-09-30T01:35:48.732 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:48.702+0000 7ff79adc6700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:35:48.732 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:48.702+0000 7ff79adc6700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:35:48.732 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:48.702+0000 7ff79adc6700 10 monclient: _send_mon_message to mon.e at v2:172.21.15.88:3304/0 2023-09-30T01:35:48.732 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:48.702+0000 7ff79adc6700 1 -- 172.21.15.88:0/2438671791 --> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff794071960 con 0x7ff7940a4b00 2023-09-30T01:35:48.732 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:48.706+0000 7ff790ff9700 1 -- 172.21.15.88:0/2438671791 <== mon.8 v2:172.21.15.88:3304/0 4 ==== mgrmap(e 5) v1 ==== 40144+0+0 (secure 0 0 0) 0x7ff78801a5c0 con 0x7ff7940a4b00 2023-09-30T01:35:48.732 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:48.706+0000 7ff790ff9700 1 --2- 172.21.15.88:0/2438671791 >> v2:172.21.15.107:6812/29926 conn(0x7ff780033290 0x7ff780035740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:35:48.732 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:48.706+0000 7ff790ff9700 1 -- 172.21.15.88:0/2438671791 <== mon.8 v2:172.21.15.88:3304/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 3502+0+0 (secure 0 0 0) 0x7ff7880446b0 con 0x7ff7940a4b00 2023-09-30T01:35:48.732 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:48.706+0000 7ff798b62700 1 --2- 172.21.15.88:0/2438671791 >> v2:172.21.15.107:6812/29926 conn(0x7ff780033290 0x7ff780035740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:35:48.732 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:48.706+0000 7ff798b62700 10 monclient: get_auth_request con 0x7ff780033290 auth_method 0 2023-09-30T01:35:48.733 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:48.706+0000 7ff798b62700 1 --2- 172.21.15.88:0/2438671791 >> v2:172.21.15.107:6812/29926 conn(0x7ff780033290 0x7ff780035740 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7ff784009db0 tx=0x7ff7840091a0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:35:48.733 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:48.710+0000 7ff790ff9700 1 -- 172.21.15.88:0/2438671791 <== mon.8 v2:172.21.15.88:3304/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7ff78801a870 con 0x7ff7940a4b00 2023-09-30T01:35:48.733 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:48.710+0000 7ff790ff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:35:48.734 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:48.710+0000 7ff790ff9700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:35:48.909 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:48.906+0000 7ff79adc6700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T01:35:48.909 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:48.906+0000 7ff79adc6700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:35:48.909 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:48.906+0000 7ff79adc6700 10 monclient: _send_mon_message to mon.e at v2:172.21.15.88:3304/0 2023-09-30T01:35:48.909 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:48.906+0000 7ff79adc6700 1 -- 172.21.15.88:0/2438671791 --> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7ff794130f30 con 0x7ff7940a4b00 2023-09-30T01:35:48.910 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:48.906+0000 7ff790ff9700 1 -- 172.21.15.88:0/2438671791 <== mon.8 v2:172.21.15.88:3304/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7ff788029080 con 0x7ff7940a4b00 2023-09-30T01:35:48.910 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:48.906+0000 7ff790ff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:35:48.910 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:48.906+0000 7ff790ff9700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:35:48.910 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-09-30T01:35:48.911 INFO:teuthology.orchestra.run.smithi088.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:35:48.912 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:48.910+0000 7ff79adc6700 1 -- 172.21.15.88:0/2438671791 >> v2:172.21.15.107:6812/29926 conn(0x7ff780033290 msgr2=0x7ff780035740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:35:48.912 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:48.910+0000 7ff79adc6700 1 --2- 172.21.15.88:0/2438671791 >> v2:172.21.15.107:6812/29926 conn(0x7ff780033290 0x7ff780035740 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7ff784009db0 tx=0x7ff7840091a0 comp rx=0 tx=0).stop 2023-09-30T01:35:48.912 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:48.910+0000 7ff79adc6700 10 monclient: shutdown 2023-09-30T01:35:48.912 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:48.910+0000 7ff79adc6700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:35:48.913 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:48.910+0000 7ff79adc6700 1 -- 172.21.15.88:0/2438671791 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7ff7940a4b00 msgr2=0x7ff79411dee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:35:48.913 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:48.910+0000 7ff79adc6700 1 --2- 172.21.15.88:0/2438671791 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7ff7940a4b00 0x7ff79411dee0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7ff788006fd0 tx=0x7ff788017000 comp rx=0 tx=0).stop 2023-09-30T01:35:48.913 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:48.910+0000 7ff79adc6700 1 -- 172.21.15.88:0/2438671791 shutdown_connections 2023-09-30T01:35:48.913 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:48.910+0000 7ff79adc6700 1 --2- 172.21.15.88:0/2438671791 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7ff7941333c0 0x7ff79411e420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:35:48.913 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:48.910+0000 7ff79adc6700 1 --2- 172.21.15.88:0/2438671791 >> v2:172.21.15.107:6812/29926 conn(0x7ff780033290 0x7ff780035740 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:35:48.913 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:48.910+0000 7ff79adc6700 1 --2- 172.21.15.88:0/2438671791 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7ff7940a4b00 0x7ff79411dee0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:35:48.913 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:48.910+0000 7ff79adc6700 1 --2- 172.21.15.88:0/2438671791 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff7940a3270 0x7ff79411d9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:35:48.913 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:48.910+0000 7ff79adc6700 1 -- 172.21.15.88:0/2438671791 >> 172.21.15.88:0/2438671791 conn(0x7ff7940a67f0 msgr2=0x7ff7940a20e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:35:48.913 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:48.910+0000 7ff79adc6700 1 -- 172.21.15.88:0/2438671791 shutdown_connections 2023-09-30T01:35:48.913 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:48.910+0000 7ff79adc6700 1 -- 172.21.15.88:0/2438671791 wait complete. 2023-09-30T01:35:51.924 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:35:52.037 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:52.034+0000 7fd8320c6700 10 monclient: get_monmap_and_config 2023-09-30T01:35:52.037 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:52.034+0000 7fd8320c6700 10 monclient: build_initial_monmap 2023-09-30T01:35:52.037 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:52.034+0000 7fd8320c6700 10 monclient: monmap: 2023-09-30T01:35:52.037 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2023-09-30T01:35:52.037 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:35:52.037 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:35:52.036991+0000 2023-09-30T01:35:52.037 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:35:52.036991+0000 2023-09-30T01:35:52.037 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2023-09-30T01:35:52.038 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:35:52.038 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a-legacy 2023-09-30T01:35:52.038 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.107:6789/0 mon.noname-f-legacy 2023-09-30T01:35:52.038 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.noname-b 2023-09-30T01:35:52.038 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.noname-g 2023-09-30T01:35:52.038 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.noname-c 2023-09-30T01:35:52.038 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.noname-h 2023-09-30T01:35:52.038 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.noname-d 2023-09-30T01:35:52.038 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.noname-i 2023-09-30T01:35:52.038 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.noname-e 2023-09-30T01:35:52.039 INFO:teuthology.orchestra.run.smithi088.stderr:9: v2:172.21.15.88:3300/0 mon.noname-a 2023-09-30T01:35:52.039 INFO:teuthology.orchestra.run.smithi088.stderr:10: v2:172.21.15.107:3300/0 mon.noname-f 2023-09-30T01:35:52.039 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:35:52.047 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:52.034+0000 7fd8320c6700 1 Processor -- start 2023-09-30T01:35:52.047 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:52.034+0000 7fd8320c6700 1 -- start start 2023-09-30T01:35:52.047 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:52.034+0000 7fd8320c6700 10 monclient: init 2023-09-30T01:35:52.047 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:52.034+0000 7fd8320c6700 10 monclient: _reopen_session rank -1 2023-09-30T01:35:52.047 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:52.034+0000 7fd8320c6700 10 monclient: _add_conns ranks=[5,1,8,4,3,10,2,6,7,9,0] 2023-09-30T01:35:52.047 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:52.034+0000 7fd8320c6700 1 --2- >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7fd82c0a3380 0x7fd82c0a48b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:35:52.047 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:52.034+0000 7fd8320c6700 10 monclient(hunting): picked mon.noname-h con 0x7fd82c0a3380 addr [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] 2023-09-30T01:35:52.047 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:52.034+0000 7fd8320c6700 10 monclient(hunting): picked mon.noname-f-legacy con 0x7fd82c09ffd0 addr v1:172.21.15.107:6789/0 2023-09-30T01:35:52.048 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:52.034+0000 7fd8320c6700 1 --2- >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7fd82c0a03a0 0x7fd82c133ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:35:52.048 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:52.034+0000 7fd8320c6700 10 monclient(hunting): picked mon.noname-e con 0x7fd82c0a03a0 addr [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] 2023-09-30T01:35:52.048 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:52.034+0000 7fd8320c6700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd82c04bb20 con 0x7fd82c09ffd0 2023-09-30T01:35:52.048 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:52.034+0000 7fd8320c6700 10 monclient(hunting): start opening mon connection 2023-09-30T01:35:52.048 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:52.034+0000 7fd8320c6700 1 -- --> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] -- mon_getmap magic: 0 v1 -- 0x7fd82c097ab0 con 0x7fd82c0a3380 2023-09-30T01:35:52.048 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:52.034+0000 7fd8320c6700 10 monclient(hunting): start opening mon connection 2023-09-30T01:35:52.048 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:52.034+0000 7fd8320c6700 1 -- --> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] -- mon_getmap magic: 0 v1 -- 0x7fd82c098480 con 0x7fd82c0a03a0 2023-09-30T01:35:52.048 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:52.034+0000 7fd8320c6700 10 monclient(hunting): _renew_subs 2023-09-30T01:35:52.048 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:52.034+0000 7fd8320c6700 10 monclient(hunting): authenticate will time out at 1024.074496s 2023-09-30T01:35:52.048 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:52.034+0000 7fd8318c5700 1 --2- >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7fd82c0a03a0 0x7fd82c133ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:35:52.049 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:52.034+0000 7fd8318c5700 1 --2- >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7fd82c0a03a0 0x7fd82c133ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3304/0 says I am v2:172.21.15.88:53804/0 (socket says 172.21.15.88:53804) 2023-09-30T01:35:52.049 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:52.034+0000 7fd8318c5700 1 -- 172.21.15.88:0/2992595993 learned_addr learned my addr 172.21.15.88:0/2992595993 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-09-30T01:35:52.049 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:52.034+0000 7fd8318c5700 10 monclient(hunting): get_auth_request con 0x7fd82c0a03a0 auth_method 0 2023-09-30T01:35:52.049 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:52.034+0000 7fd8318c5700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:35:52.049 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:52.034+0000 7fd8318c5700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:35:52.049 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:52.034+0000 7fd8318c5700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:35:52.049 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:52.034+0000 7fd8310c4700 1 --2- 172.21.15.88:0/2992595993 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7fd82c0a3380 0x7fd82c0a48b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:35:52.049 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:52.034+0000 7fd8318c5700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:35:52.049 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:52.034+0000 7fd8318c5700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:35:52.050 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:52.034+0000 7fd8318c5700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:35:52.050 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:52.034+0000 7fd8310c4700 10 monclient(hunting): get_auth_request con 0x7fd82c0a3380 auth_method 0 2023-09-30T01:35:52.050 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:52.034+0000 7fd8310c4700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:35:52.050 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:52.034+0000 7fd8310c4700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:35:52.050 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:52.034+0000 7fd8310c4700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:35:52.050 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:52.034+0000 7fd8310c4700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:35:52.050 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:52.034+0000 7fd8310c4700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:35:52.050 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:52.034+0000 7fd8310c4700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:35:52.050 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:52.034+0000 7fd8318c5700 10 monclient(hunting): handle_auth_done global_id 4688 payload 274 2023-09-30T01:35:52.051 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:52.034+0000 7fd8318c5700 1 -- 172.21.15.88:0/2992595993 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7fd82c0a3380 msgr2=0x7fd82c0a48b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:35:52.051 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:52.034+0000 7fd8318c5700 1 --2- 172.21.15.88:0/2992595993 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7fd82c0a3380 0x7fd82c0a48b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:35:52.051 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:52.034+0000 7fd8318c5700 1 -- 172.21.15.88:0/2992595993 >> v1:172.21.15.107:6789/0 conn(0x7fd82c09ffd0 legacy=0x7fd82c1333b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:35:52.051 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:52.034+0000 7fd8318c5700 10 monclient: _finish_hunting 0 2023-09-30T01:35:52.051 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:52.034+0000 7fd8318c5700 1 monclient: found mon.noname-e 2023-09-30T01:35:52.051 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:52.034+0000 7fd8318c5700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:35:52.051 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:52.034+0000 7fd8318c5700 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.88:3304/0 2023-09-30T01:35:52.051 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:52.034+0000 7fd8318c5700 1 -- 172.21.15.88:0/2992595993 --> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd82c0a3750 con 0x7fd82c0a03a0 2023-09-30T01:35:52.051 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:52.034+0000 7fd8318c5700 1 --2- 172.21.15.88:0/2992595993 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7fd82c0a03a0 0x7fd82c133ac0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fd824009f80 tx=0x7fd824009840 comp rx=0 tx=0).ready entity=mon.8 client_cookie=69b5a7b003533d8d server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:35:52.051 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:52.034+0000 7fd823fff700 1 -- 172.21.15.88:0/2992595993 <== mon.8 v2:172.21.15.88:3304/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7fd82400e070 con 0x7fd82c0a03a0 2023-09-30T01:35:52.051 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:52.034+0000 7fd823fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:35:52.051 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:52.038+0000 7fd823fff700 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2023-09-30T01:35:52.051 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:52.038+0000 7fd823fff700 10 monclient: dump: 2023-09-30T01:35:52.052 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:35:52.052 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:35:52.052 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:35:52.052 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:35:52.052 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:35:52.052 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:35:52.052 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:35:52.052 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:35:52.052 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:35:52.052 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:35:52.052 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:35:52.052 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:35:52.052 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:35:52.053 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:35:52.053 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:35:52.053 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:35:52.053 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:52.038+0000 7fd823fff700 10 monclient: _finish_auth 0 2023-09-30T01:35:52.053 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:52.038+0000 7fd823fff700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:35:52.053 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:52.038+0000 7fd823fff700 1 -- 172.21.15.88:0/2992595993 <== mon.8 v2:172.21.15.88:3304/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd824007400 con 0x7fd82c0a03a0 2023-09-30T01:35:52.053 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:52.038+0000 7fd8320c6700 5 monclient: authenticate success, global_id 4688 2023-09-30T01:35:52.053 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:52.038+0000 7fd8320c6700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:35:52.053 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:52.038+0000 7fd823fff700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:35:52.053 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:52.038+0000 7fd823fff700 1 -- 172.21.15.88:0/2992595993 <== mon.8 v2:172.21.15.88:3304/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7fd8240092e0 con 0x7fd82c0a03a0 2023-09-30T01:35:52.053 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:52.038+0000 7fd8320c6700 10 monclient: get_monmap_and_config success 2023-09-30T01:35:52.053 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:52.038+0000 7fd8320c6700 4 set_mon_vals no callback set 2023-09-30T01:35:52.054 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:52.038+0000 7fd823fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:35:52.054 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:52.038+0000 7fd823fff700 10 monclient: got monmap 1 from mon.e (according to old e1) 2023-09-30T01:35:52.054 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:52.038+0000 7fd823fff700 10 monclient: dump: 2023-09-30T01:35:52.054 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:35:52.054 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:35:52.054 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:35:52.054 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:35:52.054 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:35:52.055 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:35:52.055 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:35:52.055 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:35:52.055 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:35:52.055 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:35:52.055 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:35:52.055 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:35:52.055 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:35:52.055 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:35:52.055 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:35:52.055 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:35:52.055 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:52.038+0000 7fd8320c6700 10 monclient: shutdown 2023-09-30T01:35:52.055 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:52.038+0000 7fd8320c6700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:35:52.056 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:52.038+0000 7fd8320c6700 1 -- 172.21.15.88:0/2992595993 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7fd82c0a03a0 msgr2=0x7fd82c133ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:35:52.056 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:52.038+0000 7fd8320c6700 1 --2- 172.21.15.88:0/2992595993 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7fd82c0a03a0 0x7fd82c133ac0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fd824009f80 tx=0x7fd824009840 comp rx=0 tx=0).stop 2023-09-30T01:35:52.056 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:52.038+0000 7fd8320c6700 1 -- 172.21.15.88:0/2992595993 shutdown_connections 2023-09-30T01:35:52.056 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:52.038+0000 7fd8320c6700 1 --2- 172.21.15.88:0/2992595993 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7fd82c0a03a0 0x7fd82c133ac0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:35:52.056 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:52.038+0000 7fd8320c6700 1 --2- 172.21.15.88:0/2992595993 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7fd82c0a3380 0x7fd82c0a48b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:35:52.056 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:52.038+0000 7fd8320c6700 1 -- 172.21.15.88:0/2992595993 >> 172.21.15.88:0/2992595993 conn(0x7fd82c06e5e0 msgr2=0x7fd82c06e9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:35:52.056 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:52.038+0000 7fd8320c6700 1 -- 172.21.15.88:0/2992595993 shutdown_connections 2023-09-30T01:35:52.056 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:52.038+0000 7fd8320c6700 1 -- 172.21.15.88:0/2992595993 wait complete. 2023-09-30T01:35:52.056 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:52.038+0000 7fd8320c6700 10 monclient: build_initial_monmap 2023-09-30T01:35:52.056 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:52.038+0000 7fd8320c6700 10 monclient: monmap: 2023-09-30T01:35:52.056 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2023-09-30T01:35:52.056 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:35:52.057 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 0.000000 2023-09-30T01:35:52.057 INFO:teuthology.orchestra.run.smithi088.stderr:created 0.000000 2023-09-30T01:35:52.057 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2023-09-30T01:35:52.057 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 1 2023-09-30T01:35:52.057 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.noname-a 2023-09-30T01:35:52.057 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.noname-f 2023-09-30T01:35:52.057 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.noname-b 2023-09-30T01:35:52.057 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.noname-g 2023-09-30T01:35:52.057 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.noname-c 2023-09-30T01:35:52.057 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.noname-h 2023-09-30T01:35:52.057 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.noname-d 2023-09-30T01:35:52.057 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.noname-i 2023-09-30T01:35:52.057 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.noname-e 2023-09-30T01:35:52.058 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:35:52.058 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:52.038+0000 7fd8320c6700 1 Processor -- start 2023-09-30T01:35:52.058 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:52.038+0000 7fd8320c6700 1 -- start start 2023-09-30T01:35:52.058 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:52.038+0000 7fd8320c6700 10 monclient: init 2023-09-30T01:35:52.058 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:52.038+0000 7fd8320c6700 10 monclient: _reopen_session rank -1 2023-09-30T01:35:52.058 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:52.038+0000 7fd8320c6700 10 monclient: _add_conns ranks=[4,6,5,3,7,2,8,1,0] 2023-09-30T01:35:52.058 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:52.038+0000 7fd8320c6700 1 --2- >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7fd82c09ffd0 0x7fd82c11cf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:35:52.058 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:52.038+0000 7fd8320c6700 10 monclient(hunting): picked mon.noname-c con 0x7fd82c09ffd0 addr [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] 2023-09-30T01:35:52.058 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:52.038+0000 7fd8320c6700 1 --2- >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7fd82c0a03a0 0x7fd82c11d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:35:52.058 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:52.038+0000 7fd8320c6700 10 monclient(hunting): picked mon.noname-d con 0x7fd82c0a03a0 addr [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] 2023-09-30T01:35:52.058 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:52.038+0000 7fd8320c6700 1 --2- >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7fd82c0a3380 0x7fd82c11d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:35:52.058 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:52.038+0000 7fd8320c6700 10 monclient(hunting): picked mon.noname-h con 0x7fd82c0a3380 addr [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] 2023-09-30T01:35:52.058 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:52.038+0000 7fd8320c6700 10 monclient(hunting): start opening mon connection 2023-09-30T01:35:52.059 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:52.038+0000 7fd8320c6700 1 -- --> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] -- mon_getmap magic: 0 v1 -- 0x7fd82c0a0ac0 con 0x7fd82c09ffd0 2023-09-30T01:35:52.059 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:52.038+0000 7fd8320c6700 10 monclient(hunting): start opening mon connection 2023-09-30T01:35:52.059 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:52.038+0000 7fd8320c6700 1 -- --> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] -- mon_getmap magic: 0 v1 -- 0x7fd82c097ce0 con 0x7fd82c0a3380 2023-09-30T01:35:52.059 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:52.038+0000 7fd8320c6700 10 monclient(hunting): start opening mon connection 2023-09-30T01:35:52.059 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:52.038+0000 7fd8320c6700 1 -- --> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] -- mon_getmap magic: 0 v1 -- 0x7fd82c11f150 con 0x7fd82c0a03a0 2023-09-30T01:35:52.059 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:52.038+0000 7fd8320c6700 10 monclient(hunting): _renew_subs 2023-09-30T01:35:52.059 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:52.038+0000 7fd8320c6700 10 monclient(hunting): authenticate will time out at 1024.078929s 2023-09-30T01:35:52.059 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:52.038+0000 7fd8310c4700 1 --2- >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7fd82c09ffd0 0x7fd82c11cf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:35:52.059 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:52.038+0000 7fd8308c3700 1 --2- >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7fd82c0a03a0 0x7fd82c11d4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:35:52.059 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:52.038+0000 7fd8310c4700 1 --2- >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7fd82c09ffd0 0x7fd82c11cf60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3302/0 says I am v2:172.21.15.88:48926/0 (socket says 172.21.15.88:48926) 2023-09-30T01:35:52.059 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:52.038+0000 7fd8310c4700 1 -- 172.21.15.88:0/2627477248 learned_addr learned my addr 172.21.15.88:0/2627477248 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-09-30T01:35:52.059 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:52.038+0000 7fd8310c4700 10 monclient(hunting): get_auth_request con 0x7fd82c09ffd0 auth_method 0 2023-09-30T01:35:52.060 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:52.038+0000 7fd8310c4700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:35:52.060 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:52.038+0000 7fd8310c4700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:35:52.060 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:52.038+0000 7fd8310c4700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:35:52.060 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:52.038+0000 7fd8318c5700 1 --2- 172.21.15.88:0/2627477248 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7fd82c0a3380 0x7fd82c11d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:35:52.060 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:52.038+0000 7fd8308c3700 10 monclient(hunting): get_auth_request con 0x7fd82c0a03a0 auth_method 0 2023-09-30T01:35:52.060 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:52.038+0000 7fd8308c3700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:35:52.060 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:52.038+0000 7fd8308c3700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:35:52.060 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:52.038+0000 7fd8308c3700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:35:52.060 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:52.038+0000 7fd8318c5700 10 monclient(hunting): get_auth_request con 0x7fd82c0a3380 auth_method 0 2023-09-30T01:35:52.060 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:52.038+0000 7fd8318c5700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:35:52.060 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:52.038+0000 7fd8318c5700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:35:52.060 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:52.038+0000 7fd8318c5700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:35:52.060 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:52.038+0000 7fd8310c4700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:35:52.061 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:52.038+0000 7fd8310c4700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:35:52.061 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:52.038+0000 7fd8310c4700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:35:52.061 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:52.038+0000 7fd8318c5700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:35:52.061 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:52.038+0000 7fd8318c5700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:35:52.061 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:52.038+0000 7fd8318c5700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:35:52.061 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:52.038+0000 7fd8308c3700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:35:52.061 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:52.038+0000 7fd8308c3700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:35:52.061 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:52.038+0000 7fd8308c3700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:35:52.061 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:52.038+0000 7fd8310c4700 10 monclient(hunting): handle_auth_done global_id 4648 payload 824 2023-09-30T01:35:52.061 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:52.038+0000 7fd8310c4700 1 -- 172.21.15.88:0/2627477248 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7fd82c0a03a0 msgr2=0x7fd82c11d4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:35:52.061 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:52.038+0000 7fd8310c4700 1 --2- 172.21.15.88:0/2627477248 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7fd82c0a03a0 0x7fd82c11d4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:35:52.061 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:52.038+0000 7fd8310c4700 1 -- 172.21.15.88:0/2627477248 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7fd82c0a3380 msgr2=0x7fd82c11d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:35:52.062 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:52.038+0000 7fd8310c4700 1 --2- 172.21.15.88:0/2627477248 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7fd82c0a3380 0x7fd82c11d9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:35:52.062 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:52.038+0000 7fd8310c4700 10 monclient: _finish_hunting 0 2023-09-30T01:35:52.062 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:52.038+0000 7fd8310c4700 1 monclient: found mon.noname-c 2023-09-30T01:35:52.062 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:52.038+0000 7fd8310c4700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:35:52.062 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:52.038+0000 7fd8310c4700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.88:3302/0 2023-09-30T01:35:52.062 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:52.038+0000 7fd8310c4700 1 -- 172.21.15.88:0/2627477248 --> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd824009ae0 con 0x7fd82c09ffd0 2023-09-30T01:35:52.062 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:52.042+0000 7fd8310c4700 1 --2- 172.21.15.88:0/2627477248 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7fd82c09ffd0 0x7fd82c11cf60 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fd81c00e960 tx=0x7fd81c0091a0 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:35:52.062 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:52.042+0000 7fd8217fa700 1 -- 172.21.15.88:0/2627477248 <== mon.4 v2:172.21.15.88:3302/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7fd81c013b00 con 0x7fd82c09ffd0 2023-09-30T01:35:52.062 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:52.042+0000 7fd8217fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:35:52.062 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:52.042+0000 7fd8217fa700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T01:35:52.062 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:52.042+0000 7fd8217fa700 10 monclient: dump: 2023-09-30T01:35:52.062 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:35:52.063 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:35:52.063 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:35:52.063 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:35:52.063 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:35:52.063 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:35:52.063 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:35:52.063 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:35:52.063 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:35:52.063 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:35:52.063 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:35:52.063 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:35:52.063 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:35:52.063 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:35:52.064 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:35:52.064 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:35:52.064 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:52.042+0000 7fd8217fa700 10 monclient: _finish_auth 0 2023-09-30T01:35:52.064 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:52.042+0000 7fd8217fa700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:35:52.064 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:52.042+0000 7fd8320c6700 5 monclient: authenticate success, global_id 4648 2023-09-30T01:35:52.064 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:52.042+0000 7fd8320c6700 10 monclient: _renew_subs 2023-09-30T01:35:52.064 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:52.042+0000 7fd8320c6700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.88:3302/0 2023-09-30T01:35:52.064 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:52.042+0000 7fd8217fa700 1 -- 172.21.15.88:0/2627477248 <== mon.4 v2:172.21.15.88:3302/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd81c013ca0 con 0x7fd82c09ffd0 2023-09-30T01:35:52.064 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:52.042+0000 7fd8320c6700 1 -- 172.21.15.88:0/2627477248 --> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd82c09e930 con 0x7fd82c09ffd0 2023-09-30T01:35:52.064 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:52.042+0000 7fd8217fa700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:35:52.064 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:52.042+0000 7fd8217fa700 1 -- 172.21.15.88:0/2627477248 <== mon.4 v2:172.21.15.88:3302/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7fd81c013e10 con 0x7fd82c09ffd0 2023-09-30T01:35:52.064 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:52.042+0000 7fd8217fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:35:52.064 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:52.042+0000 7fd8217fa700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T01:35:52.065 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:52.042+0000 7fd822ffd700 4 set_mon_vals no callback set 2023-09-30T01:35:52.065 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:52.042+0000 7fd8217fa700 10 monclient: dump: 2023-09-30T01:35:52.065 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:35:52.065 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:35:52.065 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:35:52.065 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:35:52.065 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:35:52.065 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:35:52.065 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:35:52.065 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:35:52.065 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:35:52.065 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:35:52.065 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:35:52.066 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:35:52.066 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:35:52.066 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:35:52.066 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:35:52.066 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:35:52.066 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:52.042+0000 7fd8320c6700 10 monclient: _renew_subs 2023-09-30T01:35:52.066 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:52.042+0000 7fd8320c6700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.88:3302/0 2023-09-30T01:35:52.066 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:52.042+0000 7fd8320c6700 1 -- 172.21.15.88:0/2627477248 --> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd82c120370 con 0x7fd82c09ffd0 2023-09-30T01:35:52.066 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:52.042+0000 7fd8217fa700 1 -- 172.21.15.88:0/2627477248 <== mon.4 v2:172.21.15.88:3302/0 4 ==== mgrmap(e 5) v1 ==== 40144+0+0 (secure 0 0 0) 0x7fd81c013680 con 0x7fd82c09ffd0 2023-09-30T01:35:52.066 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:52.042+0000 7fd8320c6700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:35:52.066 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:52.042+0000 7fd8320c6700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:35:52.066 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:52.042+0000 7fd8320c6700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.88:3302/0 2023-09-30T01:35:52.066 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:52.042+0000 7fd8320c6700 1 -- 172.21.15.88:0/2627477248 --> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd82c11f5f0 con 0x7fd82c09ffd0 2023-09-30T01:35:52.067 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:52.042+0000 7fd8217fa700 1 --2- 172.21.15.88:0/2627477248 >> v2:172.21.15.107:6812/29926 conn(0x7fd810033360 0x7fd810035810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:35:52.067 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:52.042+0000 7fd8217fa700 1 -- 172.21.15.88:0/2627477248 <== mon.4 v2:172.21.15.88:3302/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 3502+0+0 (secure 0 0 0) 0x7fd81c0438d0 con 0x7fd82c09ffd0 2023-09-30T01:35:52.067 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:52.042+0000 7fd8308c3700 1 --2- 172.21.15.88:0/2627477248 >> v2:172.21.15.107:6812/29926 conn(0x7fd810033360 0x7fd810035810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:35:52.067 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:52.042+0000 7fd8308c3700 10 monclient: get_auth_request con 0x7fd810033360 auth_method 0 2023-09-30T01:35:52.067 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:52.046+0000 7fd8308c3700 1 --2- 172.21.15.88:0/2627477248 >> v2:172.21.15.107:6812/29926 conn(0x7fd810033360 0x7fd810035810 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fd828003aa0 tx=0x7fd828000bc0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:35:52.068 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:52.050+0000 7fd8217fa700 1 -- 172.21.15.88:0/2627477248 <== mon.4 v2:172.21.15.88:3302/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fd81c013930 con 0x7fd82c09ffd0 2023-09-30T01:35:52.068 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:52.050+0000 7fd8217fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:35:52.068 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:52.050+0000 7fd8217fa700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:35:52.245 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:52.242+0000 7fd8320c6700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T01:35:52.246 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:52.242+0000 7fd8320c6700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:35:52.246 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:52.242+0000 7fd8320c6700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.88:3302/0 2023-09-30T01:35:52.246 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:52.242+0000 7fd8320c6700 1 -- 172.21.15.88:0/2627477248 --> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fd82c0710d0 con 0x7fd82c09ffd0 2023-09-30T01:35:52.246 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:52.242+0000 7fd8217fa700 1 -- 172.21.15.88:0/2627477248 <== mon.4 v2:172.21.15.88:3302/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7fd81c01a360 con 0x7fd82c09ffd0 2023-09-30T01:35:52.246 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:52.242+0000 7fd8217fa700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:35:52.246 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:52.242+0000 7fd8217fa700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:35:52.247 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-09-30T01:35:52.247 INFO:teuthology.orchestra.run.smithi088.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:35:52.249 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:52.246+0000 7fd8320c6700 1 -- 172.21.15.88:0/2627477248 >> v2:172.21.15.107:6812/29926 conn(0x7fd810033360 msgr2=0x7fd810035810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:35:52.249 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:52.246+0000 7fd8320c6700 1 --2- 172.21.15.88:0/2627477248 >> v2:172.21.15.107:6812/29926 conn(0x7fd810033360 0x7fd810035810 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fd828003aa0 tx=0x7fd828000bc0 comp rx=0 tx=0).stop 2023-09-30T01:35:52.249 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:52.246+0000 7fd8320c6700 10 monclient: shutdown 2023-09-30T01:35:52.249 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:52.246+0000 7fd8320c6700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:35:52.249 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:52.246+0000 7fd8320c6700 1 -- 172.21.15.88:0/2627477248 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7fd82c09ffd0 msgr2=0x7fd82c11cf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:35:52.249 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:52.246+0000 7fd8320c6700 1 --2- 172.21.15.88:0/2627477248 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7fd82c09ffd0 0x7fd82c11cf60 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fd81c00e960 tx=0x7fd81c0091a0 comp rx=0 tx=0).stop 2023-09-30T01:35:52.249 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:52.246+0000 7fd8320c6700 1 -- 172.21.15.88:0/2627477248 shutdown_connections 2023-09-30T01:35:52.250 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:52.246+0000 7fd8320c6700 1 --2- 172.21.15.88:0/2627477248 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7fd82c0a3380 0x7fd82c11d9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:35:52.250 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:52.246+0000 7fd8320c6700 1 --2- 172.21.15.88:0/2627477248 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7fd82c0a03a0 0x7fd82c11d4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:35:52.250 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:52.246+0000 7fd8320c6700 1 --2- 172.21.15.88:0/2627477248 >> v2:172.21.15.107:6812/29926 conn(0x7fd810033360 0x7fd810035810 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:35:52.250 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:52.246+0000 7fd8320c6700 1 --2- 172.21.15.88:0/2627477248 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7fd82c09ffd0 0x7fd82c11cf60 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:35:52.250 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:52.246+0000 7fd8320c6700 1 -- 172.21.15.88:0/2627477248 >> 172.21.15.88:0/2627477248 conn(0x7fd82c06e5e0 msgr2=0x7fd82c131b10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:35:52.250 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:52.246+0000 7fd8320c6700 1 -- 172.21.15.88:0/2627477248 shutdown_connections 2023-09-30T01:35:52.250 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:52.246+0000 7fd8320c6700 1 -- 172.21.15.88:0/2627477248 wait complete. 2023-09-30T01:35:55.261 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:35:55.375 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:55.370+0000 7fbdf8eaf700 10 monclient: get_monmap_and_config 2023-09-30T01:35:55.376 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:55.370+0000 7fbdf8eaf700 10 monclient: build_initial_monmap 2023-09-30T01:35:55.376 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:55.370+0000 7fbdf8eaf700 10 monclient: monmap: 2023-09-30T01:35:55.376 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2023-09-30T01:35:55.376 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:35:55.376 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:35:55.375569+0000 2023-09-30T01:35:55.376 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:35:55.375569+0000 2023-09-30T01:35:55.376 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2023-09-30T01:35:55.377 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:35:55.377 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a-legacy 2023-09-30T01:35:55.377 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.107:6789/0 mon.noname-f-legacy 2023-09-30T01:35:55.377 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.noname-b 2023-09-30T01:35:55.377 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.noname-g 2023-09-30T01:35:55.377 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.noname-c 2023-09-30T01:35:55.377 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.noname-h 2023-09-30T01:35:55.377 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.noname-d 2023-09-30T01:35:55.378 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.noname-i 2023-09-30T01:35:55.378 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.noname-e 2023-09-30T01:35:55.378 INFO:teuthology.orchestra.run.smithi088.stderr:9: v2:172.21.15.88:3300/0 mon.noname-a 2023-09-30T01:35:55.378 INFO:teuthology.orchestra.run.smithi088.stderr:10: v2:172.21.15.107:3300/0 mon.noname-f 2023-09-30T01:35:55.378 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:35:55.388 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:55.374+0000 7fbdf8eaf700 1 Processor -- start 2023-09-30T01:35:55.388 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:55.374+0000 7fbdf8eaf700 1 -- start start 2023-09-30T01:35:55.388 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:55.374+0000 7fbdf8eaf700 10 monclient: init 2023-09-30T01:35:55.388 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:55.374+0000 7fbdf8eaf700 10 monclient: _reopen_session rank -1 2023-09-30T01:35:55.388 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:55.374+0000 7fbdf8eaf700 10 monclient: _add_conns ranks=[5,6,9,8,2,10,7,3,1,0,4] 2023-09-30T01:35:55.388 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:55.374+0000 7fbdf8eaf700 1 --2- >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7fbdf4097a40 0x7fbdf4132700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:35:55.388 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:55.374+0000 7fbdf8eaf700 10 monclient(hunting): picked mon.noname-h con 0x7fbdf4097a40 addr [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] 2023-09-30T01:35:55.388 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:55.374+0000 7fbdf8eaf700 1 --2- >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7fbdf41311c0 0x7fbdf4132c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:35:55.389 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:55.374+0000 7fbdf8eaf700 10 monclient(hunting): picked mon.noname-d con 0x7fbdf41311c0 addr [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] 2023-09-30T01:35:55.389 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:55.374+0000 7fbdf8eaf700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fbdf41332e0 0x7fbdf4133710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:35:55.389 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:55.374+0000 7fbdf8eaf700 10 monclient(hunting): picked mon.noname-a con 0x7fbdf41332e0 addr v2:172.21.15.88:3300/0 2023-09-30T01:35:55.389 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:55.374+0000 7fbdf8eaf700 10 monclient(hunting): start opening mon connection 2023-09-30T01:35:55.389 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:55.374+0000 7fbdf8eaf700 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbdf4131590 con 0x7fbdf41332e0 2023-09-30T01:35:55.389 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:55.374+0000 7fbdf8eaf700 10 monclient(hunting): start opening mon connection 2023-09-30T01:35:55.389 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:55.374+0000 7fbdf8eaf700 1 -- --> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] -- mon_getmap magic: 0 v1 -- 0x7fbdf409a5d0 con 0x7fbdf4097a40 2023-09-30T01:35:55.389 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:55.374+0000 7fbdf8eaf700 10 monclient(hunting): start opening mon connection 2023-09-30T01:35:55.390 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:55.374+0000 7fbdf8eaf700 1 -- --> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] -- mon_getmap magic: 0 v1 -- 0x7fbdf4097e10 con 0x7fbdf41311c0 2023-09-30T01:35:55.390 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:55.374+0000 7fbdf8eaf700 10 monclient(hunting): _renew_subs 2023-09-30T01:35:55.390 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:55.374+0000 7fbdf8eaf700 10 monclient(hunting): authenticate will time out at 1027.413103s 2023-09-30T01:35:55.390 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:55.374+0000 7fbdf2ffd700 1 --2- >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7fbdf41311c0 0x7fbdf4132c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:35:55.390 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:55.374+0000 7fbdf3fff700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fbdf41332e0 0x7fbdf4133710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:35:55.390 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:55.374+0000 7fbdf2ffd700 1 --2- >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7fbdf41311c0 0x7fbdf4132c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3303/0 says I am v2:172.21.15.88:60268/0 (socket says 172.21.15.88:60268) 2023-09-30T01:35:55.390 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:55.374+0000 7fbdf2ffd700 1 -- 172.21.15.88:0/687386657 learned_addr learned my addr 172.21.15.88:0/687386657 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-09-30T01:35:55.390 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:55.374+0000 7fbdf2ffd700 10 monclient(hunting): get_auth_request con 0x7fbdf41311c0 auth_method 0 2023-09-30T01:35:55.390 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:55.374+0000 7fbdf2ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:35:55.390 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:55.374+0000 7fbdf2ffd700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:35:55.391 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:55.374+0000 7fbdf2ffd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:35:55.391 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:55.374+0000 7fbdf3fff700 10 monclient(hunting): get_auth_request con 0x7fbdf41332e0 auth_method 0 2023-09-30T01:35:55.391 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:55.374+0000 7fbdf3fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:35:55.391 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:55.374+0000 7fbdf3fff700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:35:55.391 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:55.374+0000 7fbdf3fff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:35:55.391 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:55.374+0000 7fbdf2ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:35:55.391 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:55.374+0000 7fbdf2ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:35:55.391 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:55.374+0000 7fbdf2ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:35:55.391 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:55.374+0000 7fbdf37fe700 1 --2- 172.21.15.88:0/687386657 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7fbdf4097a40 0x7fbdf4132700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:35:55.391 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:55.374+0000 7fbdf37fe700 10 monclient(hunting): get_auth_request con 0x7fbdf4097a40 auth_method 0 2023-09-30T01:35:55.391 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:55.374+0000 7fbdf37fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:35:55.392 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:55.374+0000 7fbdf37fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:35:55.392 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:55.374+0000 7fbdf37fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:35:55.392 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:55.374+0000 7fbdf3fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:35:55.392 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:55.374+0000 7fbdf3fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:35:55.392 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:55.374+0000 7fbdf3fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:35:55.392 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:55.374+0000 7fbdf2ffd700 10 monclient(hunting): handle_auth_done global_id 4551 payload 274 2023-09-30T01:35:55.392 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:55.374+0000 7fbdf2ffd700 1 -- 172.21.15.88:0/687386657 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7fbdf4097a40 msgr2=0x7fbdf4132700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:35:55.392 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:55.374+0000 7fbdf2ffd700 1 --2- 172.21.15.88:0/687386657 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7fbdf4097a40 0x7fbdf4132700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:35:55.392 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:55.374+0000 7fbdf2ffd700 1 -- 172.21.15.88:0/687386657 >> v2:172.21.15.88:3300/0 conn(0x7fbdf41332e0 msgr2=0x7fbdf4133710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:35:55.392 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:55.374+0000 7fbdf2ffd700 1 --2- 172.21.15.88:0/687386657 >> v2:172.21.15.88:3300/0 conn(0x7fbdf41332e0 0x7fbdf4133710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:35:55.393 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:55.374+0000 7fbdf2ffd700 10 monclient: _finish_hunting 0 2023-09-30T01:35:55.393 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:55.374+0000 7fbdf2ffd700 1 monclient: found mon.noname-d 2023-09-30T01:35:55.393 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:55.374+0000 7fbdf2ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:35:55.393 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:55.374+0000 7fbdf2ffd700 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.88:3303/0 2023-09-30T01:35:55.393 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:55.374+0000 7fbdf2ffd700 1 -- 172.21.15.88:0/687386657 --> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbdf404bb20 con 0x7fbdf41311c0 2023-09-30T01:35:55.393 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:55.374+0000 7fbdf3fff700 1 --2- 172.21.15.88:0/687386657 >> v2:172.21.15.88:3300/0 conn(0x7fbdf41332e0 0x7fbdf4133710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:35:55.393 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:55.374+0000 7fbdf37fe700 1 --2- 172.21.15.88:0/687386657 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7fbdf4097a40 0x7fbdf4132700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-30T01:35:55.394 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:55.374+0000 7fbdf2ffd700 1 --2- 172.21.15.88:0/687386657 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7fbdf41311c0 0x7fbdf4132c40 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fbdec009f20 tx=0x7fbdec009840 comp rx=0 tx=0).ready entity=mon.6 client_cookie=b495dffa64f2b456 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:35:55.394 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:55.374+0000 7fbdf27fc700 1 -- 172.21.15.88:0/687386657 <== mon.6 v2:172.21.15.88:3303/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7fbdec00e070 con 0x7fbdf41311c0 2023-09-30T01:35:55.394 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:55.374+0000 7fbdf27fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:35:55.394 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:55.374+0000 7fbdf27fc700 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2023-09-30T01:35:55.394 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:55.374+0000 7fbdf27fc700 10 monclient: dump: 2023-09-30T01:35:55.394 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:35:55.394 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:35:55.394 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:35:55.395 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:35:55.395 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:35:55.395 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:35:55.395 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:35:55.395 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:35:55.395 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:35:55.395 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:35:55.395 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:35:55.395 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:35:55.395 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:35:55.395 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:35:55.396 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:35:55.396 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:35:55.396 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:55.374+0000 7fbdf27fc700 10 monclient: _finish_auth 0 2023-09-30T01:35:55.396 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:55.374+0000 7fbdf27fc700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:35:55.396 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:55.374+0000 7fbdf27fc700 1 -- 172.21.15.88:0/687386657 <== mon.6 v2:172.21.15.88:3303/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbdec00be50 con 0x7fbdf41311c0 2023-09-30T01:35:55.396 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:55.374+0000 7fbdf8eaf700 5 monclient: authenticate success, global_id 4551 2023-09-30T01:35:55.396 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:55.374+0000 7fbdf8eaf700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:35:55.396 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:55.374+0000 7fbdf27fc700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:35:55.396 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:55.374+0000 7fbdf8eaf700 10 monclient: get_monmap_and_config success 2023-09-30T01:35:55.396 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:55.374+0000 7fbdf8eaf700 4 set_mon_vals no callback set 2023-09-30T01:35:55.397 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:55.374+0000 7fbdf27fc700 1 -- 172.21.15.88:0/687386657 <== mon.6 v2:172.21.15.88:3303/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7fbdec0092e0 con 0x7fbdf41311c0 2023-09-30T01:35:55.397 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:55.374+0000 7fbdf27fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:35:55.397 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:55.374+0000 7fbdf27fc700 10 monclient: got monmap 1 from mon.d (according to old e1) 2023-09-30T01:35:55.397 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:55.374+0000 7fbdf27fc700 10 monclient: dump: 2023-09-30T01:35:55.397 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:35:55.397 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:35:55.397 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:35:55.397 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:35:55.397 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:35:55.397 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:35:55.397 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:35:55.398 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:35:55.398 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:35:55.398 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:35:55.398 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:35:55.398 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:35:55.398 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:35:55.398 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:35:55.398 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:35:55.398 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:35:55.398 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:55.374+0000 7fbdf8eaf700 10 monclient: shutdown 2023-09-30T01:35:55.399 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:55.374+0000 7fbdf8eaf700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:35:55.399 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:55.374+0000 7fbdf8eaf700 1 -- 172.21.15.88:0/687386657 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7fbdf41311c0 msgr2=0x7fbdf4132c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:35:55.399 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:55.374+0000 7fbdf8eaf700 1 --2- 172.21.15.88:0/687386657 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7fbdf41311c0 0x7fbdf4132c40 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fbdec009f20 tx=0x7fbdec009840 comp rx=0 tx=0).stop 2023-09-30T01:35:55.399 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:55.374+0000 7fbdf8eaf700 1 -- 172.21.15.88:0/687386657 shutdown_connections 2023-09-30T01:35:55.399 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:55.374+0000 7fbdf8eaf700 1 --2- 172.21.15.88:0/687386657 >> v2:172.21.15.88:3300/0 conn(0x7fbdf41332e0 0x7fbdf4133710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:35:55.399 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:55.374+0000 7fbdf8eaf700 1 --2- 172.21.15.88:0/687386657 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7fbdf41311c0 0x7fbdf4132c40 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:35:55.399 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:55.374+0000 7fbdf8eaf700 1 --2- 172.21.15.88:0/687386657 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7fbdf4097a40 0x7fbdf4132700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:35:55.399 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:55.374+0000 7fbdf8eaf700 1 -- 172.21.15.88:0/687386657 >> 172.21.15.88:0/687386657 conn(0x7fbdf406e5e0 msgr2=0x7fbdf406e9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:35:55.399 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:55.374+0000 7fbdf8eaf700 1 -- 172.21.15.88:0/687386657 shutdown_connections 2023-09-30T01:35:55.399 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:55.374+0000 7fbdf8eaf700 1 -- 172.21.15.88:0/687386657 wait complete. 2023-09-30T01:35:55.399 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:55.378+0000 7fbdf8eaf700 10 monclient: build_initial_monmap 2023-09-30T01:35:55.399 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:55.378+0000 7fbdf8eaf700 10 monclient: monmap: 2023-09-30T01:35:55.400 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2023-09-30T01:35:55.400 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:35:55.400 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 0.000000 2023-09-30T01:35:55.400 INFO:teuthology.orchestra.run.smithi088.stderr:created 0.000000 2023-09-30T01:35:55.400 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2023-09-30T01:35:55.400 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 1 2023-09-30T01:35:55.400 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.noname-a 2023-09-30T01:35:55.400 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.noname-f 2023-09-30T01:35:55.400 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.noname-b 2023-09-30T01:35:55.400 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.noname-g 2023-09-30T01:35:55.400 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.noname-c 2023-09-30T01:35:55.400 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.noname-h 2023-09-30T01:35:55.401 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.noname-d 2023-09-30T01:35:55.401 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.noname-i 2023-09-30T01:35:55.401 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.noname-e 2023-09-30T01:35:55.401 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:35:55.401 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:55.378+0000 7fbdf8eaf700 1 Processor -- start 2023-09-30T01:35:55.401 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:55.378+0000 7fbdf8eaf700 1 -- start start 2023-09-30T01:35:55.401 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:55.378+0000 7fbdf8eaf700 10 monclient: init 2023-09-30T01:35:55.401 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:55.378+0000 7fbdf8eaf700 10 monclient: _reopen_session rank -1 2023-09-30T01:35:55.401 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:55.378+0000 7fbdf8eaf700 10 monclient: _add_conns ranks=[0,6,3,1,4,8,2,5,7] 2023-09-30T01:35:55.401 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:55.378+0000 7fbdf8eaf700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbdf4097a40 0x7fbdf40a1eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:35:55.401 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:55.378+0000 7fbdf8eaf700 10 monclient(hunting): picked mon.noname-a con 0x7fbdf4097a40 addr [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] 2023-09-30T01:35:55.402 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:55.378+0000 7fbdf8eaf700 1 --2- >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7fbdf41311c0 0x7fbdf40a23f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:35:55.402 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:55.378+0000 7fbdf8eaf700 10 monclient(hunting): picked mon.noname-d con 0x7fbdf41311c0 addr [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] 2023-09-30T01:35:55.402 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:55.378+0000 7fbdf8eaf700 1 --2- >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7fbdf41332e0 0x7fbdf40a2930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:35:55.402 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:55.378+0000 7fbdf8eaf700 10 monclient(hunting): picked mon.noname-g con 0x7fbdf41332e0 addr [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] 2023-09-30T01:35:55.402 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:55.378+0000 7fbdf8eaf700 10 monclient(hunting): start opening mon connection 2023-09-30T01:35:55.402 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:55.378+0000 7fbdf8eaf700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbdf41303d0 con 0x7fbdf4097a40 2023-09-30T01:35:55.402 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:55.378+0000 7fbdf8eaf700 10 monclient(hunting): start opening mon connection 2023-09-30T01:35:55.402 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:55.378+0000 7fbdf8eaf700 1 -- --> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbdf4099c30 con 0x7fbdf41332e0 2023-09-30T01:35:55.402 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:55.378+0000 7fbdf8eaf700 10 monclient(hunting): start opening mon connection 2023-09-30T01:35:55.402 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:55.378+0000 7fbdf8eaf700 1 -- --> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] -- mon_getmap magic: 0 v1 -- 0x7fbdf40996b0 con 0x7fbdf41311c0 2023-09-30T01:35:55.402 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:55.378+0000 7fbdf8eaf700 10 monclient(hunting): _renew_subs 2023-09-30T01:35:55.403 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:55.378+0000 7fbdf8eaf700 10 monclient(hunting): authenticate will time out at 1027.417837s 2023-09-30T01:35:55.403 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:55.378+0000 7fbdf2ffd700 1 --2- >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7fbdf41311c0 0x7fbdf40a23f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:35:55.403 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:55.378+0000 7fbdf2ffd700 1 --2- >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7fbdf41311c0 0x7fbdf40a23f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3303/0 says I am v2:172.21.15.88:60278/0 (socket says 172.21.15.88:60278) 2023-09-30T01:35:55.403 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:55.378+0000 7fbdf2ffd700 1 -- 172.21.15.88:0/1983933764 learned_addr learned my addr 172.21.15.88:0/1983933764 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-09-30T01:35:55.403 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:55.378+0000 7fbdf2ffd700 10 monclient(hunting): get_auth_request con 0x7fbdf41311c0 auth_method 0 2023-09-30T01:35:55.403 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:55.378+0000 7fbdf2ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:35:55.403 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:55.378+0000 7fbdf2ffd700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:35:55.404 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:55.378+0000 7fbdf2ffd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:35:55.404 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:55.378+0000 7fbdf37fe700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbdf4097a40 0x7fbdf40a1eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:35:55.404 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:55.378+0000 7fbdf37fe700 10 monclient(hunting): get_auth_request con 0x7fbdf4097a40 auth_method 0 2023-09-30T01:35:55.404 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:55.378+0000 7fbdf37fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:35:55.404 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:55.378+0000 7fbdf37fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:35:55.404 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:55.378+0000 7fbdf37fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:35:55.404 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:55.378+0000 7fbdf2ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:35:55.405 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:55.378+0000 7fbdf2ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:35:55.405 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:55.378+0000 7fbdf2ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:35:55.405 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:55.378+0000 7fbdf37fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:35:55.405 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:55.378+0000 7fbdf37fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:35:55.405 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:55.378+0000 7fbdf37fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:35:55.405 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:55.378+0000 7fbdf3fff700 1 --2- 172.21.15.88:0/1983933764 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7fbdf41332e0 0x7fbdf40a2930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:35:55.405 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:55.378+0000 7fbdf3fff700 10 monclient(hunting): get_auth_request con 0x7fbdf41332e0 auth_method 0 2023-09-30T01:35:55.405 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:55.378+0000 7fbdf3fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:35:55.405 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:55.378+0000 7fbdf3fff700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:35:55.405 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:55.378+0000 7fbdf3fff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:35:55.405 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:55.378+0000 7fbdf2ffd700 10 monclient(hunting): handle_auth_done global_id 4560 payload 824 2023-09-30T01:35:55.405 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:55.378+0000 7fbdf2ffd700 1 -- 172.21.15.88:0/1983933764 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7fbdf41332e0 msgr2=0x7fbdf40a2930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:35:55.406 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:55.378+0000 7fbdf2ffd700 1 --2- 172.21.15.88:0/1983933764 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7fbdf41332e0 0x7fbdf40a2930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:35:55.406 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:55.378+0000 7fbdf2ffd700 1 -- 172.21.15.88:0/1983933764 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbdf4097a40 msgr2=0x7fbdf40a1eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:35:55.406 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:55.378+0000 7fbdf2ffd700 1 --2- 172.21.15.88:0/1983933764 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbdf4097a40 0x7fbdf40a1eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:35:55.406 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:55.378+0000 7fbdf2ffd700 10 monclient: _finish_hunting 0 2023-09-30T01:35:55.406 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:55.378+0000 7fbdf2ffd700 1 monclient: found mon.noname-d 2023-09-30T01:35:55.406 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:55.378+0000 7fbdf2ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:35:55.406 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:55.378+0000 7fbdf2ffd700 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.88:3303/0 2023-09-30T01:35:55.406 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:55.378+0000 7fbdf2ffd700 1 -- 172.21.15.88:0/1983933764 --> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbde4009420 con 0x7fbdf41311c0 2023-09-30T01:35:55.406 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:55.378+0000 7fbdf3fff700 1 --2- 172.21.15.88:0/1983933764 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7fbdf41332e0 0x7fbdf40a2930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-30T01:35:55.406 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:55.378+0000 7fbdf2ffd700 1 --2- 172.21.15.88:0/1983933764 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7fbdf41311c0 0x7fbdf40a23f0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fbdf4097f50 tx=0x7fbdec018000 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:35:55.407 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:55.378+0000 7fbdeb7fe700 1 -- 172.21.15.88:0/1983933764 <== mon.6 v2:172.21.15.88:3303/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7fbdec003300 con 0x7fbdf41311c0 2023-09-30T01:35:55.407 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:55.378+0000 7fbdeb7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:35:55.407 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:55.378+0000 7fbdeb7fe700 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2023-09-30T01:35:55.407 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:55.378+0000 7fbdeb7fe700 10 monclient: dump: 2023-09-30T01:35:55.407 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:35:55.407 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:35:55.407 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:35:55.407 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:35:55.407 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:35:55.407 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:35:55.407 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:35:55.408 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:35:55.408 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:35:55.408 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:35:55.408 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:35:55.408 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:35:55.408 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:35:55.408 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:35:55.408 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:35:55.408 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:35:55.408 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:55.378+0000 7fbdeb7fe700 10 monclient: _finish_auth 0 2023-09-30T01:35:55.408 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:55.378+0000 7fbdeb7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:35:55.408 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:55.378+0000 7fbdf8eaf700 5 monclient: authenticate success, global_id 4560 2023-09-30T01:35:55.409 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:55.378+0000 7fbdeb7fe700 1 -- 172.21.15.88:0/1983933764 <== mon.6 v2:172.21.15.88:3303/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbdec0092e0 con 0x7fbdf41311c0 2023-09-30T01:35:55.409 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:55.378+0000 7fbdf8eaf700 10 monclient: _renew_subs 2023-09-30T01:35:55.409 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:55.378+0000 7fbdf8eaf700 10 monclient: _send_mon_message to mon.d at v2:172.21.15.88:3303/0 2023-09-30T01:35:55.409 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:55.378+0000 7fbdf8eaf700 1 -- 172.21.15.88:0/1983933764 --> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbdec009ae0 con 0x7fbdf41311c0 2023-09-30T01:35:55.409 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:55.378+0000 7fbdeb7fe700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:35:55.409 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:55.378+0000 7fbdeb7fe700 1 -- 172.21.15.88:0/1983933764 <== mon.6 v2:172.21.15.88:3303/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7fbdec007000 con 0x7fbdf41311c0 2023-09-30T01:35:55.409 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:55.378+0000 7fbdeb7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:35:55.409 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:55.378+0000 7fbdeb7fe700 10 monclient: got monmap 1 from mon.d (according to old e1) 2023-09-30T01:35:55.409 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:55.378+0000 7fbdf1ffb700 4 set_mon_vals no callback set 2023-09-30T01:35:55.409 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:55.378+0000 7fbdeb7fe700 10 monclient: dump: 2023-09-30T01:35:55.409 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:35:55.410 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:35:55.410 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:35:55.410 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:35:55.410 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:35:55.410 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:35:55.410 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:35:55.410 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:35:55.410 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:35:55.410 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:35:55.410 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:35:55.411 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:35:55.411 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:35:55.411 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:35:55.411 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:35:55.411 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:35:55.411 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:55.378+0000 7fbdf8eaf700 10 monclient: _renew_subs 2023-09-30T01:35:55.411 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:55.378+0000 7fbdf8eaf700 10 monclient: _send_mon_message to mon.d at v2:172.21.15.88:3303/0 2023-09-30T01:35:55.411 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:55.378+0000 7fbdf8eaf700 1 -- 172.21.15.88:0/1983933764 --> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbdf411f380 con 0x7fbdf41311c0 2023-09-30T01:35:55.411 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:55.382+0000 7fbdeb7fe700 1 -- 172.21.15.88:0/1983933764 <== mon.6 v2:172.21.15.88:3303/0 4 ==== mgrmap(e 5) v1 ==== 40144+0+0 (secure 0 0 0) 0x7fbdec003050 con 0x7fbdf41311c0 2023-09-30T01:35:55.411 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:55.382+0000 7fbdeb7fe700 1 --2- 172.21.15.88:0/1983933764 >> v2:172.21.15.107:6812/29926 conn(0x7fbdd4033360 0x7fbdd4035810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:35:55.411 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:55.382+0000 7fbdeb7fe700 1 -- 172.21.15.88:0/1983933764 <== mon.6 v2:172.21.15.88:3303/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 3502+0+0 (secure 0 0 0) 0x7fbdec044d20 con 0x7fbdf41311c0 2023-09-30T01:35:55.412 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:55.382+0000 7fbdf37fe700 1 --2- 172.21.15.88:0/1983933764 >> v2:172.21.15.107:6812/29926 conn(0x7fbdd4033360 0x7fbdd4035810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:35:55.412 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:55.382+0000 7fbdf37fe700 10 monclient: get_auth_request con 0x7fbdd4033360 auth_method 0 2023-09-30T01:35:55.412 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:55.382+0000 7fbdf8eaf700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:35:55.412 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:55.382+0000 7fbdf8eaf700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:35:55.412 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:55.382+0000 7fbdf8eaf700 10 monclient: _send_mon_message to mon.d at v2:172.21.15.88:3303/0 2023-09-30T01:35:55.412 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:55.382+0000 7fbdf8eaf700 1 -- 172.21.15.88:0/1983933764 --> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbdf411f510 con 0x7fbdf41311c0 2023-09-30T01:35:55.412 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:55.382+0000 7fbdf37fe700 1 --2- 172.21.15.88:0/1983933764 >> v2:172.21.15.107:6812/29926 conn(0x7fbdd4033360 0x7fbdd4035810 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fbddc006f00 tx=0x7fbddc008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:35:55.413 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:55.390+0000 7fbdeb7fe700 1 -- 172.21.15.88:0/1983933764 <== mon.6 v2:172.21.15.88:3303/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fbdec007dd0 con 0x7fbdf41311c0 2023-09-30T01:35:55.413 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:55.390+0000 7fbdeb7fe700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:35:55.413 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:55.390+0000 7fbdeb7fe700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:35:55.590 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:55.586+0000 7fbdf8eaf700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T01:35:55.591 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:55.586+0000 7fbdf8eaf700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:35:55.591 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:55.586+0000 7fbdf8eaf700 10 monclient: _send_mon_message to mon.d at v2:172.21.15.88:3303/0 2023-09-30T01:35:55.591 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:55.586+0000 7fbdf8eaf700 1 -- 172.21.15.88:0/1983933764 --> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fbdf411edc0 con 0x7fbdf41311c0 2023-09-30T01:35:55.592 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:55.586+0000 7fbdeb7fe700 1 -- 172.21.15.88:0/1983933764 <== mon.6 v2:172.21.15.88:3303/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7fbdec011e60 con 0x7fbdf41311c0 2023-09-30T01:35:55.592 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:55.586+0000 7fbdeb7fe700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:35:55.592 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:55.586+0000 7fbdeb7fe700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:35:55.593 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-09-30T01:35:55.593 INFO:teuthology.orchestra.run.smithi088.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:35:55.594 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:55.590+0000 7fbdf8eaf700 1 -- 172.21.15.88:0/1983933764 >> v2:172.21.15.107:6812/29926 conn(0x7fbdd4033360 msgr2=0x7fbdd4035810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:35:55.595 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:55.590+0000 7fbdf8eaf700 1 --2- 172.21.15.88:0/1983933764 >> v2:172.21.15.107:6812/29926 conn(0x7fbdd4033360 0x7fbdd4035810 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fbddc006f00 tx=0x7fbddc008040 comp rx=0 tx=0).stop 2023-09-30T01:35:55.595 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:55.590+0000 7fbdf8eaf700 10 monclient: shutdown 2023-09-30T01:35:55.595 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:55.590+0000 7fbdf8eaf700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:35:55.595 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:55.590+0000 7fbdf8eaf700 1 -- 172.21.15.88:0/1983933764 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7fbdf41311c0 msgr2=0x7fbdf40a23f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:35:55.595 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:55.590+0000 7fbdf8eaf700 1 --2- 172.21.15.88:0/1983933764 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7fbdf41311c0 0x7fbdf40a23f0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fbdf4097f50 tx=0x7fbdec018000 comp rx=0 tx=0).stop 2023-09-30T01:35:55.595 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:55.590+0000 7fbdf8eaf700 1 -- 172.21.15.88:0/1983933764 shutdown_connections 2023-09-30T01:35:55.595 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:55.590+0000 7fbdf8eaf700 1 --2- 172.21.15.88:0/1983933764 >> v2:172.21.15.107:6812/29926 conn(0x7fbdd4033360 0x7fbdd4035810 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:35:55.595 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:55.590+0000 7fbdf8eaf700 1 --2- 172.21.15.88:0/1983933764 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7fbdf41332e0 0x7fbdf40a2930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:35:55.596 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:55.590+0000 7fbdf8eaf700 1 --2- 172.21.15.88:0/1983933764 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7fbdf41311c0 0x7fbdf40a23f0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:35:55.596 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:55.590+0000 7fbdf8eaf700 1 --2- 172.21.15.88:0/1983933764 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbdf4097a40 0x7fbdf40a1eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:35:55.596 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:55.590+0000 7fbdf8eaf700 1 -- 172.21.15.88:0/1983933764 >> 172.21.15.88:0/1983933764 conn(0x7fbdf406e5e0 msgr2=0x7fbdf40a4d10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:35:55.596 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:55.590+0000 7fbdf8eaf700 1 -- 172.21.15.88:0/1983933764 shutdown_connections 2023-09-30T01:35:55.596 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:55.590+0000 7fbdf8eaf700 1 -- 172.21.15.88:0/1983933764 wait complete. 2023-09-30T01:35:58.605 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:35:58.714 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:58.710+0000 7ffb00297700 10 monclient: get_monmap_and_config 2023-09-30T01:35:58.715 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:58.710+0000 7ffb00297700 10 monclient: build_initial_monmap 2023-09-30T01:35:58.715 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:58.710+0000 7ffb00297700 10 monclient: monmap: 2023-09-30T01:35:58.715 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2023-09-30T01:35:58.715 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:35:58.715 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:35:58.714908+0000 2023-09-30T01:35:58.715 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:35:58.714908+0000 2023-09-30T01:35:58.715 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2023-09-30T01:35:58.715 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:35:58.716 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a-legacy 2023-09-30T01:35:58.716 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.107:6789/0 mon.noname-f-legacy 2023-09-30T01:35:58.716 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.noname-b 2023-09-30T01:35:58.716 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.noname-g 2023-09-30T01:35:58.716 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.noname-c 2023-09-30T01:35:58.716 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.noname-h 2023-09-30T01:35:58.716 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.noname-d 2023-09-30T01:35:58.716 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.noname-i 2023-09-30T01:35:58.716 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.noname-e 2023-09-30T01:35:58.716 INFO:teuthology.orchestra.run.smithi088.stderr:9: v2:172.21.15.88:3300/0 mon.noname-a 2023-09-30T01:35:58.717 INFO:teuthology.orchestra.run.smithi088.stderr:10: v2:172.21.15.107:3300/0 mon.noname-f 2023-09-30T01:35:58.717 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:35:58.720 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:58.710+0000 7ffb00297700 1 Processor -- start 2023-09-30T01:35:58.720 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:58.710+0000 7ffb00297700 1 -- start start 2023-09-30T01:35:58.720 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:58.714+0000 7ffb00297700 10 monclient: init 2023-09-30T01:35:58.720 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:58.714+0000 7ffb00297700 10 monclient: _reopen_session rank -1 2023-09-30T01:35:58.720 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:58.714+0000 7ffb00297700 10 monclient: _add_conns ranks=[5,8,6,4,9,3,0,7,1,10,2] 2023-09-30T01:35:58.721 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:58.714+0000 7ffb00297700 1 --2- >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7ffaf80a3270 0x7ffaf80a0160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:35:58.721 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:58.714+0000 7ffb00297700 10 monclient(hunting): picked mon.noname-h con 0x7ffaf80a3270 addr [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] 2023-09-30T01:35:58.721 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:58.714+0000 7ffb00297700 1 --2- >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7ffaf80a4b00 0x7ffaf80a06a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:35:58.721 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:58.714+0000 7ffb00297700 10 monclient(hunting): picked mon.noname-e con 0x7ffaf80a4b00 addr [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] 2023-09-30T01:35:58.721 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:58.714+0000 7ffb00297700 1 --2- >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7ffaf81333c0 0x7ffaf81337b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:35:58.721 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:58.714+0000 7ffb00297700 10 monclient(hunting): picked mon.noname-d con 0x7ffaf81333c0 addr [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] 2023-09-30T01:35:58.721 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:58.714+0000 7ffb00297700 10 monclient(hunting): start opening mon connection 2023-09-30T01:35:58.721 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:58.714+0000 7ffb00297700 1 -- --> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] -- mon_getmap magic: 0 v1 -- 0x7ffaf80a4ed0 con 0x7ffaf80a3270 2023-09-30T01:35:58.721 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:58.714+0000 7ffb00297700 10 monclient(hunting): start opening mon connection 2023-09-30T01:35:58.721 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:58.714+0000 7ffb00297700 1 -- --> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] -- mon_getmap magic: 0 v1 -- 0x7ffaf80a6330 con 0x7ffaf81333c0 2023-09-30T01:35:58.722 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:58.714+0000 7ffb00297700 10 monclient(hunting): start opening mon connection 2023-09-30T01:35:58.722 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:58.714+0000 7ffb00297700 1 -- --> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] -- mon_getmap magic: 0 v1 -- 0x7ffaf8098610 con 0x7ffaf80a4b00 2023-09-30T01:35:58.722 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:58.714+0000 7ffb00297700 10 monclient(hunting): _renew_subs 2023-09-30T01:35:58.722 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:58.714+0000 7ffb00297700 10 monclient(hunting): authenticate will time out at 1030.752713s 2023-09-30T01:35:58.722 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:58.714+0000 7ffafe834700 1 --2- >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7ffaf81333c0 0x7ffaf81337b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:35:58.722 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:58.714+0000 7ffafd832700 1 --2- >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7ffaf80a4b00 0x7ffaf80a06a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:35:58.722 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:58.714+0000 7ffafe834700 1 --2- >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7ffaf81333c0 0x7ffaf81337b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3303/0 says I am v2:172.21.15.88:60284/0 (socket says 172.21.15.88:60284) 2023-09-30T01:35:58.722 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:58.714+0000 7ffafe834700 1 -- 172.21.15.88:0/728489321 learned_addr learned my addr 172.21.15.88:0/728489321 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-09-30T01:35:58.722 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:58.714+0000 7ffafe834700 10 monclient(hunting): get_auth_request con 0x7ffaf81333c0 auth_method 0 2023-09-30T01:35:58.723 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:58.714+0000 7ffafe834700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:35:58.723 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:58.714+0000 7ffafe834700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:35:58.723 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:58.714+0000 7ffafe834700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:35:58.723 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:58.714+0000 7ffafd832700 10 monclient(hunting): get_auth_request con 0x7ffaf80a4b00 auth_method 0 2023-09-30T01:35:58.723 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:58.714+0000 7ffafd832700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:35:58.723 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:58.714+0000 7ffafd832700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:35:58.723 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:58.714+0000 7ffafd832700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:35:58.723 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:58.714+0000 7ffafd832700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:35:58.723 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:58.714+0000 7ffafd832700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:35:58.723 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:58.714+0000 7ffafe033700 1 --2- 172.21.15.88:0/728489321 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7ffaf80a3270 0x7ffaf80a0160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:35:58.723 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:58.714+0000 7ffafd832700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:35:58.724 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:58.714+0000 7ffafe834700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:35:58.724 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:58.714+0000 7ffafe834700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:35:58.724 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:58.714+0000 7ffafe834700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:35:58.724 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:58.714+0000 7ffafe033700 10 monclient(hunting): get_auth_request con 0x7ffaf80a3270 auth_method 0 2023-09-30T01:35:58.724 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:58.714+0000 7ffafe033700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:35:58.724 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:58.714+0000 7ffafe033700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:35:58.724 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:58.714+0000 7ffafe033700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:35:58.724 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:58.714+0000 7ffafd832700 10 monclient(hunting): handle_auth_done global_id 4697 payload 274 2023-09-30T01:35:58.724 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:58.714+0000 7ffafd832700 1 -- 172.21.15.88:0/728489321 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7ffaf81333c0 msgr2=0x7ffaf81337b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:35:58.724 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:58.714+0000 7ffafd832700 1 --2- 172.21.15.88:0/728489321 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7ffaf81333c0 0x7ffaf81337b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:35:58.725 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:58.714+0000 7ffafd832700 1 -- 172.21.15.88:0/728489321 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7ffaf80a3270 msgr2=0x7ffaf80a0160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:35:58.725 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:58.714+0000 7ffafd832700 1 --2- 172.21.15.88:0/728489321 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7ffaf80a3270 0x7ffaf80a0160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:35:58.725 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:58.714+0000 7ffafd832700 10 monclient: _finish_hunting 0 2023-09-30T01:35:58.725 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:58.714+0000 7ffafd832700 1 monclient: found mon.noname-e 2023-09-30T01:35:58.725 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:58.714+0000 7ffafd832700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:35:58.725 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:58.714+0000 7ffafd832700 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.88:3304/0 2023-09-30T01:35:58.725 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:58.714+0000 7ffafd832700 1 -- 172.21.15.88:0/728489321 --> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffaf805ecb0 con 0x7ffaf80a4b00 2023-09-30T01:35:58.725 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:58.714+0000 7ffafe033700 1 --2- 172.21.15.88:0/728489321 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7ffaf80a3270 0x7ffaf80a0160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-30T01:35:58.725 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:58.714+0000 7ffafd832700 1 --2- 172.21.15.88:0/728489321 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7ffaf80a4b00 0x7ffaf80a06a0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7ffae8009910 tx=0x7ffae80091a0 comp rx=0 tx=0).ready entity=mon.8 client_cookie=6ad51a766e25ff95 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:35:58.725 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:58.714+0000 7ffafd031700 1 -- 172.21.15.88:0/728489321 <== mon.8 v2:172.21.15.88:3304/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7ffae8011ab0 con 0x7ffaf80a4b00 2023-09-30T01:35:58.725 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:58.714+0000 7ffafd031700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:35:58.726 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:58.714+0000 7ffafd031700 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2023-09-30T01:35:58.726 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:58.714+0000 7ffafd031700 10 monclient: dump: 2023-09-30T01:35:58.726 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:35:58.726 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:35:58.726 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:35:58.726 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:35:58.726 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:35:58.726 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:35:58.726 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:35:58.726 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:35:58.727 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:35:58.727 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:35:58.727 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:35:58.727 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:35:58.727 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:35:58.727 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:35:58.727 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:35:58.727 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:35:58.727 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:58.714+0000 7ffafd031700 10 monclient: _finish_auth 0 2023-09-30T01:35:58.727 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:58.714+0000 7ffafd031700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:35:58.727 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:58.714+0000 7ffb00297700 5 monclient: authenticate success, global_id 4697 2023-09-30T01:35:58.728 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:58.714+0000 7ffafd031700 1 -- 172.21.15.88:0/728489321 <== mon.8 v2:172.21.15.88:3304/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffae8011c30 con 0x7ffaf80a4b00 2023-09-30T01:35:58.728 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:58.714+0000 7ffb00297700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:35:58.728 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:58.714+0000 7ffafd031700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:35:58.728 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:58.714+0000 7ffb00297700 10 monclient: get_monmap_and_config success 2023-09-30T01:35:58.728 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:58.714+0000 7ffb00297700 4 set_mon_vals no callback set 2023-09-30T01:35:58.728 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:58.714+0000 7ffafd031700 1 -- 172.21.15.88:0/728489321 <== mon.8 v2:172.21.15.88:3304/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7ffae8011da0 con 0x7ffaf80a4b00 2023-09-30T01:35:58.728 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:58.714+0000 7ffafd031700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:35:58.728 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:58.714+0000 7ffafd031700 10 monclient: got monmap 1 from mon.e (according to old e1) 2023-09-30T01:35:58.728 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:58.714+0000 7ffafd031700 10 monclient: dump: 2023-09-30T01:35:58.728 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:35:58.729 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:35:58.729 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:35:58.729 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:35:58.729 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:35:58.729 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:35:58.729 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:35:58.729 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:35:58.729 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:35:58.729 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:35:58.729 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:35:58.729 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:35:58.730 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:35:58.730 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:35:58.730 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:35:58.730 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:35:58.730 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:58.714+0000 7ffb00297700 10 monclient: shutdown 2023-09-30T01:35:58.730 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:58.714+0000 7ffb00297700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:35:58.730 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:58.714+0000 7ffb00297700 1 -- 172.21.15.88:0/728489321 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7ffaf80a4b00 msgr2=0x7ffaf80a06a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:35:58.730 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:58.714+0000 7ffb00297700 1 --2- 172.21.15.88:0/728489321 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7ffaf80a4b00 0x7ffaf80a06a0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7ffae8009910 tx=0x7ffae80091a0 comp rx=0 tx=0).stop 2023-09-30T01:35:58.730 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:58.714+0000 7ffb00297700 1 -- 172.21.15.88:0/728489321 shutdown_connections 2023-09-30T01:35:58.730 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:58.714+0000 7ffb00297700 1 --2- 172.21.15.88:0/728489321 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7ffaf80a4b00 0x7ffaf80a06a0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:35:58.731 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:58.714+0000 7ffb00297700 1 --2- 172.21.15.88:0/728489321 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7ffaf81333c0 0x7ffaf81337b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:35:58.731 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:58.714+0000 7ffb00297700 1 --2- 172.21.15.88:0/728489321 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7ffaf80a3270 0x7ffaf80a0160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:35:58.731 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:58.714+0000 7ffb00297700 1 -- 172.21.15.88:0/728489321 >> 172.21.15.88:0/728489321 conn(0x7ffaf80a67f0 msgr2=0x7ffaf80a6bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:35:58.731 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:58.714+0000 7ffb00297700 1 -- 172.21.15.88:0/728489321 shutdown_connections 2023-09-30T01:35:58.731 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:58.714+0000 7ffb00297700 1 -- 172.21.15.88:0/728489321 wait complete. 2023-09-30T01:35:58.731 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:58.714+0000 7ffb00297700 10 monclient: build_initial_monmap 2023-09-30T01:35:58.731 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:58.714+0000 7ffb00297700 10 monclient: monmap: 2023-09-30T01:35:58.731 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2023-09-30T01:35:58.731 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:35:58.731 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 0.000000 2023-09-30T01:35:58.732 INFO:teuthology.orchestra.run.smithi088.stderr:created 0.000000 2023-09-30T01:35:58.732 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2023-09-30T01:35:58.732 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 1 2023-09-30T01:35:58.732 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.noname-a 2023-09-30T01:35:58.732 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.noname-f 2023-09-30T01:35:58.732 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.noname-b 2023-09-30T01:35:58.732 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.noname-g 2023-09-30T01:35:58.732 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.noname-c 2023-09-30T01:35:58.732 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.noname-h 2023-09-30T01:35:58.732 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.noname-d 2023-09-30T01:35:58.732 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.noname-i 2023-09-30T01:35:58.733 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.noname-e 2023-09-30T01:35:58.733 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:35:58.733 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:58.718+0000 7ffb00297700 1 Processor -- start 2023-09-30T01:35:58.733 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:58.718+0000 7ffb00297700 1 -- start start 2023-09-30T01:35:58.733 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:58.718+0000 7ffb00297700 10 monclient: init 2023-09-30T01:35:58.737 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:58.718+0000 7ffb00297700 10 monclient: _reopen_session rank -1 2023-09-30T01:35:58.737 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:58.718+0000 7ffb00297700 10 monclient: _add_conns ranks=[5,7,3,1,0,8,2,4,6] 2023-09-30T01:35:58.738 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:58.718+0000 7ffb00297700 1 --2- >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7ffaf80a3270 0x7ffaf811d9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:35:58.738 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:58.718+0000 7ffb00297700 10 monclient(hunting): picked mon.noname-h con 0x7ffaf80a3270 addr [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] 2023-09-30T01:35:58.738 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:58.718+0000 7ffb00297700 1 --2- >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7ffaf80a4b00 0x7ffaf811df10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:35:58.738 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:58.718+0000 7ffb00297700 10 monclient(hunting): picked mon.noname-i con 0x7ffaf80a4b00 addr [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] 2023-09-30T01:35:58.738 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:58.718+0000 7ffb00297700 1 --2- >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7ffaf81333c0 0x7ffaf811e450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:35:58.738 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:58.718+0000 7ffb00297700 10 monclient(hunting): picked mon.noname-g con 0x7ffaf81333c0 addr [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] 2023-09-30T01:35:58.738 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:58.718+0000 7ffb00297700 10 monclient(hunting): start opening mon connection 2023-09-30T01:35:58.738 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:58.718+0000 7ffb00297700 1 -- --> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffaf8097c70 con 0x7ffaf81333c0 2023-09-30T01:35:58.738 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:58.718+0000 7ffb00297700 10 monclient(hunting): start opening mon connection 2023-09-30T01:35:58.739 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:58.718+0000 7ffb00297700 1 -- --> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] -- mon_getmap magic: 0 v1 -- 0x7ffaf80a0e80 con 0x7ffaf80a3270 2023-09-30T01:35:58.739 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:58.718+0000 7ffb00297700 10 monclient(hunting): start opening mon connection 2023-09-30T01:35:58.739 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:58.718+0000 7ffb00297700 1 -- --> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] -- mon_getmap magic: 0 v1 -- 0x7ffaf811f460 con 0x7ffaf80a4b00 2023-09-30T01:35:58.739 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:58.718+0000 7ffb00297700 10 monclient(hunting): _renew_subs 2023-09-30T01:35:58.739 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:58.718+0000 7ffb00297700 10 monclient(hunting): authenticate will time out at 1030.757355s 2023-09-30T01:35:58.739 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:58.718+0000 7ffafe033700 1 --2- >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7ffaf80a3270 0x7ffaf811d9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:35:58.739 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:58.718+0000 7ffafd832700 1 --2- >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7ffaf80a4b00 0x7ffaf811df10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:35:58.739 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:58.718+0000 7ffafe033700 1 --2- >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7ffaf80a3270 0x7ffaf811d9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3302/0 says I am v2:172.21.15.88:58750/0 (socket says 172.21.15.88:58750) 2023-09-30T01:35:58.739 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:58.718+0000 7ffafe834700 1 --2- >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7ffaf81333c0 0x7ffaf811e450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:35:58.739 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:58.718+0000 7ffafd832700 1 --2- >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7ffaf80a4b00 0x7ffaf811df10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3303/0 says I am v2:172.21.15.88:57990/0 (socket says 172.21.15.88:57990) 2023-09-30T01:35:58.740 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:58.718+0000 7ffafe033700 1 -- 172.21.15.88:0/2763761633 learned_addr learned my addr 172.21.15.88:0/2763761633 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-09-30T01:35:58.740 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:58.718+0000 7ffafe033700 10 monclient(hunting): get_auth_request con 0x7ffaf80a3270 auth_method 0 2023-09-30T01:35:58.740 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:58.718+0000 7ffafe033700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:35:58.740 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:58.718+0000 7ffafe033700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:35:58.740 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:58.718+0000 7ffafe033700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:35:58.740 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:58.718+0000 7ffafd832700 10 monclient(hunting): get_auth_request con 0x7ffaf80a4b00 auth_method 0 2023-09-30T01:35:58.740 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:58.718+0000 7ffafd832700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:35:58.740 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:58.718+0000 7ffafd832700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:35:58.740 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:58.718+0000 7ffafd832700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:35:58.740 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:58.718+0000 7ffafe834700 10 monclient(hunting): get_auth_request con 0x7ffaf81333c0 auth_method 0 2023-09-30T01:35:58.740 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:58.718+0000 7ffafe834700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:35:58.741 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:58.718+0000 7ffafe834700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:35:58.741 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:58.718+0000 7ffafe834700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:35:58.741 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:58.718+0000 7ffafe033700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:35:58.741 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:58.718+0000 7ffafe033700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:35:58.741 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:58.718+0000 7ffafe033700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:35:58.741 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:58.718+0000 7ffafd832700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:35:58.741 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:58.718+0000 7ffafd832700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:35:58.741 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:58.718+0000 7ffafd832700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:35:58.741 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:58.718+0000 7ffafe834700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:35:58.741 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:58.718+0000 7ffafe834700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:35:58.742 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:58.718+0000 7ffafe834700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:35:58.742 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:58.718+0000 7ffafe033700 10 monclient(hunting): handle_auth_done global_id 4604 payload 824 2023-09-30T01:35:58.742 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:58.718+0000 7ffafe033700 1 -- 172.21.15.88:0/2763761633 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7ffaf80a4b00 msgr2=0x7ffaf811df10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:35:58.742 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:58.718+0000 7ffafe033700 1 --2- 172.21.15.88:0/2763761633 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7ffaf80a4b00 0x7ffaf811df10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:35:58.742 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:58.718+0000 7ffafe033700 1 -- 172.21.15.88:0/2763761633 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7ffaf81333c0 msgr2=0x7ffaf811e450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:35:58.742 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:58.718+0000 7ffafe033700 1 --2- 172.21.15.88:0/2763761633 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7ffaf81333c0 0x7ffaf811e450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:35:58.742 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:58.718+0000 7ffafe033700 10 monclient: _finish_hunting 0 2023-09-30T01:35:58.742 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:58.718+0000 7ffafe033700 1 monclient: found mon.noname-h 2023-09-30T01:35:58.742 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:58.718+0000 7ffafe033700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:35:58.742 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:58.718+0000 7ffafe033700 10 monclient: _send_mon_message to mon.noname-h at v2:172.21.15.107:3302/0 2023-09-30T01:35:58.742 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:58.718+0000 7ffafe033700 1 -- 172.21.15.88:0/2763761633 --> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffae8009420 con 0x7ffaf80a3270 2023-09-30T01:35:58.743 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:58.718+0000 7ffafd832700 1 --2- 172.21.15.88:0/2763761633 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7ffaf80a4b00 0x7ffaf811df10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:35:58.743 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:58.718+0000 7ffafe033700 1 --2- 172.21.15.88:0/2763761633 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7ffaf80a3270 0x7ffaf811d9d0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7ffaf400e900 tx=0x7ffaf4008040 comp rx=0 tx=0).ready entity=mon.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:35:58.743 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:58.718+0000 7ffaee7fc700 1 -- 172.21.15.88:0/2763761633 <== mon.5 v2:172.21.15.107:3302/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7ffaf4013890 con 0x7ffaf80a3270 2023-09-30T01:35:58.743 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:58.718+0000 7ffaee7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:35:58.743 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:58.718+0000 7ffaee7fc700 10 monclient: got monmap 1 from mon.noname-h (according to old e1) 2023-09-30T01:35:58.743 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:58.718+0000 7ffaee7fc700 10 monclient: dump: 2023-09-30T01:35:58.743 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:35:58.743 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:35:58.743 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:35:58.744 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:35:58.744 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:35:58.744 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:35:58.744 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:35:58.744 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:35:58.744 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:35:58.744 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:35:58.744 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:35:58.744 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:35:58.744 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:35:58.745 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:35:58.745 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:35:58.745 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:35:58.745 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:58.718+0000 7ffaee7fc700 10 monclient: _finish_auth 0 2023-09-30T01:35:58.745 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:58.718+0000 7ffaee7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:35:58.745 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:58.718+0000 7ffb00297700 5 monclient: authenticate success, global_id 4604 2023-09-30T01:35:58.745 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:58.718+0000 7ffaee7fc700 1 -- 172.21.15.88:0/2763761633 <== mon.5 v2:172.21.15.107:3302/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffaf4013a30 con 0x7ffaf80a3270 2023-09-30T01:35:58.745 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:58.718+0000 7ffb00297700 10 monclient: _renew_subs 2023-09-30T01:35:58.745 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:58.718+0000 7ffb00297700 10 monclient: _send_mon_message to mon.h at v2:172.21.15.107:3302/0 2023-09-30T01:35:58.746 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:58.718+0000 7ffb00297700 1 -- 172.21.15.88:0/2763761633 --> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffaf8120f00 con 0x7ffaf80a3270 2023-09-30T01:35:58.746 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:58.718+0000 7ffaee7fc700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:35:58.746 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:58.718+0000 7ffaee7fc700 1 -- 172.21.15.88:0/2763761633 <== mon.5 v2:172.21.15.107:3302/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7ffaf4013ba0 con 0x7ffaf80a3270 2023-09-30T01:35:58.746 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:58.718+0000 7ffaee7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:35:58.746 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:58.718+0000 7ffaee7fc700 10 monclient: got monmap 1 from mon.h (according to old e1) 2023-09-30T01:35:58.746 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:58.718+0000 7ffafc830700 4 set_mon_vals no callback set 2023-09-30T01:35:58.746 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:58.718+0000 7ffaee7fc700 10 monclient: dump: 2023-09-30T01:35:58.746 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:35:58.746 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:35:58.746 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:35:58.747 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:35:58.747 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:35:58.747 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:35:58.747 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:35:58.747 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:35:58.747 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:35:58.747 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:35:58.747 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:35:58.747 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:35:58.747 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:35:58.748 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:35:58.748 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:35:58.748 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:35:58.748 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:58.718+0000 7ffb00297700 10 monclient: _renew_subs 2023-09-30T01:35:58.748 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:58.718+0000 7ffb00297700 10 monclient: _send_mon_message to mon.h at v2:172.21.15.107:3302/0 2023-09-30T01:35:58.748 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:58.718+0000 7ffb00297700 1 -- 172.21.15.88:0/2763761633 --> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffaf8064b80 con 0x7ffaf80a3270 2023-09-30T01:35:58.748 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:58.722+0000 7ffaee7fc700 1 -- 172.21.15.88:0/2763761633 <== mon.5 v2:172.21.15.107:3302/0 4 ==== mgrmap(e 5) v1 ==== 40144+0+0 (secure 0 0 0) 0x7ffaf4013d00 con 0x7ffaf80a3270 2023-09-30T01:35:58.748 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:58.722+0000 7ffb00297700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:35:58.748 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:58.722+0000 7ffb00297700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:35:58.749 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:58.722+0000 7ffb00297700 10 monclient: _send_mon_message to mon.h at v2:172.21.15.107:3302/0 2023-09-30T01:35:58.749 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:58.722+0000 7ffb00297700 1 -- 172.21.15.88:0/2763761633 --> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffaf8071960 con 0x7ffaf80a3270 2023-09-30T01:35:58.749 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:58.722+0000 7ffaee7fc700 1 --2- 172.21.15.88:0/2763761633 >> v2:172.21.15.107:6812/29926 conn(0x7ffadc033360 0x7ffadc035810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:35:58.749 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:58.722+0000 7ffaee7fc700 1 -- 172.21.15.88:0/2763761633 <== mon.5 v2:172.21.15.107:3302/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 3502+0+0 (secure 0 0 0) 0x7ffaf40430c0 con 0x7ffaf80a3270 2023-09-30T01:35:58.749 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:58.722+0000 7ffafd832700 1 --2- 172.21.15.88:0/2763761633 >> v2:172.21.15.107:6812/29926 conn(0x7ffadc033360 0x7ffadc035810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:35:58.749 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:58.722+0000 7ffafd832700 10 monclient: get_auth_request con 0x7ffadc033360 auth_method 0 2023-09-30T01:35:58.749 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:58.722+0000 7ffafd832700 1 --2- 172.21.15.88:0/2763761633 >> v2:172.21.15.107:6812/29926 conn(0x7ffadc033360 0x7ffadc035810 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7ffae8009910 tx=0x7ffae8017000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:35:58.749 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:58.730+0000 7ffaee7fc700 1 -- 172.21.15.88:0/2763761633 <== mon.5 v2:172.21.15.107:3302/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7ffaf40226c0 con 0x7ffaf80a3270 2023-09-30T01:35:58.749 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:58.730+0000 7ffaee7fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:35:58.749 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:58.730+0000 7ffaee7fc700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:35:58.933 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:58.930+0000 7ffb00297700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T01:35:58.933 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:58.930+0000 7ffb00297700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:35:58.933 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:58.930+0000 7ffb00297700 10 monclient: _send_mon_message to mon.h at v2:172.21.15.107:3302/0 2023-09-30T01:35:58.933 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:58.930+0000 7ffb00297700 1 -- 172.21.15.88:0/2763761633 --> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7ffaf8062c70 con 0x7ffaf80a3270 2023-09-30T01:35:58.934 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:58.930+0000 7ffaee7fc700 1 -- 172.21.15.88:0/2763761633 <== mon.5 v2:172.21.15.107:3302/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7ffaf4020100 con 0x7ffaf80a3270 2023-09-30T01:35:58.934 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:58.930+0000 7ffaee7fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:35:58.934 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:58.930+0000 7ffaee7fc700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:35:58.934 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-09-30T01:35:58.934 INFO:teuthology.orchestra.run.smithi088.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:35:58.936 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:58.934+0000 7ffb00297700 1 -- 172.21.15.88:0/2763761633 >> v2:172.21.15.107:6812/29926 conn(0x7ffadc033360 msgr2=0x7ffadc035810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:35:58.937 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:58.934+0000 7ffb00297700 1 --2- 172.21.15.88:0/2763761633 >> v2:172.21.15.107:6812/29926 conn(0x7ffadc033360 0x7ffadc035810 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7ffae8009910 tx=0x7ffae8017000 comp rx=0 tx=0).stop 2023-09-30T01:35:58.937 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:58.934+0000 7ffb00297700 10 monclient: shutdown 2023-09-30T01:35:58.937 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:58.934+0000 7ffb00297700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:35:58.937 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:58.934+0000 7ffb00297700 1 -- 172.21.15.88:0/2763761633 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7ffaf80a3270 msgr2=0x7ffaf811d9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:35:58.937 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:58.934+0000 7ffb00297700 1 --2- 172.21.15.88:0/2763761633 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7ffaf80a3270 0x7ffaf811d9d0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7ffaf400e900 tx=0x7ffaf4008040 comp rx=0 tx=0).stop 2023-09-30T01:35:58.937 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:58.934+0000 7ffb00297700 1 -- 172.21.15.88:0/2763761633 shutdown_connections 2023-09-30T01:35:58.937 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:58.934+0000 7ffb00297700 1 --2- 172.21.15.88:0/2763761633 >> v2:172.21.15.107:6812/29926 conn(0x7ffadc033360 0x7ffadc035810 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:35:58.937 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:58.934+0000 7ffb00297700 1 --2- 172.21.15.88:0/2763761633 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7ffaf81333c0 0x7ffaf811e450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:35:58.937 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:58.934+0000 7ffb00297700 1 --2- 172.21.15.88:0/2763761633 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7ffaf80a4b00 0x7ffaf811df10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:35:58.937 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:58.934+0000 7ffb00297700 1 --2- 172.21.15.88:0/2763761633 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7ffaf80a3270 0x7ffaf811d9d0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:35:58.937 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:58.934+0000 7ffb00297700 1 -- 172.21.15.88:0/2763761633 >> 172.21.15.88:0/2763761633 conn(0x7ffaf80a67f0 msgr2=0x7ffaf80a20e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:35:58.938 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:58.934+0000 7ffb00297700 1 -- 172.21.15.88:0/2763761633 shutdown_connections 2023-09-30T01:35:58.938 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:35:58.934+0000 7ffb00297700 1 -- 172.21.15.88:0/2763761633 wait complete. 2023-09-30T01:36:01.949 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:36:02.075 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:02.058+0000 7f36ac8d5700 10 monclient: get_monmap_and_config 2023-09-30T01:36:02.076 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:02.058+0000 7f36ac8d5700 10 monclient: build_initial_monmap 2023-09-30T01:36:02.076 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:02.058+0000 7f36ac8d5700 10 monclient: monmap: 2023-09-30T01:36:02.076 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2023-09-30T01:36:02.076 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:36:02.076 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:36:02.061235+0000 2023-09-30T01:36:02.076 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:36:02.061235+0000 2023-09-30T01:36:02.076 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2023-09-30T01:36:02.076 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:36:02.076 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a-legacy 2023-09-30T01:36:02.076 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.107:6789/0 mon.noname-f-legacy 2023-09-30T01:36:02.076 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.noname-b 2023-09-30T01:36:02.077 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.noname-g 2023-09-30T01:36:02.077 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.noname-c 2023-09-30T01:36:02.077 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.noname-h 2023-09-30T01:36:02.077 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.noname-d 2023-09-30T01:36:02.077 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.noname-i 2023-09-30T01:36:02.077 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.noname-e 2023-09-30T01:36:02.077 INFO:teuthology.orchestra.run.smithi088.stderr:9: v2:172.21.15.88:3300/0 mon.noname-a 2023-09-30T01:36:02.077 INFO:teuthology.orchestra.run.smithi088.stderr:10: v2:172.21.15.107:3300/0 mon.noname-f 2023-09-30T01:36:02.077 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:36:02.077 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:02.058+0000 7f36ac8d5700 1 Processor -- start 2023-09-30T01:36:02.077 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:02.058+0000 7f36ac8d5700 1 -- start start 2023-09-30T01:36:02.077 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:02.058+0000 7f36ac8d5700 10 monclient: init 2023-09-30T01:36:02.078 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:02.058+0000 7f36ac8d5700 10 monclient: _reopen_session rank -1 2023-09-30T01:36:02.078 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:02.058+0000 7f36ac8d5700 10 monclient: _add_conns ranks=[8,2,9,7,0,1,4,6,10,3,5] 2023-09-30T01:36:02.078 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:02.058+0000 7f36ac8d5700 1 --2- >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f36a8099a20 0x7f36a809b270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:36:02.078 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:02.058+0000 7f36ac8d5700 10 monclient(hunting): picked mon.noname-e con 0x7f36a8099a20 addr [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] 2023-09-30T01:36:02.078 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:02.058+0000 7f36ac8d5700 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f36a809cf30 0x7f36a809d300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:36:02.078 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:02.058+0000 7f36ac8d5700 10 monclient(hunting): picked mon.noname-b con 0x7f36a809cf30 addr [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] 2023-09-30T01:36:02.078 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:02.058+0000 7f36ac8d5700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f36a809d840 0x7f36a808d110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:36:02.078 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:02.058+0000 7f36ac8d5700 10 monclient(hunting): picked mon.noname-a con 0x7f36a809d840 addr v2:172.21.15.88:3300/0 2023-09-30T01:36:02.078 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:02.058+0000 7f36ac8d5700 10 monclient(hunting): start opening mon connection 2023-09-30T01:36:02.078 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:02.058+0000 7f36ac8d5700 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f36a809eee0 con 0x7f36a809d840 2023-09-30T01:36:02.078 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:02.058+0000 7f36ac8d5700 10 monclient(hunting): start opening mon connection 2023-09-30T01:36:02.079 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:02.058+0000 7f36ac8d5700 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f36a809c9d0 con 0x7f36a809cf30 2023-09-30T01:36:02.079 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:02.058+0000 7f36ac8d5700 10 monclient(hunting): start opening mon connection 2023-09-30T01:36:02.079 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:02.058+0000 7f36ac8d5700 1 -- --> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] -- mon_getmap magic: 0 v1 -- 0x7f36a809a890 con 0x7f36a8099a20 2023-09-30T01:36:02.079 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:02.058+0000 7f36ac8d5700 10 monclient(hunting): _renew_subs 2023-09-30T01:36:02.079 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:02.058+0000 7f36ac8d5700 10 monclient(hunting): authenticate will time out at 1034.098625s 2023-09-30T01:36:02.079 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:02.058+0000 7f36a6ffd700 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f36a809cf30 0x7f36a809d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:36:02.079 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:02.058+0000 7f36a6ffd700 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f36a809cf30 0x7f36a809d300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3301/0 says I am v2:172.21.15.88:51518/0 (socket says 172.21.15.88:51518) 2023-09-30T01:36:02.079 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:02.058+0000 7f36a6ffd700 1 -- 172.21.15.88:0/4255346062 learned_addr learned my addr 172.21.15.88:0/4255346062 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-09-30T01:36:02.079 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:02.058+0000 7f36a6ffd700 10 monclient(hunting): get_auth_request con 0x7f36a809cf30 auth_method 0 2023-09-30T01:36:02.079 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:02.058+0000 7f36a77fe700 1 --2- 172.21.15.88:0/4255346062 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f36a8099a20 0x7f36a809b270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:36:02.079 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:02.058+0000 7f36a6ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:36:02.080 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:02.058+0000 7f36a7fff700 1 --2- 172.21.15.88:0/4255346062 >> v2:172.21.15.88:3300/0 conn(0x7f36a809d840 0x7f36a808d110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:36:02.080 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:02.070+0000 7f36a6ffd700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:36:02.080 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:02.070+0000 7f36a6ffd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:36:02.080 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:02.070+0000 7f36a7fff700 10 monclient(hunting): get_auth_request con 0x7f36a809d840 auth_method 0 2023-09-30T01:36:02.080 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:02.070+0000 7f36a7fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:36:02.080 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:02.070+0000 7f36a7fff700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:36:02.080 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:02.070+0000 7f36a7fff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:36:02.080 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:02.070+0000 7f36a77fe700 10 monclient(hunting): get_auth_request con 0x7f36a8099a20 auth_method 0 2023-09-30T01:36:02.080 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:02.070+0000 7f36a77fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:36:02.080 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:02.070+0000 7f36a77fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:36:02.080 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:02.070+0000 7f36a77fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:36:02.080 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:02.070+0000 7f36a6ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:36:02.081 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:02.070+0000 7f36a6ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:36:02.081 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:02.070+0000 7f36a6ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:36:02.081 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:02.070+0000 7f36a7fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:36:02.081 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:02.070+0000 7f36a7fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:36:02.081 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:02.070+0000 7f36a7fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:36:02.081 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:02.070+0000 7f36a77fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:36:02.081 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:02.070+0000 7f36a77fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:36:02.081 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:02.070+0000 7f36a77fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:36:02.081 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:02.070+0000 7f36a6ffd700 10 monclient(hunting): handle_auth_done global_id 4628 payload 274 2023-09-30T01:36:02.081 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:02.070+0000 7f36a6ffd700 1 -- 172.21.15.88:0/4255346062 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f36a8099a20 msgr2=0x7f36a809b270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:36:02.081 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:02.070+0000 7f36a6ffd700 1 --2- 172.21.15.88:0/4255346062 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f36a8099a20 0x7f36a809b270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:36:02.081 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:02.070+0000 7f36a6ffd700 1 -- 172.21.15.88:0/4255346062 >> v2:172.21.15.88:3300/0 conn(0x7f36a809d840 msgr2=0x7f36a808d110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:36:02.082 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:02.070+0000 7f36a6ffd700 1 --2- 172.21.15.88:0/4255346062 >> v2:172.21.15.88:3300/0 conn(0x7f36a809d840 0x7f36a808d110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:36:02.082 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:02.070+0000 7f36a6ffd700 10 monclient: _finish_hunting 0 2023-09-30T01:36:02.082 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:02.070+0000 7f36a6ffd700 1 monclient: found mon.noname-b 2023-09-30T01:36:02.082 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:02.070+0000 7f36a6ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:36:02.082 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:02.070+0000 7f36a6ffd700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.88:3301/0 2023-09-30T01:36:02.082 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:02.070+0000 7f36a6ffd700 1 -- 172.21.15.88:0/4255346062 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36a804bb20 con 0x7f36a809cf30 2023-09-30T01:36:02.082 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:02.070+0000 7f36a6ffd700 1 --2- 172.21.15.88:0/4255346062 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f36a809cf30 0x7f36a809d300 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f369800bfc0 tx=0x7f369800d040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6695a03f384240ae server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:36:02.082 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:02.070+0000 7f36a67fc700 1 -- 172.21.15.88:0/4255346062 <== mon.2 v2:172.21.15.88:3301/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f369800ce40 con 0x7f36a809cf30 2023-09-30T01:36:02.082 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:02.070+0000 7f36a67fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:36:02.082 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:02.070+0000 7f36a67fc700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T01:36:02.082 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:02.070+0000 7f36a67fc700 10 monclient: dump: 2023-09-30T01:36:02.083 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:36:02.083 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:36:02.083 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:36:02.083 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:36:02.083 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:36:02.083 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:36:02.083 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:36:02.083 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:36:02.083 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:36:02.083 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:36:02.083 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:36:02.083 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:36:02.084 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:36:02.084 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:36:02.084 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:36:02.084 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:36:02.084 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:02.070+0000 7f36a67fc700 10 monclient: _finish_auth 0 2023-09-30T01:36:02.084 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:02.070+0000 7f36a67fc700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:36:02.084 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:02.070+0000 7f36ac8d5700 5 monclient: authenticate success, global_id 4628 2023-09-30T01:36:02.084 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:02.070+0000 7f36ac8d5700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:36:02.084 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:02.070+0000 7f36a67fc700 1 -- 172.21.15.88:0/4255346062 <== mon.2 v2:172.21.15.88:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f369800b2e0 con 0x7f36a809cf30 2023-09-30T01:36:02.084 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:02.070+0000 7f36a67fc700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:36:02.084 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:02.070+0000 7f36a67fc700 1 -- 172.21.15.88:0/4255346062 <== mon.2 v2:172.21.15.88:3301/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f369800b450 con 0x7f36a809cf30 2023-09-30T01:36:02.084 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:02.070+0000 7f36ac8d5700 10 monclient: get_monmap_and_config success 2023-09-30T01:36:02.085 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:02.070+0000 7f36ac8d5700 4 set_mon_vals no callback set 2023-09-30T01:36:02.085 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:02.070+0000 7f36a67fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:36:02.085 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:02.070+0000 7f36a67fc700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T01:36:02.085 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:02.070+0000 7f36a67fc700 10 monclient: dump: 2023-09-30T01:36:02.085 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:36:02.085 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:36:02.085 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:36:02.085 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:36:02.085 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:36:02.085 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:36:02.085 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:36:02.085 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:36:02.086 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:36:02.086 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:36:02.086 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:36:02.086 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:36:02.086 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:36:02.086 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:36:02.086 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:36:02.086 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:36:02.086 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:02.070+0000 7f36ac8d5700 10 monclient: shutdown 2023-09-30T01:36:02.086 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:02.070+0000 7f36ac8d5700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:36:02.086 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:02.070+0000 7f36ac8d5700 1 -- 172.21.15.88:0/4255346062 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f36a809cf30 msgr2=0x7f36a809d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:36:02.086 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:02.070+0000 7f36ac8d5700 1 --2- 172.21.15.88:0/4255346062 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f36a809cf30 0x7f36a809d300 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f369800bfc0 tx=0x7f369800d040 comp rx=0 tx=0).stop 2023-09-30T01:36:02.087 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:02.070+0000 7f36ac8d5700 1 -- 172.21.15.88:0/4255346062 shutdown_connections 2023-09-30T01:36:02.087 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:02.070+0000 7f36ac8d5700 1 --2- 172.21.15.88:0/4255346062 >> v2:172.21.15.88:3300/0 conn(0x7f36a809d840 0x7f36a808d110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:36:02.087 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:02.070+0000 7f36ac8d5700 1 --2- 172.21.15.88:0/4255346062 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f36a809cf30 0x7f36a809d300 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:36:02.087 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:02.070+0000 7f36ac8d5700 1 --2- 172.21.15.88:0/4255346062 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f36a8099a20 0x7f36a809b270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:36:02.087 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:02.070+0000 7f36ac8d5700 1 -- 172.21.15.88:0/4255346062 >> 172.21.15.88:0/4255346062 conn(0x7f36a80a2540 msgr2=0x7f36a80a2910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:36:02.087 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:02.070+0000 7f36ac8d5700 1 -- 172.21.15.88:0/4255346062 shutdown_connections 2023-09-30T01:36:02.087 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:02.070+0000 7f36ac8d5700 1 -- 172.21.15.88:0/4255346062 wait complete. 2023-09-30T01:36:02.092 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:02.074+0000 7f36ac8d5700 10 monclient: build_initial_monmap 2023-09-30T01:36:02.092 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:02.074+0000 7f36ac8d5700 10 monclient: monmap: 2023-09-30T01:36:02.093 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2023-09-30T01:36:02.093 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:36:02.093 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 0.000000 2023-09-30T01:36:02.093 INFO:teuthology.orchestra.run.smithi088.stderr:created 0.000000 2023-09-30T01:36:02.093 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2023-09-30T01:36:02.093 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 1 2023-09-30T01:36:02.093 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.noname-a 2023-09-30T01:36:02.093 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.noname-f 2023-09-30T01:36:02.093 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.noname-b 2023-09-30T01:36:02.093 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.noname-g 2023-09-30T01:36:02.093 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.noname-c 2023-09-30T01:36:02.094 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.noname-h 2023-09-30T01:36:02.094 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.noname-d 2023-09-30T01:36:02.094 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.noname-i 2023-09-30T01:36:02.094 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.noname-e 2023-09-30T01:36:02.094 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:36:02.094 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:02.074+0000 7f36ac8d5700 1 Processor -- start 2023-09-30T01:36:02.094 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:02.074+0000 7f36ac8d5700 1 -- start start 2023-09-30T01:36:02.094 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:02.074+0000 7f36ac8d5700 10 monclient: init 2023-09-30T01:36:02.094 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:02.074+0000 7f36ac8d5700 10 monclient: _reopen_session rank -1 2023-09-30T01:36:02.094 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:02.074+0000 7f36ac8d5700 10 monclient: _add_conns ranks=[5,0,7,6,2,8,1,4,3] 2023-09-30T01:36:02.094 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:02.074+0000 7f36ac8d5700 1 --2- >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7f36a8099a20 0x7f36a80990c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:36:02.094 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:02.074+0000 7f36ac8d5700 10 monclient(hunting): picked mon.noname-h con 0x7f36a8099a20 addr [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] 2023-09-30T01:36:02.095 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:02.074+0000 7f36ac8d5700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f36a809cf30 0x7f36a807c840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:36:02.095 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:02.074+0000 7f36ac8d5700 10 monclient(hunting): picked mon.noname-a con 0x7f36a809cf30 addr [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] 2023-09-30T01:36:02.095 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:02.074+0000 7f36ac8d5700 1 --2- >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7f36a809d840 0x7f36a807cd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:36:02.095 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:02.074+0000 7f36ac8d5700 10 monclient(hunting): picked mon.noname-i con 0x7f36a809d840 addr [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] 2023-09-30T01:36:02.095 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:02.074+0000 7f36ac8d5700 10 monclient(hunting): start opening mon connection 2023-09-30T01:36:02.095 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:02.074+0000 7f36ac8d5700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36a809bc70 con 0x7f36a809cf30 2023-09-30T01:36:02.095 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:02.074+0000 7f36ac8d5700 10 monclient(hunting): start opening mon connection 2023-09-30T01:36:02.095 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:02.074+0000 7f36ac8d5700 1 -- --> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] -- mon_getmap magic: 0 v1 -- 0x7f36a809a090 con 0x7f36a8099a20 2023-09-30T01:36:02.095 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:02.074+0000 7f36ac8d5700 10 monclient(hunting): start opening mon connection 2023-09-30T01:36:02.095 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:02.074+0000 7f36ac8d5700 1 -- --> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] -- mon_getmap magic: 0 v1 -- 0x7f36a8080c60 con 0x7f36a809d840 2023-09-30T01:36:02.095 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:02.074+0000 7f36ac8d5700 10 monclient(hunting): _renew_subs 2023-09-30T01:36:02.095 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:02.074+0000 7f36ac8d5700 10 monclient(hunting): authenticate will time out at 1034.115278s 2023-09-30T01:36:02.096 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:02.074+0000 7f36a6ffd700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f36a809cf30 0x7f36a807c840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:36:02.096 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:02.074+0000 7f36a6ffd700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f36a809cf30 0x7f36a807c840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:36650/0 (socket says 172.21.15.88:36650) 2023-09-30T01:36:02.096 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:02.074+0000 7f36a6ffd700 1 -- 172.21.15.88:0/4117192797 learned_addr learned my addr 172.21.15.88:0/4117192797 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-09-30T01:36:02.096 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:02.074+0000 7f36a6ffd700 10 monclient(hunting): get_auth_request con 0x7f36a809cf30 auth_method 0 2023-09-30T01:36:02.096 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:02.074+0000 7f36a6ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:36:02.096 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:02.074+0000 7f36a6ffd700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:36:02.096 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:02.074+0000 7f36a6ffd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:36:02.096 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:02.074+0000 7f36a77fe700 1 --2- 172.21.15.88:0/4117192797 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7f36a8099a20 0x7f36a80990c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:36:02.096 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:02.074+0000 7f36a77fe700 10 monclient(hunting): get_auth_request con 0x7f36a8099a20 auth_method 0 2023-09-30T01:36:02.096 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:02.074+0000 7f36a77fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:36:02.096 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:02.074+0000 7f36a77fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:36:02.097 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:02.074+0000 7f36a77fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:36:02.097 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:02.074+0000 7f36a7fff700 1 --2- 172.21.15.88:0/4117192797 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7f36a809d840 0x7f36a807cd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:36:02.097 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:02.074+0000 7f36a7fff700 10 monclient(hunting): get_auth_request con 0x7f36a809d840 auth_method 0 2023-09-30T01:36:02.097 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:02.074+0000 7f36a7fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:36:02.097 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:02.074+0000 7f36a7fff700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:36:02.097 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:02.074+0000 7f36a7fff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:36:02.097 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:02.074+0000 7f36a6ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:36:02.097 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:02.074+0000 7f36a6ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:36:02.097 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:02.074+0000 7f36a6ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:36:02.097 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:02.074+0000 7f36a77fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:36:02.097 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:02.074+0000 7f36a77fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:36:02.097 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:02.074+0000 7f36a77fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:36:02.098 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:02.074+0000 7f36a7fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:36:02.098 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:02.074+0000 7f36a7fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:36:02.098 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:02.078+0000 7f36a7fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:36:02.098 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:02.078+0000 7f36a6ffd700 10 monclient(hunting): handle_auth_done global_id 4896 payload 824 2023-09-30T01:36:02.098 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:02.078+0000 7f36a6ffd700 1 -- 172.21.15.88:0/4117192797 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7f36a809d840 msgr2=0x7f36a807cd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:36:02.098 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:02.078+0000 7f36a6ffd700 1 --2- 172.21.15.88:0/4117192797 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7f36a809d840 0x7f36a807cd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:36:02.098 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:02.078+0000 7f36a6ffd700 1 -- 172.21.15.88:0/4117192797 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7f36a8099a20 msgr2=0x7f36a80990c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:36:02.098 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:02.078+0000 7f36a6ffd700 1 --2- 172.21.15.88:0/4117192797 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7f36a8099a20 0x7f36a80990c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:36:02.098 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:02.078+0000 7f36a6ffd700 10 monclient: _finish_hunting 0 2023-09-30T01:36:02.098 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:02.078+0000 7f36a6ffd700 1 monclient: found mon.noname-a 2023-09-30T01:36:02.098 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:02.078+0000 7f36a6ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:36:02.098 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:02.078+0000 7f36a6ffd700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.88:3300/0 2023-09-30T01:36:02.099 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:02.078+0000 7f36a6ffd700 1 -- 172.21.15.88:0/4117192797 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f369800bb00 con 0x7f36a809cf30 2023-09-30T01:36:02.099 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:02.078+0000 7f36a77fe700 1 --2- 172.21.15.88:0/4117192797 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7f36a8099a20 0x7f36a80990c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:36:02.099 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:02.078+0000 7f36a6ffd700 1 --2- 172.21.15.88:0/4117192797 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f36a809cf30 0x7f36a807c840 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f369800a8d0 tx=0x7f3698016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:36:02.099 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:02.078+0000 7f368f7fe700 1 -- 172.21.15.88:0/4117192797 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f369800ad00 con 0x7f36a809cf30 2023-09-30T01:36:02.099 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:02.078+0000 7f368f7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:36:02.099 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:02.078+0000 7f368f7fe700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T01:36:02.099 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:02.078+0000 7f368f7fe700 10 monclient: dump: 2023-09-30T01:36:02.099 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:36:02.099 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:36:02.099 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:36:02.099 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:36:02.100 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:36:02.100 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:36:02.100 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:36:02.100 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:36:02.100 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:36:02.100 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:36:02.100 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:36:02.100 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:36:02.100 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:36:02.100 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:36:02.100 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:36:02.100 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:36:02.100 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:02.078+0000 7f368f7fe700 10 monclient: _finish_auth 0 2023-09-30T01:36:02.101 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:02.078+0000 7f368f7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:36:02.101 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:02.078+0000 7f368f7fe700 1 -- 172.21.15.88:0/4117192797 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f369801d890 con 0x7f36a809cf30 2023-09-30T01:36:02.101 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:02.078+0000 7f36ac8d5700 5 monclient: authenticate success, global_id 4896 2023-09-30T01:36:02.101 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:02.078+0000 7f36ac8d5700 10 monclient: _renew_subs 2023-09-30T01:36:02.101 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:02.078+0000 7f36ac8d5700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.88:3300/0 2023-09-30T01:36:02.101 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:02.078+0000 7f36ac8d5700 1 -- 172.21.15.88:0/4117192797 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f36a807e9f0 con 0x7f36a809cf30 2023-09-30T01:36:02.101 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:02.078+0000 7f368f7fe700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:36:02.101 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:02.078+0000 7f368f7fe700 1 -- 172.21.15.88:0/4117192797 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f369800aea0 con 0x7f36a809cf30 2023-09-30T01:36:02.101 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:02.078+0000 7f368f7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:36:02.101 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:02.078+0000 7f368f7fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:36:02.101 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:02.078+0000 7f368f7fe700 10 monclient: dump: 2023-09-30T01:36:02.101 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:36:02.102 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:36:02.102 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:36:02.102 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:36:02.102 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:36:02.102 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:36:02.102 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:36:02.102 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:36:02.102 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:36:02.102 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:36:02.102 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:36:02.102 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:36:02.102 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:36:02.103 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:36:02.103 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:36:02.103 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:36:02.103 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:02.078+0000 7f36ac8d5700 10 monclient: _renew_subs 2023-09-30T01:36:02.103 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:02.078+0000 7f36a5ffb700 4 set_mon_vals no callback set 2023-09-30T01:36:02.103 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:02.078+0000 7f36ac8d5700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.88:3300/0 2023-09-30T01:36:02.103 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:02.078+0000 7f36ac8d5700 1 -- 172.21.15.88:0/4117192797 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f36a807e470 con 0x7f36a809cf30 2023-09-30T01:36:02.103 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:02.078+0000 7f368f7fe700 1 -- 172.21.15.88:0/4117192797 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 40144+0+0 (secure 0 0 0) 0x7f369801a070 con 0x7f36a809cf30 2023-09-30T01:36:02.103 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:02.078+0000 7f36ac8d5700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:36:02.103 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:02.078+0000 7f36ac8d5700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:36:02.103 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:02.078+0000 7f36ac8d5700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.88:3300/0 2023-09-30T01:36:02.103 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:02.078+0000 7f36ac8d5700 1 -- 172.21.15.88:0/4117192797 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f36a80710d0 con 0x7f36a809cf30 2023-09-30T01:36:02.103 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:02.078+0000 7f368f7fe700 1 --2- 172.21.15.88:0/4117192797 >> v2:172.21.15.107:6812/29926 conn(0x7f3694033360 0x7f3694035810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:36:02.104 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:02.078+0000 7f368f7fe700 1 -- 172.21.15.88:0/4117192797 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 3502+0+0 (secure 0 0 0) 0x7f36980448a0 con 0x7f36a809cf30 2023-09-30T01:36:02.104 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:02.078+0000 7f36a77fe700 1 --2- 172.21.15.88:0/4117192797 >> v2:172.21.15.107:6812/29926 conn(0x7f3694033360 0x7f3694035810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:36:02.104 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:02.078+0000 7f36a77fe700 10 monclient: get_auth_request con 0x7f3694033360 auth_method 0 2023-09-30T01:36:02.104 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:02.081+0000 7f36a77fe700 1 --2- 172.21.15.88:0/4117192797 >> v2:172.21.15.107:6812/29926 conn(0x7f3694033360 0x7f3694035810 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f369c011480 tx=0x7f369c0091a0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:36:02.104 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:02.085+0000 7f368f7fe700 1 -- 172.21.15.88:0/4117192797 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f3698024360 con 0x7f36a809cf30 2023-09-30T01:36:02.104 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:02.085+0000 7f368f7fe700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:36:02.104 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:02.085+0000 7f368f7fe700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:36:02.286 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:02.281+0000 7f36ac8d5700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T01:36:02.286 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:02.281+0000 7f36ac8d5700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:36:02.286 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:02.281+0000 7f36ac8d5700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.88:3300/0 2023-09-30T01:36:02.286 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:02.281+0000 7f36ac8d5700 1 -- 172.21.15.88:0/4117192797 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f36a8067fc0 con 0x7f36a809cf30 2023-09-30T01:36:02.286 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:02.281+0000 7f368f7fe700 1 -- 172.21.15.88:0/4117192797 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f3698021080 con 0x7f36a809cf30 2023-09-30T01:36:02.287 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:02.281+0000 7f368f7fe700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:36:02.287 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:02.281+0000 7f368f7fe700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:36:02.287 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-09-30T01:36:02.287 INFO:teuthology.orchestra.run.smithi088.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:36:02.290 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:02.285+0000 7f36ac8d5700 1 -- 172.21.15.88:0/4117192797 >> v2:172.21.15.107:6812/29926 conn(0x7f3694033360 msgr2=0x7f3694035810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:36:02.290 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:02.285+0000 7f36ac8d5700 1 --2- 172.21.15.88:0/4117192797 >> v2:172.21.15.107:6812/29926 conn(0x7f3694033360 0x7f3694035810 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f369c011480 tx=0x7f369c0091a0 comp rx=0 tx=0).stop 2023-09-30T01:36:02.291 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:02.285+0000 7f36ac8d5700 10 monclient: shutdown 2023-09-30T01:36:02.291 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:02.285+0000 7f36ac8d5700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:36:02.291 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:02.285+0000 7f36ac8d5700 1 -- 172.21.15.88:0/4117192797 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f36a809cf30 msgr2=0x7f36a807c840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:36:02.291 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:02.285+0000 7f36ac8d5700 1 --2- 172.21.15.88:0/4117192797 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f36a809cf30 0x7f36a807c840 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f369800a8d0 tx=0x7f3698016000 comp rx=0 tx=0).stop 2023-09-30T01:36:02.291 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:02.285+0000 7f36ac8d5700 1 -- 172.21.15.88:0/4117192797 shutdown_connections 2023-09-30T01:36:02.291 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:02.285+0000 7f36ac8d5700 1 --2- 172.21.15.88:0/4117192797 >> v2:172.21.15.107:6812/29926 conn(0x7f3694033360 0x7f3694035810 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:36:02.291 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:02.285+0000 7f36ac8d5700 1 --2- 172.21.15.88:0/4117192797 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7f36a809d840 0x7f36a807cd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:36:02.291 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:02.285+0000 7f36ac8d5700 1 --2- 172.21.15.88:0/4117192797 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f36a809cf30 0x7f36a807c840 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:36:02.291 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:02.285+0000 7f36ac8d5700 1 --2- 172.21.15.88:0/4117192797 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7f36a8099a20 0x7f36a80990c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:36:02.291 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:02.285+0000 7f36ac8d5700 1 -- 172.21.15.88:0/4117192797 >> 172.21.15.88:0/4117192797 conn(0x7f36a80a2540 msgr2=0x7f36a80a18d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:36:02.292 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:02.285+0000 7f36ac8d5700 1 -- 172.21.15.88:0/4117192797 shutdown_connections 2023-09-30T01:36:02.292 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:02.285+0000 7f36ac8d5700 1 -- 172.21.15.88:0/4117192797 wait complete. 2023-09-30T01:36:05.305 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:36:05.421 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:05.413+0000 7fe4620ee700 10 monclient: get_monmap_and_config 2023-09-30T01:36:05.421 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:05.413+0000 7fe4620ee700 10 monclient: build_initial_monmap 2023-09-30T01:36:05.421 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:05.413+0000 7fe4620ee700 10 monclient: monmap: 2023-09-30T01:36:05.421 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2023-09-30T01:36:05.422 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:36:05.422 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:36:05.417169+0000 2023-09-30T01:36:05.422 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:36:05.417169+0000 2023-09-30T01:36:05.422 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2023-09-30T01:36:05.422 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:36:05.422 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a-legacy 2023-09-30T01:36:05.422 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.107:6789/0 mon.noname-f-legacy 2023-09-30T01:36:05.422 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.noname-b 2023-09-30T01:36:05.422 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.noname-g 2023-09-30T01:36:05.422 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.noname-c 2023-09-30T01:36:05.422 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.noname-h 2023-09-30T01:36:05.423 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.noname-d 2023-09-30T01:36:05.423 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.noname-i 2023-09-30T01:36:05.423 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.noname-e 2023-09-30T01:36:05.423 INFO:teuthology.orchestra.run.smithi088.stderr:9: v2:172.21.15.88:3300/0 mon.noname-a 2023-09-30T01:36:05.423 INFO:teuthology.orchestra.run.smithi088.stderr:10: v2:172.21.15.107:3300/0 mon.noname-f 2023-09-30T01:36:05.423 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:36:05.423 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:05.413+0000 7fe4620ee700 1 Processor -- start 2023-09-30T01:36:05.423 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:05.413+0000 7fe4620ee700 1 -- start start 2023-09-30T01:36:05.423 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:05.413+0000 7fe4620ee700 10 monclient: init 2023-09-30T01:36:05.423 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:05.413+0000 7fe4620ee700 10 monclient: _reopen_session rank -1 2023-09-30T01:36:05.423 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:05.413+0000 7fe4620ee700 10 monclient: _add_conns ranks=[4,5,0,3,8,2,9,10,6,7,1] 2023-09-30T01:36:05.423 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:05.413+0000 7fe4620ee700 1 --2- >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7fe45c09ed80 0x7fe45c0637f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:36:05.424 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:05.413+0000 7fe4620ee700 10 monclient(hunting): picked mon.noname-c con 0x7fe45c09ed80 addr [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] 2023-09-30T01:36:05.424 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:05.413+0000 7fe4620ee700 1 --2- >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7fe45c09f150 0x7fe45c09f520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:36:05.424 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:05.413+0000 7fe4620ee700 10 monclient(hunting): picked mon.noname-h con 0x7fe45c09f150 addr [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] 2023-09-30T01:36:05.424 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:05.413+0000 7fe4620ee700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fe45c131580 addr v1:172.21.15.88:6789/0 2023-09-30T01:36:05.424 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:05.413+0000 7fe4620ee700 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe45c04bb20 con 0x7fe45c131580 2023-09-30T01:36:05.424 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:05.413+0000 7fe4620ee700 10 monclient(hunting): start opening mon connection 2023-09-30T01:36:05.424 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:05.413+0000 7fe4620ee700 1 -- --> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] -- mon_getmap magic: 0 v1 -- 0x7fe45c09a110 con 0x7fe45c09ed80 2023-09-30T01:36:05.424 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:05.413+0000 7fe4620ee700 10 monclient(hunting): start opening mon connection 2023-09-30T01:36:05.424 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:05.413+0000 7fe4620ee700 1 -- --> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] -- mon_getmap magic: 0 v1 -- 0x7fe45c09d830 con 0x7fe45c09f150 2023-09-30T01:36:05.424 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:05.413+0000 7fe4620ee700 10 monclient(hunting): _renew_subs 2023-09-30T01:36:05.424 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:05.413+0000 7fe4620ee700 10 monclient(hunting): authenticate will time out at 1037.454733s 2023-09-30T01:36:05.424 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:05.413+0000 7fe4610ec700 1 --2- >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7fe45c09ed80 0x7fe45c0637f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:36:05.425 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:05.413+0000 7fe4618ed700 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7fe45c131580 0x7fe45c1319f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:49586/0 (socket says 172.21.15.88:49586) 2023-09-30T01:36:05.425 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:05.413+0000 7fe4618ed700 1 -- 172.21.15.88:0/3276149726 learned_addr learned my addr 172.21.15.88:0/3276149726 (peer_addr_for_me v1:172.21.15.88:0/0) 2023-09-30T01:36:05.425 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:05.413+0000 7fe4610ec700 10 monclient(hunting): get_auth_request con 0x7fe45c09ed80 auth_method 0 2023-09-30T01:36:05.425 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:05.413+0000 7fe4610ec700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:36:05.425 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:05.413+0000 7fe4610ec700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:36:05.425 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:05.413+0000 7fe4610ec700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:36:05.425 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:05.413+0000 7fe4610ec700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:36:05.425 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:05.413+0000 7fe4610ec700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:36:05.425 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:05.413+0000 7fe4610ec700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:36:05.425 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:05.413+0000 7fe4608eb700 1 --2- 172.21.15.88:0/3276149726 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7fe45c09f150 0x7fe45c09f520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:36:05.425 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:05.413+0000 7fe4608eb700 10 monclient(hunting): get_auth_request con 0x7fe45c09f150 auth_method 0 2023-09-30T01:36:05.425 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:05.413+0000 7fe4608eb700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:36:05.426 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:05.413+0000 7fe4608eb700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:36:05.426 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:05.413+0000 7fe4608eb700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:36:05.426 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:05.413+0000 7fe4610ec700 10 monclient(hunting): handle_auth_done global_id 4657 payload 274 2023-09-30T01:36:05.426 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:05.413+0000 7fe4610ec700 1 -- 172.21.15.88:0/3276149726 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7fe45c09f150 msgr2=0x7fe45c09f520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:36:05.426 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:05.413+0000 7fe4610ec700 1 --2- 172.21.15.88:0/3276149726 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7fe45c09f150 0x7fe45c09f520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:36:05.426 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:05.413+0000 7fe453fff700 1 -- 172.21.15.88:0/3276149726 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 606677956 0 0) 0x7fe45c04bb20 con 0x7fe45c131580 2023-09-30T01:36:05.426 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:05.413+0000 7fe4610ec700 1 -- 172.21.15.88:0/3276149726 >> v1:172.21.15.88:6789/0 conn(0x7fe45c131580 legacy=0x7fe45c1319f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:36:05.426 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:05.413+0000 7fe4610ec700 10 monclient: _finish_hunting 0 2023-09-30T01:36:05.426 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:05.413+0000 7fe4610ec700 1 monclient: found mon.noname-c 2023-09-30T01:36:05.426 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:05.413+0000 7fe4610ec700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:36:05.426 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:05.413+0000 7fe4610ec700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.88:3302/0 2023-09-30T01:36:05.426 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:05.413+0000 7fe4610ec700 1 -- 172.21.15.88:0/3276149726 --> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe45c063440 con 0x7fe45c09ed80 2023-09-30T01:36:05.427 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:05.413+0000 7fe453fff700 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2023-09-30T01:36:05.427 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:05.417+0000 7fe4610ec700 1 --2- 172.21.15.88:0/3276149726 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7fe45c09ed80 0x7fe45c0637f0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fe44c009900 tx=0x7fe44c0091a0 comp rx=0 tx=0).ready entity=mon.4 client_cookie=14ad18a70da6ec22 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:36:05.427 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:05.417+0000 7fe453fff700 1 -- 172.21.15.88:0/3276149726 <== mon.4 v2:172.21.15.88:3302/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7fe44c009210 con 0x7fe45c09ed80 2023-09-30T01:36:05.427 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:05.417+0000 7fe453fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:36:05.427 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:05.417+0000 7fe453fff700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T01:36:05.427 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:05.417+0000 7fe453fff700 10 monclient: dump: 2023-09-30T01:36:05.427 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:36:05.427 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:36:05.427 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:36:05.427 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:36:05.427 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:36:05.427 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:36:05.428 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:36:05.428 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:36:05.428 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:36:05.428 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:36:05.428 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:36:05.428 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:36:05.428 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:36:05.428 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:36:05.428 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:36:05.428 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:36:05.428 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:05.417+0000 7fe453fff700 10 monclient: _finish_auth 0 2023-09-30T01:36:05.428 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:05.417+0000 7fe453fff700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:36:05.429 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:05.417+0000 7fe4620ee700 5 monclient: authenticate success, global_id 4657 2023-09-30T01:36:05.429 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:05.417+0000 7fe4620ee700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:36:05.429 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:05.417+0000 7fe453fff700 1 -- 172.21.15.88:0/3276149726 <== mon.4 v2:172.21.15.88:3302/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe44c00bce0 con 0x7fe45c09ed80 2023-09-30T01:36:05.429 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:05.417+0000 7fe453fff700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:36:05.429 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:05.417+0000 7fe4620ee700 10 monclient: get_monmap_and_config success 2023-09-30T01:36:05.429 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:05.417+0000 7fe4620ee700 4 set_mon_vals no callback set 2023-09-30T01:36:05.429 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:05.417+0000 7fe453fff700 1 -- 172.21.15.88:0/3276149726 <== mon.4 v2:172.21.15.88:3302/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7fe44c00be50 con 0x7fe45c09ed80 2023-09-30T01:36:05.429 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:05.417+0000 7fe453fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:36:05.429 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:05.417+0000 7fe453fff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T01:36:05.429 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:05.417+0000 7fe453fff700 10 monclient: dump: 2023-09-30T01:36:05.429 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:36:05.429 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:36:05.429 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:36:05.430 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:36:05.430 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:36:05.430 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:36:05.430 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:36:05.430 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:36:05.430 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:36:05.430 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:36:05.430 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:36:05.430 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:36:05.430 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:36:05.430 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:36:05.431 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:36:05.431 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:36:05.431 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:05.417+0000 7fe4620ee700 10 monclient: shutdown 2023-09-30T01:36:05.431 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:05.417+0000 7fe4620ee700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:36:05.431 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:05.417+0000 7fe4620ee700 1 -- 172.21.15.88:0/3276149726 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7fe45c09ed80 msgr2=0x7fe45c0637f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:36:05.431 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:05.417+0000 7fe4620ee700 1 --2- 172.21.15.88:0/3276149726 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7fe45c09ed80 0x7fe45c0637f0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fe44c009900 tx=0x7fe44c0091a0 comp rx=0 tx=0).stop 2023-09-30T01:36:05.431 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:05.417+0000 7fe4620ee700 1 -- 172.21.15.88:0/3276149726 shutdown_connections 2023-09-30T01:36:05.431 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:05.417+0000 7fe4620ee700 1 --2- 172.21.15.88:0/3276149726 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7fe45c09f150 0x7fe45c09f520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:36:05.431 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:05.417+0000 7fe4620ee700 1 --2- 172.21.15.88:0/3276149726 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7fe45c09ed80 0x7fe45c0637f0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:36:05.431 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:05.417+0000 7fe4620ee700 1 -- 172.21.15.88:0/3276149726 >> 172.21.15.88:0/3276149726 conn(0x7fe45c0a0910 msgr2=0x7fe45c0a33c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:36:05.431 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:05.417+0000 7fe4620ee700 1 -- 172.21.15.88:0/3276149726 shutdown_connections 2023-09-30T01:36:05.431 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:05.417+0000 7fe4620ee700 1 -- 172.21.15.88:0/3276149726 wait complete. 2023-09-30T01:36:05.435 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:05.417+0000 7fe4620ee700 10 monclient: build_initial_monmap 2023-09-30T01:36:05.435 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:05.417+0000 7fe4620ee700 10 monclient: monmap: 2023-09-30T01:36:05.436 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2023-09-30T01:36:05.436 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:36:05.436 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 0.000000 2023-09-30T01:36:05.436 INFO:teuthology.orchestra.run.smithi088.stderr:created 0.000000 2023-09-30T01:36:05.436 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2023-09-30T01:36:05.436 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 1 2023-09-30T01:36:05.436 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.noname-a 2023-09-30T01:36:05.436 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.noname-f 2023-09-30T01:36:05.436 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.noname-b 2023-09-30T01:36:05.436 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.noname-g 2023-09-30T01:36:05.436 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.noname-c 2023-09-30T01:36:05.436 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.noname-h 2023-09-30T01:36:05.437 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.noname-d 2023-09-30T01:36:05.437 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.noname-i 2023-09-30T01:36:05.437 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.noname-e 2023-09-30T01:36:05.437 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:36:05.437 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:05.417+0000 7fe4620ee700 1 Processor -- start 2023-09-30T01:36:05.437 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:05.417+0000 7fe4620ee700 1 -- start start 2023-09-30T01:36:05.437 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:05.417+0000 7fe4620ee700 10 monclient: init 2023-09-30T01:36:05.437 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:05.417+0000 7fe4620ee700 10 monclient: _reopen_session rank -1 2023-09-30T01:36:05.437 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:05.417+0000 7fe4620ee700 10 monclient: _add_conns ranks=[1,7,2,0,8,5,4,6,3] 2023-09-30T01:36:05.437 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:05.417+0000 7fe4620ee700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe45c09ed80 0x7fe45c12ca70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:36:05.437 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:05.417+0000 7fe4620ee700 10 monclient(hunting): picked mon.noname-f con 0x7fe45c09ed80 addr [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] 2023-09-30T01:36:05.437 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:05.417+0000 7fe4620ee700 1 --2- >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7fe45c09f150 0x7fe45c1306e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:36:05.438 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:05.417+0000 7fe4620ee700 10 monclient(hunting): picked mon.noname-i con 0x7fe45c09f150 addr [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] 2023-09-30T01:36:05.438 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:05.417+0000 7fe4620ee700 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe45c131580 0x7fe45c1284f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:36:05.438 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:05.417+0000 7fe4620ee700 10 monclient(hunting): picked mon.noname-b con 0x7fe45c131580 addr [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] 2023-09-30T01:36:05.438 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:05.417+0000 7fe4620ee700 10 monclient(hunting): start opening mon connection 2023-09-30T01:36:05.438 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:05.417+0000 7fe4620ee700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe45c129d40 con 0x7fe45c09ed80 2023-09-30T01:36:05.438 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:05.417+0000 7fe4620ee700 10 monclient(hunting): start opening mon connection 2023-09-30T01:36:05.438 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:05.417+0000 7fe4620ee700 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe45c12cfb0 con 0x7fe45c131580 2023-09-30T01:36:05.438 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:05.417+0000 7fe4620ee700 10 monclient(hunting): start opening mon connection 2023-09-30T01:36:05.438 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:05.417+0000 7fe4620ee700 1 -- --> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] -- mon_getmap magic: 0 v1 -- 0x7fe45c09ccd0 con 0x7fe45c09f150 2023-09-30T01:36:05.438 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:05.417+0000 7fe4620ee700 10 monclient(hunting): _renew_subs 2023-09-30T01:36:05.438 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:05.417+0000 7fe4620ee700 10 monclient(hunting): authenticate will time out at 1037.458796s 2023-09-30T01:36:05.439 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:05.417+0000 7fe4618ed700 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe45c131580 0x7fe45c1284f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:36:05.439 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:05.417+0000 7fe4610ec700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe45c09ed80 0x7fe45c12ca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:36:05.439 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:05.417+0000 7fe4618ed700 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe45c131580 0x7fe45c1284f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3301/0 says I am v2:172.21.15.88:51520/0 (socket says 172.21.15.88:51520) 2023-09-30T01:36:05.439 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:05.417+0000 7fe4618ed700 1 -- 172.21.15.88:0/2686455963 learned_addr learned my addr 172.21.15.88:0/2686455963 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-09-30T01:36:05.439 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:05.417+0000 7fe4618ed700 10 monclient(hunting): get_auth_request con 0x7fe45c131580 auth_method 0 2023-09-30T01:36:05.439 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:05.417+0000 7fe4618ed700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:36:05.439 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:05.417+0000 7fe4618ed700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:36:05.439 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:05.417+0000 7fe4618ed700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:36:05.439 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:05.417+0000 7fe4608eb700 1 --2- 172.21.15.88:0/2686455963 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7fe45c09f150 0x7fe45c1306e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:36:05.439 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:05.417+0000 7fe4610ec700 10 monclient(hunting): get_auth_request con 0x7fe45c09ed80 auth_method 0 2023-09-30T01:36:05.439 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:05.417+0000 7fe4610ec700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:36:05.439 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:05.417+0000 7fe4610ec700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:36:05.440 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:05.417+0000 7fe4610ec700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:36:05.440 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:05.417+0000 7fe4608eb700 10 monclient(hunting): get_auth_request con 0x7fe45c09f150 auth_method 0 2023-09-30T01:36:05.440 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:05.417+0000 7fe4608eb700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:36:05.440 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:05.417+0000 7fe4608eb700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:36:05.440 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:05.417+0000 7fe4608eb700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:36:05.440 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:05.417+0000 7fe4618ed700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:36:05.440 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:05.417+0000 7fe4618ed700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:36:05.440 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:05.417+0000 7fe4618ed700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:36:05.440 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:05.417+0000 7fe4610ec700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:36:05.440 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:05.417+0000 7fe4610ec700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:36:05.440 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:05.417+0000 7fe4610ec700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:36:05.440 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:05.417+0000 7fe4608eb700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:36:05.441 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:05.417+0000 7fe4608eb700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:36:05.441 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:05.417+0000 7fe4608eb700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:36:05.441 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:05.417+0000 7fe4618ed700 10 monclient(hunting): handle_auth_done global_id 4637 payload 824 2023-09-30T01:36:05.441 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:05.417+0000 7fe4618ed700 1 -- 172.21.15.88:0/2686455963 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7fe45c09f150 msgr2=0x7fe45c1306e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:36:05.441 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:05.417+0000 7fe4618ed700 1 --2- 172.21.15.88:0/2686455963 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7fe45c09f150 0x7fe45c1306e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:36:05.441 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:05.417+0000 7fe4618ed700 1 -- 172.21.15.88:0/2686455963 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe45c09ed80 msgr2=0x7fe45c12ca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:36:05.441 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:05.417+0000 7fe4618ed700 1 --2- 172.21.15.88:0/2686455963 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe45c09ed80 0x7fe45c12ca70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:36:05.441 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:05.417+0000 7fe4618ed700 10 monclient: _finish_hunting 0 2023-09-30T01:36:05.441 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:05.417+0000 7fe4618ed700 1 monclient: found mon.noname-b 2023-09-30T01:36:05.441 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:05.417+0000 7fe4618ed700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:36:05.441 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:05.417+0000 7fe4618ed700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.88:3301/0 2023-09-30T01:36:05.442 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:05.421+0000 7fe4618ed700 1 -- 172.21.15.88:0/2686455963 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe44c0093c0 con 0x7fe45c131580 2023-09-30T01:36:05.442 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:05.421+0000 7fe4618ed700 1 --2- 172.21.15.88:0/2686455963 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe45c131580 0x7fe45c1284f0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fe45400c4f0 tx=0x7fe454007f20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:36:05.442 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:05.421+0000 7fe4517fa700 1 -- 172.21.15.88:0/2686455963 <== mon.2 v2:172.21.15.88:3301/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7fe45400ce60 con 0x7fe45c131580 2023-09-30T01:36:05.442 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:05.421+0000 7fe4517fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:36:05.442 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:05.421+0000 7fe4517fa700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T01:36:05.442 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:05.421+0000 7fe4517fa700 10 monclient: dump: 2023-09-30T01:36:05.442 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:36:05.442 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:36:05.442 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:36:05.442 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:36:05.442 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:36:05.442 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:36:05.443 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:36:05.443 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:36:05.443 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:36:05.443 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:36:05.443 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:36:05.443 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:36:05.443 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:36:05.443 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:36:05.443 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:36:05.443 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:36:05.443 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:05.421+0000 7fe4517fa700 10 monclient: _finish_auth 0 2023-09-30T01:36:05.443 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:05.421+0000 7fe4517fa700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:36:05.443 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:05.421+0000 7fe4620ee700 5 monclient: authenticate success, global_id 4637 2023-09-30T01:36:05.444 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:05.421+0000 7fe4620ee700 10 monclient: _renew_subs 2023-09-30T01:36:05.444 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:05.421+0000 7fe4620ee700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.88:3301/0 2023-09-30T01:36:05.444 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:05.421+0000 7fe4620ee700 1 -- 172.21.15.88:0/2686455963 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe45c12a470 con 0x7fe45c131580 2023-09-30T01:36:05.444 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:05.421+0000 7fe4517fa700 1 -- 172.21.15.88:0/2686455963 <== mon.2 v2:172.21.15.88:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe454005b30 con 0x7fe45c131580 2023-09-30T01:36:05.444 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:05.421+0000 7fe4517fa700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:36:05.444 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:05.421+0000 7fe4620ee700 10 monclient: _renew_subs 2023-09-30T01:36:05.444 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:05.421+0000 7fe4620ee700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.88:3301/0 2023-09-30T01:36:05.444 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:05.421+0000 7fe4620ee700 1 -- 172.21.15.88:0/2686455963 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe45c12bac0 con 0x7fe45c131580 2023-09-30T01:36:05.444 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:05.421+0000 7fe452ffd700 4 set_mon_vals no callback set 2023-09-30T01:36:05.444 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:05.421+0000 7fe4517fa700 1 -- 172.21.15.88:0/2686455963 <== mon.2 v2:172.21.15.88:3301/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7fe454005ca0 con 0x7fe45c131580 2023-09-30T01:36:05.444 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:05.421+0000 7fe4517fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:36:05.444 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:05.421+0000 7fe4517fa700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T01:36:05.445 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:05.421+0000 7fe4517fa700 10 monclient: dump: 2023-09-30T01:36:05.445 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:36:05.445 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:36:05.445 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:36:05.445 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:36:05.445 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:36:05.445 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:36:05.445 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:36:05.445 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:36:05.445 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:36:05.445 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:36:05.445 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:36:05.446 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:36:05.446 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:36:05.446 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:36:05.446 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:36:05.446 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:36:05.446 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:05.421+0000 7fe4517fa700 1 -- 172.21.15.88:0/2686455963 <== mon.2 v2:172.21.15.88:3301/0 4 ==== mgrmap(e 5) v1 ==== 40144+0+0 (secure 0 0 0) 0x7fe454007070 con 0x7fe45c131580 2023-09-30T01:36:05.446 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:05.421+0000 7fe4517fa700 1 --2- 172.21.15.88:0/2686455963 >> v2:172.21.15.107:6812/29926 conn(0x7fe440033360 0x7fe440035810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:36:05.446 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:05.421+0000 7fe4517fa700 1 -- 172.21.15.88:0/2686455963 <== mon.2 v2:172.21.15.88:3301/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 3502+0+0 (secure 0 0 0) 0x7fe4540424a0 con 0x7fe45c131580 2023-09-30T01:36:05.446 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:05.421+0000 7fe4610ec700 1 --2- 172.21.15.88:0/2686455963 >> v2:172.21.15.107:6812/29926 conn(0x7fe440033360 0x7fe440035810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:36:05.446 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:05.421+0000 7fe4610ec700 10 monclient: get_auth_request con 0x7fe440033360 auth_method 0 2023-09-30T01:36:05.446 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:05.421+0000 7fe4620ee700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:36:05.446 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:05.421+0000 7fe4620ee700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:36:05.446 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:05.421+0000 7fe4620ee700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.88:3301/0 2023-09-30T01:36:05.447 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:05.421+0000 7fe4620ee700 1 -- 172.21.15.88:0/2686455963 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe45c071960 con 0x7fe45c131580 2023-09-30T01:36:05.447 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:05.421+0000 7fe4610ec700 1 --2- 172.21.15.88:0/2686455963 >> v2:172.21.15.107:6812/29926 conn(0x7fe440033360 0x7fe440035810 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fe44c009fd0 tx=0x7fe44c017000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:36:05.447 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:05.425+0000 7fe4517fa700 1 -- 172.21.15.88:0/2686455963 <== mon.2 v2:172.21.15.88:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fe454017070 con 0x7fe45c131580 2023-09-30T01:36:05.447 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:05.429+0000 7fe4517fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:36:05.447 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:05.429+0000 7fe4517fa700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:36:05.631 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:05.625+0000 7fe4620ee700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T01:36:05.631 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:05.625+0000 7fe4620ee700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:36:05.631 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:05.625+0000 7fe4620ee700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.88:3301/0 2023-09-30T01:36:05.631 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:05.625+0000 7fe4620ee700 1 -- 172.21.15.88:0/2686455963 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fe45c12a8a0 con 0x7fe45c131580 2023-09-30T01:36:05.631 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:05.625+0000 7fe4517fa700 1 -- 172.21.15.88:0/2686455963 <== mon.2 v2:172.21.15.88:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7fe454018630 con 0x7fe45c131580 2023-09-30T01:36:05.631 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:05.625+0000 7fe4517fa700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:36:05.631 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:05.625+0000 7fe4517fa700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:36:05.632 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-09-30T01:36:05.632 INFO:teuthology.orchestra.run.smithi088.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:36:05.634 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:05.629+0000 7fe4620ee700 1 -- 172.21.15.88:0/2686455963 >> v2:172.21.15.107:6812/29926 conn(0x7fe440033360 msgr2=0x7fe440035810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:36:05.634 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:05.629+0000 7fe4620ee700 1 --2- 172.21.15.88:0/2686455963 >> v2:172.21.15.107:6812/29926 conn(0x7fe440033360 0x7fe440035810 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fe44c009fd0 tx=0x7fe44c017000 comp rx=0 tx=0).stop 2023-09-30T01:36:05.634 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:05.629+0000 7fe4620ee700 10 monclient: shutdown 2023-09-30T01:36:05.634 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:05.629+0000 7fe4620ee700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:36:05.634 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:05.629+0000 7fe4620ee700 1 -- 172.21.15.88:0/2686455963 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe45c131580 msgr2=0x7fe45c1284f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:36:05.634 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:05.629+0000 7fe4620ee700 1 --2- 172.21.15.88:0/2686455963 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe45c131580 0x7fe45c1284f0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fe45400c4f0 tx=0x7fe454007f20 comp rx=0 tx=0).stop 2023-09-30T01:36:05.634 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:05.629+0000 7fe4620ee700 1 -- 172.21.15.88:0/2686455963 shutdown_connections 2023-09-30T01:36:05.634 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:05.629+0000 7fe4620ee700 1 --2- 172.21.15.88:0/2686455963 >> v2:172.21.15.107:6812/29926 conn(0x7fe440033360 0x7fe440035810 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:36:05.635 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:05.629+0000 7fe4620ee700 1 --2- 172.21.15.88:0/2686455963 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe45c131580 0x7fe45c1284f0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:36:05.635 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:05.629+0000 7fe4620ee700 1 --2- 172.21.15.88:0/2686455963 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7fe45c09f150 0x7fe45c1306e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:36:05.635 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:05.629+0000 7fe4620ee700 1 --2- 172.21.15.88:0/2686455963 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe45c09ed80 0x7fe45c12ca70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:36:05.635 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:05.629+0000 7fe4620ee700 1 -- 172.21.15.88:0/2686455963 >> 172.21.15.88:0/2686455963 conn(0x7fe45c0a0910 msgr2=0x7fe45c09c0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:36:05.635 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:05.629+0000 7fe4620ee700 1 -- 172.21.15.88:0/2686455963 shutdown_connections 2023-09-30T01:36:05.635 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:05.629+0000 7fe4620ee700 1 -- 172.21.15.88:0/2686455963 wait complete. 2023-09-30T01:36:08.645 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:36:08.756 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:08.753+0000 7f499db2f700 10 monclient: get_monmap_and_config 2023-09-30T01:36:08.756 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:08.753+0000 7f499db2f700 10 monclient: build_initial_monmap 2023-09-30T01:36:08.756 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:08.753+0000 7f499db2f700 10 monclient: monmap: 2023-09-30T01:36:08.756 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2023-09-30T01:36:08.756 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:36:08.757 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:36:08.756310+0000 2023-09-30T01:36:08.757 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:36:08.756310+0000 2023-09-30T01:36:08.757 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2023-09-30T01:36:08.757 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:36:08.757 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a-legacy 2023-09-30T01:36:08.757 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.107:6789/0 mon.noname-f-legacy 2023-09-30T01:36:08.757 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.noname-b 2023-09-30T01:36:08.757 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.noname-g 2023-09-30T01:36:08.757 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.noname-c 2023-09-30T01:36:08.757 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.noname-h 2023-09-30T01:36:08.757 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.noname-d 2023-09-30T01:36:08.757 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.noname-i 2023-09-30T01:36:08.758 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.noname-e 2023-09-30T01:36:08.758 INFO:teuthology.orchestra.run.smithi088.stderr:9: v2:172.21.15.88:3300/0 mon.noname-a 2023-09-30T01:36:08.758 INFO:teuthology.orchestra.run.smithi088.stderr:10: v2:172.21.15.107:3300/0 mon.noname-f 2023-09-30T01:36:08.758 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:36:08.766 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:08.753+0000 7f499db2f700 1 Processor -- start 2023-09-30T01:36:08.767 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:08.753+0000 7f499db2f700 1 -- start start 2023-09-30T01:36:08.767 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:08.753+0000 7f499db2f700 10 monclient: init 2023-09-30T01:36:08.767 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:08.753+0000 7f499db2f700 10 monclient: _reopen_session rank -1 2023-09-30T01:36:08.767 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:08.753+0000 7f499db2f700 10 monclient: _add_conns ranks=[8,5,7,2,4,6,0,10,3,1,9] 2023-09-30T01:36:08.767 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:08.753+0000 7f499db2f700 1 --2- >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f4998097a40 0x7f4998132700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:36:08.767 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:08.753+0000 7f499db2f700 10 monclient(hunting): picked mon.noname-e con 0x7f4998097a40 addr [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] 2023-09-30T01:36:08.767 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:08.753+0000 7f499db2f700 1 --2- >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7f49981311c0 0x7f4998132c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:36:08.767 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:08.753+0000 7f499db2f700 10 monclient(hunting): picked mon.noname-h con 0x7f49981311c0 addr [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] 2023-09-30T01:36:08.767 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:08.753+0000 7f499db2f700 1 --2- >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7f49981332e0 0x7f4998133710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:36:08.767 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:08.753+0000 7f499db2f700 10 monclient(hunting): picked mon.noname-i con 0x7f49981332e0 addr [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] 2023-09-30T01:36:08.768 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:08.753+0000 7f499db2f700 10 monclient(hunting): start opening mon connection 2023-09-30T01:36:08.768 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:08.753+0000 7f499db2f700 1 -- --> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] -- mon_getmap magic: 0 v1 -- 0x7f4998131590 con 0x7f49981311c0 2023-09-30T01:36:08.768 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:08.753+0000 7f499db2f700 10 monclient(hunting): start opening mon connection 2023-09-30T01:36:08.768 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:08.753+0000 7f499db2f700 1 -- --> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] -- mon_getmap magic: 0 v1 -- 0x7f499809a5d0 con 0x7f49981332e0 2023-09-30T01:36:08.768 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:08.753+0000 7f499db2f700 10 monclient(hunting): start opening mon connection 2023-09-30T01:36:08.768 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:08.753+0000 7f499db2f700 1 -- --> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] -- mon_getmap magic: 0 v1 -- 0x7f4998097e10 con 0x7f4998097a40 2023-09-30T01:36:08.768 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:08.753+0000 7f499db2f700 10 monclient(hunting): _renew_subs 2023-09-30T01:36:08.768 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:08.753+0000 7f499db2f700 10 monclient(hunting): authenticate will time out at 1040.793749s 2023-09-30T01:36:08.768 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:08.753+0000 7f499cb2d700 1 --2- >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f4998097a40 0x7f4998132700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:36:08.768 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:08.753+0000 7f499cb2d700 1 --2- >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f4998097a40 0x7f4998132700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3304/0 says I am v2:172.21.15.88:46736/0 (socket says 172.21.15.88:46736) 2023-09-30T01:36:08.768 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:08.753+0000 7f499cb2d700 1 -- 172.21.15.88:0/248888521 learned_addr learned my addr 172.21.15.88:0/248888521 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-09-30T01:36:08.768 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:08.753+0000 7f4997fff700 1 --2- 172.21.15.88:0/248888521 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7f49981311c0 0x7f4998132c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:36:08.769 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:08.753+0000 7f499cb2d700 10 monclient(hunting): get_auth_request con 0x7f4998097a40 auth_method 0 2023-09-30T01:36:08.769 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:08.753+0000 7f499cb2d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:36:08.769 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:08.753+0000 7f499cb2d700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:36:08.769 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:08.753+0000 7f499cb2d700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:36:08.769 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:08.753+0000 7f499d32e700 1 --2- 172.21.15.88:0/248888521 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7f49981332e0 0x7f4998133710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:36:08.769 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:08.753+0000 7f4997fff700 10 monclient(hunting): get_auth_request con 0x7f49981311c0 auth_method 0 2023-09-30T01:36:08.769 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:08.753+0000 7f4997fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:36:08.769 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:08.753+0000 7f4997fff700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:36:08.769 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:08.753+0000 7f4997fff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:36:08.769 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:08.753+0000 7f499d32e700 10 monclient(hunting): get_auth_request con 0x7f49981332e0 auth_method 0 2023-09-30T01:36:08.769 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:08.753+0000 7f499d32e700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:36:08.769 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:08.753+0000 7f499d32e700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:36:08.770 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:08.753+0000 7f499d32e700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:36:08.770 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:08.753+0000 7f4997fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:36:08.770 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:08.753+0000 7f4997fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:36:08.770 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:08.753+0000 7f4997fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:36:08.770 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:08.753+0000 7f499d32e700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:36:08.770 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:08.753+0000 7f499d32e700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:36:08.770 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:08.753+0000 7f499d32e700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:36:08.770 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:08.753+0000 7f499cb2d700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:36:08.770 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:08.753+0000 7f499cb2d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:36:08.770 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:08.753+0000 7f499cb2d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:36:08.770 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:08.753+0000 7f4997fff700 10 monclient(hunting): handle_auth_done global_id 4631 payload 274 2023-09-30T01:36:08.770 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:08.753+0000 7f4997fff700 1 -- 172.21.15.88:0/248888521 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f4998097a40 msgr2=0x7f4998132700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:36:08.771 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:08.753+0000 7f4997fff700 1 --2- 172.21.15.88:0/248888521 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f4998097a40 0x7f4998132700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:36:08.771 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:08.753+0000 7f4997fff700 1 -- 172.21.15.88:0/248888521 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7f49981332e0 msgr2=0x7f4998133710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:36:08.771 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:08.753+0000 7f4997fff700 1 --2- 172.21.15.88:0/248888521 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7f49981332e0 0x7f4998133710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:36:08.771 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:08.753+0000 7f4997fff700 10 monclient: _finish_hunting 0 2023-09-30T01:36:08.771 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:08.753+0000 7f4997fff700 1 monclient: found mon.noname-h 2023-09-30T01:36:08.771 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:08.753+0000 7f4997fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:36:08.771 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:08.753+0000 7f4997fff700 10 monclient: _send_mon_message to mon.noname-h at v2:172.21.15.107:3302/0 2023-09-30T01:36:08.771 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:08.753+0000 7f4997fff700 1 -- 172.21.15.88:0/248888521 --> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f499804bb20 con 0x7f49981311c0 2023-09-30T01:36:08.771 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:08.753+0000 7f499d32e700 1 --2- 172.21.15.88:0/248888521 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7f49981332e0 0x7f4998133710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:36:08.771 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:08.753+0000 7f4997fff700 1 --2- 172.21.15.88:0/248888521 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7f49981311c0 0x7f4998132c40 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f4988009910 tx=0x7f49880091a0 comp rx=0 tx=0).ready entity=mon.5 client_cookie=e37912f335b58eb5 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:36:08.771 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:08.753+0000 7f49977fe700 1 -- 172.21.15.88:0/248888521 <== mon.5 v2:172.21.15.107:3302/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f4988011ab0 con 0x7f49981311c0 2023-09-30T01:36:08.772 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:08.753+0000 7f49977fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:36:08.772 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:08.753+0000 7f49977fe700 10 monclient: got monmap 1 from mon.noname-h (according to old e1) 2023-09-30T01:36:08.772 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:08.753+0000 7f49977fe700 10 monclient: dump: 2023-09-30T01:36:08.772 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:36:08.772 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:36:08.772 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:36:08.772 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:36:08.772 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:36:08.772 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:36:08.772 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:36:08.772 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:36:08.772 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:36:08.773 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:36:08.773 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:36:08.773 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:36:08.773 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:36:08.773 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:36:08.773 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:36:08.773 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:36:08.773 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:08.753+0000 7f49977fe700 10 monclient: _finish_auth 0 2023-09-30T01:36:08.773 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:08.753+0000 7f49977fe700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:36:08.773 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:08.753+0000 7f499db2f700 5 monclient: authenticate success, global_id 4631 2023-09-30T01:36:08.773 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:08.753+0000 7f499db2f700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:36:08.773 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:08.753+0000 7f49977fe700 1 -- 172.21.15.88:0/248888521 <== mon.5 v2:172.21.15.107:3302/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4988011c30 con 0x7f49981311c0 2023-09-30T01:36:08.774 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:08.753+0000 7f49977fe700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:36:08.774 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:08.753+0000 7f499db2f700 10 monclient: get_monmap_and_config success 2023-09-30T01:36:08.774 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:08.753+0000 7f499db2f700 4 set_mon_vals no callback set 2023-09-30T01:36:08.774 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:08.753+0000 7f49977fe700 1 -- 172.21.15.88:0/248888521 <== mon.5 v2:172.21.15.107:3302/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f4988011da0 con 0x7f49981311c0 2023-09-30T01:36:08.774 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:08.753+0000 7f49977fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:36:08.774 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:08.753+0000 7f49977fe700 10 monclient: got monmap 1 from mon.h (according to old e1) 2023-09-30T01:36:08.774 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:08.753+0000 7f49977fe700 10 monclient: dump: 2023-09-30T01:36:08.774 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:36:08.774 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:36:08.774 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:36:08.774 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:36:08.774 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:36:08.775 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:36:08.775 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:36:08.775 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:36:08.775 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:36:08.775 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:36:08.775 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:36:08.775 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:36:08.775 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:36:08.775 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:36:08.775 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:36:08.775 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:36:08.775 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:08.757+0000 7f499db2f700 10 monclient: shutdown 2023-09-30T01:36:08.776 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:08.757+0000 7f499db2f700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:36:08.776 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:08.757+0000 7f499db2f700 1 -- 172.21.15.88:0/248888521 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7f49981311c0 msgr2=0x7f4998132c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:36:08.776 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:08.757+0000 7f499db2f700 1 --2- 172.21.15.88:0/248888521 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7f49981311c0 0x7f4998132c40 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f4988009910 tx=0x7f49880091a0 comp rx=0 tx=0).stop 2023-09-30T01:36:08.776 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:08.757+0000 7f499db2f700 1 -- 172.21.15.88:0/248888521 shutdown_connections 2023-09-30T01:36:08.776 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:08.757+0000 7f499db2f700 1 --2- 172.21.15.88:0/248888521 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7f49981332e0 0x7f4998133710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:36:08.776 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:08.757+0000 7f499db2f700 1 --2- 172.21.15.88:0/248888521 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7f49981311c0 0x7f4998132c40 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:36:08.776 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:08.757+0000 7f499db2f700 1 --2- 172.21.15.88:0/248888521 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f4998097a40 0x7f4998132700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:36:08.776 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:08.757+0000 7f499db2f700 1 -- 172.21.15.88:0/248888521 >> 172.21.15.88:0/248888521 conn(0x7f499806e5e0 msgr2=0x7f499806e9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:36:08.776 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:08.757+0000 7f499db2f700 1 -- 172.21.15.88:0/248888521 shutdown_connections 2023-09-30T01:36:08.776 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:08.757+0000 7f499db2f700 1 -- 172.21.15.88:0/248888521 wait complete. 2023-09-30T01:36:08.776 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:08.757+0000 7f499db2f700 10 monclient: build_initial_monmap 2023-09-30T01:36:08.776 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:08.757+0000 7f499db2f700 10 monclient: monmap: 2023-09-30T01:36:08.776 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2023-09-30T01:36:08.777 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:36:08.777 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 0.000000 2023-09-30T01:36:08.777 INFO:teuthology.orchestra.run.smithi088.stderr:created 0.000000 2023-09-30T01:36:08.777 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2023-09-30T01:36:08.777 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 1 2023-09-30T01:36:08.777 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.noname-a 2023-09-30T01:36:08.777 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.noname-f 2023-09-30T01:36:08.777 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.noname-b 2023-09-30T01:36:08.777 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.noname-g 2023-09-30T01:36:08.777 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.noname-c 2023-09-30T01:36:08.777 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.noname-h 2023-09-30T01:36:08.777 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.noname-d 2023-09-30T01:36:08.778 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.noname-i 2023-09-30T01:36:08.778 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.noname-e 2023-09-30T01:36:08.778 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:36:08.778 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:08.757+0000 7f499db2f700 1 Processor -- start 2023-09-30T01:36:08.778 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:08.757+0000 7f499db2f700 1 -- start start 2023-09-30T01:36:08.778 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:08.757+0000 7f499db2f700 10 monclient: init 2023-09-30T01:36:08.778 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:08.757+0000 7f499db2f700 10 monclient: _reopen_session rank -1 2023-09-30T01:36:08.778 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:08.757+0000 7f499db2f700 10 monclient: _add_conns ranks=[0,7,3,6,4,1,8,2,5] 2023-09-30T01:36:08.778 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:08.757+0000 7f499db2f700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4998097a40 0x7f49980a1e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:36:08.778 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:08.757+0000 7f499db2f700 10 monclient(hunting): picked mon.noname-a con 0x7f4998097a40 addr [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] 2023-09-30T01:36:08.778 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:08.757+0000 7f499db2f700 1 --2- >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7f49981311c0 0x7f49980a23c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:36:08.778 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:08.757+0000 7f499db2f700 10 monclient(hunting): picked mon.noname-i con 0x7f49981311c0 addr [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] 2023-09-30T01:36:08.779 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:08.757+0000 7f499db2f700 1 --2- >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7f49981332e0 0x7f49980a2900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:36:08.779 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:08.757+0000 7f499db2f700 10 monclient(hunting): picked mon.noname-g con 0x7f49981332e0 addr [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] 2023-09-30T01:36:08.779 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:08.757+0000 7f499db2f700 10 monclient(hunting): start opening mon connection 2023-09-30T01:36:08.779 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:08.757+0000 7f499db2f700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f499811ef10 con 0x7f4998097a40 2023-09-30T01:36:08.779 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:08.757+0000 7f499db2f700 10 monclient(hunting): start opening mon connection 2023-09-30T01:36:08.779 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:08.757+0000 7f499db2f700 1 -- --> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f499811f440 con 0x7f49981332e0 2023-09-30T01:36:08.779 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:08.757+0000 7f499db2f700 10 monclient(hunting): start opening mon connection 2023-09-30T01:36:08.779 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:08.757+0000 7f499db2f700 1 -- --> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] -- mon_getmap magic: 0 v1 -- 0x7f4998099c30 con 0x7f49981311c0 2023-09-30T01:36:08.779 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:08.757+0000 7f499db2f700 10 monclient(hunting): _renew_subs 2023-09-30T01:36:08.779 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:08.757+0000 7f499db2f700 10 monclient(hunting): authenticate will time out at 1040.797947s 2023-09-30T01:36:08.779 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:08.757+0000 7f499cb2d700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4998097a40 0x7f49980a1e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:36:08.779 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:08.757+0000 7f499cb2d700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4998097a40 0x7f49980a1e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:36662/0 (socket says 172.21.15.88:36662) 2023-09-30T01:36:08.780 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:08.757+0000 7f499cb2d700 1 -- 172.21.15.88:0/2628527359 learned_addr learned my addr 172.21.15.88:0/2628527359 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-09-30T01:36:08.780 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:08.757+0000 7f499cb2d700 10 monclient(hunting): get_auth_request con 0x7f4998097a40 auth_method 0 2023-09-30T01:36:08.780 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:08.757+0000 7f499cb2d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:36:08.780 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:08.757+0000 7f499cb2d700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:36:08.780 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:08.757+0000 7f499cb2d700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:36:08.780 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:08.757+0000 7f499d32e700 1 --2- 172.21.15.88:0/2628527359 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7f49981332e0 0x7f49980a2900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:36:08.780 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:08.757+0000 7f4997fff700 1 --2- 172.21.15.88:0/2628527359 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7f49981311c0 0x7f49980a23c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:36:08.780 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:08.757+0000 7f499d32e700 10 monclient(hunting): get_auth_request con 0x7f49981332e0 auth_method 0 2023-09-30T01:36:08.780 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:08.757+0000 7f499d32e700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:36:08.780 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:08.757+0000 7f499d32e700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:36:08.780 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:08.757+0000 7f499d32e700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:36:08.781 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:08.757+0000 7f499cb2d700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:36:08.781 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:08.757+0000 7f499cb2d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:36:08.781 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:08.757+0000 7f499cb2d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:36:08.781 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:08.757+0000 7f4997fff700 10 monclient(hunting): get_auth_request con 0x7f49981311c0 auth_method 0 2023-09-30T01:36:08.781 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:08.757+0000 7f4997fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:36:08.781 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:08.757+0000 7f4997fff700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:36:08.781 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:08.757+0000 7f4997fff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:36:08.781 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:08.757+0000 7f499d32e700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:36:08.781 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:08.757+0000 7f499d32e700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:36:08.781 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:08.757+0000 7f499d32e700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:36:08.781 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:08.757+0000 7f4997fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:36:08.781 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:08.757+0000 7f4997fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:36:08.782 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:08.757+0000 7f4997fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:36:08.782 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:08.757+0000 7f499cb2d700 10 monclient(hunting): handle_auth_done global_id 4914 payload 824 2023-09-30T01:36:08.782 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:08.757+0000 7f499cb2d700 1 -- 172.21.15.88:0/2628527359 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7f49981311c0 msgr2=0x7f49980a23c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:36:08.782 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:08.757+0000 7f499cb2d700 1 --2- 172.21.15.88:0/2628527359 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7f49981311c0 0x7f49980a23c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:36:08.782 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:08.757+0000 7f499cb2d700 1 -- 172.21.15.88:0/2628527359 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7f49981332e0 msgr2=0x7f49980a2900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:36:08.782 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:08.757+0000 7f499cb2d700 1 --2- 172.21.15.88:0/2628527359 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7f49981332e0 0x7f49980a2900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:36:08.782 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:08.757+0000 7f499cb2d700 10 monclient: _finish_hunting 0 2023-09-30T01:36:08.782 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:08.757+0000 7f499cb2d700 1 monclient: found mon.noname-a 2023-09-30T01:36:08.782 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:08.757+0000 7f499cb2d700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:36:08.782 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:08.757+0000 7f499cb2d700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.88:3300/0 2023-09-30T01:36:08.782 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:08.757+0000 7f499cb2d700 1 -- 172.21.15.88:0/2628527359 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4990009a20 con 0x7f4998097a40 2023-09-30T01:36:08.782 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:08.757+0000 7f499cb2d700 1 --2- 172.21.15.88:0/2628527359 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4998097a40 0x7f49980a1e80 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f498c00e9e0 tx=0x7f498c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:36:08.783 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:08.757+0000 7f4994ff9700 1 -- 172.21.15.88:0/2628527359 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f498c013b00 con 0x7f4998097a40 2023-09-30T01:36:08.783 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:08.757+0000 7f4994ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:36:08.783 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:08.757+0000 7f4994ff9700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T01:36:08.783 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:08.757+0000 7f4994ff9700 10 monclient: dump: 2023-09-30T01:36:08.783 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:36:08.783 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:36:08.783 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:36:08.783 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:36:08.783 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:36:08.783 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:36:08.783 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:36:08.783 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:36:08.784 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:36:08.784 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:36:08.784 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:36:08.784 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:36:08.784 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:36:08.784 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:36:08.784 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:36:08.784 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:36:08.784 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:08.757+0000 7f4994ff9700 10 monclient: _finish_auth 0 2023-09-30T01:36:08.784 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:08.757+0000 7f4994ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:36:08.784 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:08.757+0000 7f4994ff9700 1 -- 172.21.15.88:0/2628527359 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f498c013ca0 con 0x7f4998097a40 2023-09-30T01:36:08.784 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:08.757+0000 7f4994ff9700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:36:08.785 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:08.757+0000 7f4994ff9700 1 -- 172.21.15.88:0/2628527359 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f498c013e10 con 0x7f4998097a40 2023-09-30T01:36:08.785 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:08.757+0000 7f499db2f700 5 monclient: authenticate success, global_id 4914 2023-09-30T01:36:08.785 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:08.757+0000 7f499db2f700 10 monclient: _renew_subs 2023-09-30T01:36:08.785 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:08.757+0000 7f499db2f700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.88:3300/0 2023-09-30T01:36:08.785 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:08.757+0000 7f499db2f700 1 -- 172.21.15.88:0/2628527359 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4988009420 con 0x7f4998097a40 2023-09-30T01:36:08.785 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:08.757+0000 7f4994ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:36:08.785 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:08.757+0000 7f4996ffd700 4 set_mon_vals no callback set 2023-09-30T01:36:08.785 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:08.757+0000 7f4994ff9700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:36:08.785 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:08.757+0000 7f4994ff9700 10 monclient: dump: 2023-09-30T01:36:08.785 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:36:08.785 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:36:08.785 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:36:08.785 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:36:08.786 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:36:08.786 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:36:08.786 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:36:08.786 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:36:08.786 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:36:08.786 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:36:08.786 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:36:08.786 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:36:08.786 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:36:08.786 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:36:08.786 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:36:08.786 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:36:08.787 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:08.757+0000 7f499db2f700 10 monclient: _renew_subs 2023-09-30T01:36:08.787 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:08.757+0000 7f499db2f700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.88:3300/0 2023-09-30T01:36:08.787 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:08.757+0000 7f499db2f700 1 -- 172.21.15.88:0/2628527359 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f49980996b0 con 0x7f4998097a40 2023-09-30T01:36:08.787 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:08.761+0000 7f4994ff9700 1 -- 172.21.15.88:0/2628527359 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 40144+0+0 (secure 0 0 0) 0x7f498c013680 con 0x7f4998097a40 2023-09-30T01:36:08.787 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:08.761+0000 7f4994ff9700 1 --2- 172.21.15.88:0/2628527359 >> v2:172.21.15.107:6812/29926 conn(0x7f4984033390 0x7f4984035840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:36:08.787 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:08.761+0000 7f4994ff9700 1 -- 172.21.15.88:0/2628527359 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 3502+0+0 (secure 0 0 0) 0x7f498c043800 con 0x7f4998097a40 2023-09-30T01:36:08.787 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:08.761+0000 7f4997fff700 1 --2- 172.21.15.88:0/2628527359 >> v2:172.21.15.107:6812/29926 conn(0x7f4984033390 0x7f4984035840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:36:08.787 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:08.761+0000 7f4997fff700 10 monclient: get_auth_request con 0x7f4984033390 auth_method 0 2023-09-30T01:36:08.787 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:08.761+0000 7f499db2f700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:36:08.787 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:08.761+0000 7f499db2f700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:36:08.787 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:08.761+0000 7f499db2f700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.88:3300/0 2023-09-30T01:36:08.787 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:08.761+0000 7f499db2f700 1 -- 172.21.15.88:0/2628527359 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4998071960 con 0x7f4998097a40 2023-09-30T01:36:08.788 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:08.761+0000 7f4997fff700 1 --2- 172.21.15.88:0/2628527359 >> v2:172.21.15.107:6812/29926 conn(0x7f4984033390 0x7f4984035840 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f4988009910 tx=0x7f4988017000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:36:08.788 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:08.765+0000 7f4994ff9700 1 -- 172.21.15.88:0/2628527359 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f498c01c970 con 0x7f4998097a40 2023-09-30T01:36:08.788 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:08.765+0000 7f4994ff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:36:08.788 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:08.765+0000 7f4994ff9700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:36:08.967 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:08.961+0000 7f499db2f700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T01:36:08.967 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:08.961+0000 7f499db2f700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:36:08.967 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:08.961+0000 7f499db2f700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.88:3300/0 2023-09-30T01:36:08.967 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:08.961+0000 7f499db2f700 1 -- 172.21.15.88:0/2628527359 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f4998067b60 con 0x7f4998097a40 2023-09-30T01:36:08.968 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:08.965+0000 7f4994ff9700 1 -- 172.21.15.88:0/2628527359 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f498c02a080 con 0x7f4998097a40 2023-09-30T01:36:08.968 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:08.965+0000 7f4994ff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:36:08.968 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:08.965+0000 7f4994ff9700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:36:08.968 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-09-30T01:36:08.968 INFO:teuthology.orchestra.run.smithi088.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:36:08.970 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:08.965+0000 7f499db2f700 1 -- 172.21.15.88:0/2628527359 >> v2:172.21.15.107:6812/29926 conn(0x7f4984033390 msgr2=0x7f4984035840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:36:08.970 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:08.965+0000 7f499db2f700 1 --2- 172.21.15.88:0/2628527359 >> v2:172.21.15.107:6812/29926 conn(0x7f4984033390 0x7f4984035840 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f4988009910 tx=0x7f4988017000 comp rx=0 tx=0).stop 2023-09-30T01:36:08.970 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:08.965+0000 7f499db2f700 10 monclient: shutdown 2023-09-30T01:36:08.970 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:08.965+0000 7f499db2f700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:36:08.970 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:08.965+0000 7f499db2f700 1 -- 172.21.15.88:0/2628527359 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4998097a40 msgr2=0x7f49980a1e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:36:08.970 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:08.965+0000 7f499db2f700 1 --2- 172.21.15.88:0/2628527359 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4998097a40 0x7f49980a1e80 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f498c00e9e0 tx=0x7f498c009250 comp rx=0 tx=0).stop 2023-09-30T01:36:08.970 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:08.965+0000 7f499db2f700 1 -- 172.21.15.88:0/2628527359 shutdown_connections 2023-09-30T01:36:08.970 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:08.965+0000 7f499db2f700 1 --2- 172.21.15.88:0/2628527359 >> v2:172.21.15.107:6812/29926 conn(0x7f4984033390 0x7f4984035840 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:36:08.971 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:08.965+0000 7f499db2f700 1 --2- 172.21.15.88:0/2628527359 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7f49981332e0 0x7f49980a2900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:36:08.971 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:08.965+0000 7f499db2f700 1 --2- 172.21.15.88:0/2628527359 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7f49981311c0 0x7f49980a23c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:36:08.971 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:08.965+0000 7f499db2f700 1 --2- 172.21.15.88:0/2628527359 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4998097a40 0x7f49980a1e80 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:36:08.971 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:08.965+0000 7f499db2f700 1 -- 172.21.15.88:0/2628527359 >> 172.21.15.88:0/2628527359 conn(0x7f499806e5e0 msgr2=0x7f4998131a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:36:08.971 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:08.965+0000 7f499db2f700 1 -- 172.21.15.88:0/2628527359 shutdown_connections 2023-09-30T01:36:08.971 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:08.965+0000 7f499db2f700 1 -- 172.21.15.88:0/2628527359 wait complete. 2023-09-30T01:36:11.982 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:36:12.094 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:12.089+0000 7f8cf528c700 10 monclient: get_monmap_and_config 2023-09-30T01:36:12.094 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:12.089+0000 7f8cf528c700 10 monclient: build_initial_monmap 2023-09-30T01:36:12.094 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:12.089+0000 7f8cf528c700 10 monclient: monmap: 2023-09-30T01:36:12.094 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2023-09-30T01:36:12.094 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:36:12.094 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:36:12.094195+0000 2023-09-30T01:36:12.094 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:36:12.094195+0000 2023-09-30T01:36:12.095 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2023-09-30T01:36:12.095 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:36:12.095 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a-legacy 2023-09-30T01:36:12.095 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.107:6789/0 mon.noname-f-legacy 2023-09-30T01:36:12.095 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.noname-b 2023-09-30T01:36:12.095 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.noname-g 2023-09-30T01:36:12.095 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.noname-c 2023-09-30T01:36:12.095 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.noname-h 2023-09-30T01:36:12.095 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.noname-d 2023-09-30T01:36:12.095 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.noname-i 2023-09-30T01:36:12.095 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.noname-e 2023-09-30T01:36:12.095 INFO:teuthology.orchestra.run.smithi088.stderr:9: v2:172.21.15.88:3300/0 mon.noname-a 2023-09-30T01:36:12.095 INFO:teuthology.orchestra.run.smithi088.stderr:10: v2:172.21.15.107:3300/0 mon.noname-f 2023-09-30T01:36:12.096 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:36:12.100 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:12.089+0000 7f8cf528c700 1 Processor -- start 2023-09-30T01:36:12.100 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:12.089+0000 7f8cf528c700 1 -- start start 2023-09-30T01:36:12.100 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:12.089+0000 7f8cf528c700 10 monclient: init 2023-09-30T01:36:12.100 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:12.093+0000 7f8cf528c700 10 monclient: _reopen_session rank -1 2023-09-30T01:36:12.100 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:12.093+0000 7f8cf528c700 10 monclient: _add_conns ranks=[4,0,5,9,10,7,8,6,3,2,1] 2023-09-30T01:36:12.100 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:12.093+0000 7f8cf528c700 1 --2- >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f8cf009be50 0x7f8cf006aa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:36:12.100 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:12.093+0000 7f8cf528c700 10 monclient(hunting): picked mon.noname-c con 0x7f8cf009be50 addr [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] 2023-09-30T01:36:12.100 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:12.093+0000 7f8cf528c700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f8cf009c220 addr v1:172.21.15.88:6789/0 2023-09-30T01:36:12.100 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:12.093+0000 7f8cf528c700 1 --2- >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7f8cf0138860 0x7f8cf013ac40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:36:12.101 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:12.093+0000 7f8cf528c700 10 monclient(hunting): picked mon.noname-h con 0x7f8cf0138860 addr [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] 2023-09-30T01:36:12.101 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:12.093+0000 7f8cf528c700 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8cf004bb20 con 0x7f8cf009c220 2023-09-30T01:36:12.101 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:12.093+0000 7f8cf528c700 10 monclient(hunting): start opening mon connection 2023-09-30T01:36:12.101 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:12.093+0000 7f8cf528c700 1 -- --> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] -- mon_getmap magic: 0 v1 -- 0x7f8cf009cd20 con 0x7f8cf009be50 2023-09-30T01:36:12.101 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:12.093+0000 7f8cf528c700 10 monclient(hunting): start opening mon connection 2023-09-30T01:36:12.101 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:12.093+0000 7f8cf528c700 1 -- --> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] -- mon_getmap magic: 0 v1 -- 0x7f8cf009f430 con 0x7f8cf0138860 2023-09-30T01:36:12.101 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:12.093+0000 7f8cf528c700 10 monclient(hunting): _renew_subs 2023-09-30T01:36:12.101 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:12.093+0000 7f8cf528c700 10 monclient(hunting): authenticate will time out at 1044.132459s 2023-09-30T01:36:12.101 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:12.093+0000 7f8ceffff700 1 --2- >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f8cf009be50 0x7f8cf006aa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:36:12.101 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:12.093+0000 7f8cef7fe700 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f8cf009c220 0x7f8cf009c610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:48324/0 (socket says 172.21.15.88:48324) 2023-09-30T01:36:12.101 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:12.093+0000 7f8cef7fe700 1 -- 172.21.15.88:0/2028413742 learned_addr learned my addr 172.21.15.88:0/2028413742 (peer_addr_for_me v1:172.21.15.88:0/0) 2023-09-30T01:36:12.101 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:12.093+0000 7f8ceffff700 10 monclient(hunting): get_auth_request con 0x7f8cf009be50 auth_method 0 2023-09-30T01:36:12.102 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:12.093+0000 7f8ceffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:36:12.102 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:12.093+0000 7f8ceffff700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:36:12.102 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:12.093+0000 7f8ceffff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:36:12.102 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:12.093+0000 7f8ceffff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:36:12.102 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:12.093+0000 7f8ceffff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:36:12.102 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:12.093+0000 7f8cf4a8b700 1 --2- 172.21.15.88:0/2028413742 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7f8cf0138860 0x7f8cf013ac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:36:12.102 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:12.093+0000 7f8ceffff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:36:12.102 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:12.093+0000 7f8cf4a8b700 10 monclient(hunting): get_auth_request con 0x7f8cf0138860 auth_method 0 2023-09-30T01:36:12.102 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:12.093+0000 7f8cf4a8b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:36:12.102 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:12.093+0000 7f8cf4a8b700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:36:12.102 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:12.093+0000 7f8cf4a8b700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:36:12.102 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:12.093+0000 7f8cf4a8b700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:36:12.103 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:12.093+0000 7f8cf4a8b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:36:12.103 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:12.093+0000 7f8cf4a8b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:36:12.103 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:12.093+0000 7f8ceffff700 10 monclient(hunting): handle_auth_done global_id 4666 payload 274 2023-09-30T01:36:12.103 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:12.093+0000 7f8ceffff700 1 -- 172.21.15.88:0/2028413742 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7f8cf0138860 msgr2=0x7f8cf013ac40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:36:12.103 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:12.093+0000 7f8ceffff700 1 --2- 172.21.15.88:0/2028413742 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7f8cf0138860 0x7f8cf013ac40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:36:12.103 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:12.093+0000 7f8ceffff700 1 -- 172.21.15.88:0/2028413742 >> v1:172.21.15.88:6789/0 conn(0x7f8cf009c220 legacy=0x7f8cf009c610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:36:12.103 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:12.093+0000 7f8ceffff700 10 monclient: _finish_hunting 0 2023-09-30T01:36:12.103 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:12.093+0000 7f8ceffff700 1 monclient: found mon.noname-c 2023-09-30T01:36:12.103 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:12.093+0000 7f8ceffff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:36:12.103 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:12.093+0000 7f8ceffff700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.88:3302/0 2023-09-30T01:36:12.103 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:12.093+0000 7f8ceffff700 1 -- 172.21.15.88:0/2028413742 --> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8cf009d7c0 con 0x7f8cf009be50 2023-09-30T01:36:12.103 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:12.093+0000 7f8ceffff700 1 --2- 172.21.15.88:0/2028413742 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f8cf009be50 0x7f8cf006aa20 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f8ce0009910 tx=0x7f8ce00091a0 comp rx=0 tx=0).ready entity=mon.4 client_cookie=467e305d733b226f server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:36:12.104 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:12.093+0000 7f8ceeffd700 1 -- 172.21.15.88:0/2028413742 <== mon.4 v2:172.21.15.88:3302/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f8ce000bb00 con 0x7f8cf009be50 2023-09-30T01:36:12.104 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:12.093+0000 7f8ceeffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:36:12.104 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:12.093+0000 7f8ceeffd700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T01:36:12.104 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:12.093+0000 7f8ceeffd700 10 monclient: dump: 2023-09-30T01:36:12.104 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:36:12.104 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:36:12.104 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:36:12.104 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:36:12.104 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:36:12.104 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:36:12.104 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:36:12.104 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:36:12.104 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:36:12.105 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:36:12.105 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:36:12.105 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:36:12.105 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:36:12.105 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:36:12.105 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:36:12.105 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:36:12.105 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:12.093+0000 7f8ceeffd700 10 monclient: _finish_auth 0 2023-09-30T01:36:12.105 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:12.093+0000 7f8ceeffd700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:36:12.105 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:12.093+0000 7f8ceeffd700 1 -- 172.21.15.88:0/2028413742 <== mon.4 v2:172.21.15.88:3302/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ce000bc80 con 0x7f8cf009be50 2023-09-30T01:36:12.105 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:12.093+0000 7f8cf528c700 5 monclient: authenticate success, global_id 4666 2023-09-30T01:36:12.105 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:12.093+0000 7f8cf528c700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:36:12.105 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:12.093+0000 7f8ceeffd700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:36:12.106 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:12.093+0000 7f8cf528c700 10 monclient: get_monmap_and_config success 2023-09-30T01:36:12.106 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:12.093+0000 7f8ceeffd700 1 -- 172.21.15.88:0/2028413742 <== mon.4 v2:172.21.15.88:3302/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f8ce000bdf0 con 0x7f8cf009be50 2023-09-30T01:36:12.106 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:12.093+0000 7f8cf528c700 4 set_mon_vals no callback set 2023-09-30T01:36:12.106 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:12.093+0000 7f8ceeffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:36:12.106 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:12.093+0000 7f8ceeffd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T01:36:12.106 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:12.093+0000 7f8ceeffd700 10 monclient: dump: 2023-09-30T01:36:12.106 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:36:12.106 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:36:12.106 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:36:12.106 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:36:12.106 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:36:12.106 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:36:12.107 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:36:12.107 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:36:12.107 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:36:12.107 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:36:12.107 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:36:12.107 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:36:12.107 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:36:12.107 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:36:12.107 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:36:12.107 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:36:12.107 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:12.093+0000 7f8cf528c700 10 monclient: shutdown 2023-09-30T01:36:12.107 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:12.093+0000 7f8cf528c700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:36:12.107 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:12.093+0000 7f8cf528c700 1 -- 172.21.15.88:0/2028413742 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f8cf009be50 msgr2=0x7f8cf006aa20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:36:12.108 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:12.093+0000 7f8cf528c700 1 --2- 172.21.15.88:0/2028413742 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f8cf009be50 0x7f8cf006aa20 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f8ce0009910 tx=0x7f8ce00091a0 comp rx=0 tx=0).stop 2023-09-30T01:36:12.108 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:12.093+0000 7f8cf528c700 1 -- 172.21.15.88:0/2028413742 shutdown_connections 2023-09-30T01:36:12.108 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:12.093+0000 7f8cf528c700 1 --2- 172.21.15.88:0/2028413742 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7f8cf0138860 0x7f8cf013ac40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:36:12.108 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:12.093+0000 7f8cf528c700 1 --2- 172.21.15.88:0/2028413742 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f8cf009be50 0x7f8cf006aa20 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:36:12.108 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:12.093+0000 7f8cf528c700 1 -- 172.21.15.88:0/2028413742 >> 172.21.15.88:0/2028413742 conn(0x7f8cf0063610 msgr2=0x7f8cf00a2b90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:36:12.108 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:12.093+0000 7f8cf528c700 1 -- 172.21.15.88:0/2028413742 shutdown_connections 2023-09-30T01:36:12.108 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:12.093+0000 7f8cf528c700 1 -- 172.21.15.88:0/2028413742 wait complete. 2023-09-30T01:36:12.108 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:12.097+0000 7f8cf528c700 10 monclient: build_initial_monmap 2023-09-30T01:36:12.108 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:12.097+0000 7f8cf528c700 10 monclient: monmap: 2023-09-30T01:36:12.108 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2023-09-30T01:36:12.108 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:36:12.108 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 0.000000 2023-09-30T01:36:12.108 INFO:teuthology.orchestra.run.smithi088.stderr:created 0.000000 2023-09-30T01:36:12.109 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2023-09-30T01:36:12.109 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 1 2023-09-30T01:36:12.109 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.noname-a 2023-09-30T01:36:12.109 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.noname-f 2023-09-30T01:36:12.109 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.noname-b 2023-09-30T01:36:12.109 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.noname-g 2023-09-30T01:36:12.109 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.noname-c 2023-09-30T01:36:12.109 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.noname-h 2023-09-30T01:36:12.109 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.noname-d 2023-09-30T01:36:12.109 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.noname-i 2023-09-30T01:36:12.109 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.noname-e 2023-09-30T01:36:12.109 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:36:12.114 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:12.097+0000 7f8cf528c700 1 Processor -- start 2023-09-30T01:36:12.114 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:12.097+0000 7f8cf528c700 1 -- start start 2023-09-30T01:36:12.114 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:12.097+0000 7f8cf528c700 10 monclient: init 2023-09-30T01:36:12.114 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:12.097+0000 7f8cf528c700 10 monclient: _reopen_session rank -1 2023-09-30T01:36:12.114 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:12.097+0000 7f8cf528c700 10 monclient: _add_conns ranks=[0,7,8,3,5,4,6,1,2] 2023-09-30T01:36:12.114 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:12.097+0000 7f8cf528c700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8cf009c220 0x7f8cf011fc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:36:12.115 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:12.097+0000 7f8cf528c700 10 monclient(hunting): picked mon.noname-a con 0x7f8cf009c220 addr [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] 2023-09-30T01:36:12.115 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:12.097+0000 7f8cf528c700 1 --2- >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7f8cf0138860 0x7f8cf0120150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:36:12.115 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:12.097+0000 7f8cf528c700 10 monclient(hunting): picked mon.noname-i con 0x7f8cf0138860 addr [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] 2023-09-30T01:36:12.115 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:12.097+0000 7f8cf528c700 1 --2- >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f8cf01223a0 0x7f8cf0120690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:36:12.115 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:12.097+0000 7f8cf528c700 10 monclient(hunting): picked mon.noname-e con 0x7f8cf01223a0 addr [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] 2023-09-30T01:36:12.115 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:12.097+0000 7f8cf528c700 10 monclient(hunting): start opening mon connection 2023-09-30T01:36:12.115 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:12.097+0000 7f8cf528c700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cf0138370 con 0x7f8cf009c220 2023-09-30T01:36:12.115 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:12.097+0000 7f8cf528c700 10 monclient(hunting): start opening mon connection 2023-09-30T01:36:12.115 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:12.097+0000 7f8cf528c700 1 -- --> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] -- mon_getmap magic: 0 v1 -- 0x7f8cf006e580 con 0x7f8cf0138860 2023-09-30T01:36:12.115 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:12.097+0000 7f8cf528c700 10 monclient(hunting): start opening mon connection 2023-09-30T01:36:12.115 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:12.097+0000 7f8cf528c700 1 -- --> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] -- mon_getmap magic: 0 v1 -- 0x7f8cf009e8e0 con 0x7f8cf01223a0 2023-09-30T01:36:12.115 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:12.097+0000 7f8cf528c700 10 monclient(hunting): _renew_subs 2023-09-30T01:36:12.116 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:12.097+0000 7f8cf528c700 10 monclient(hunting): authenticate will time out at 1044.138945s 2023-09-30T01:36:12.116 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:12.097+0000 7f8ceffff700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8cf009c220 0x7f8cf011fc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:36:12.116 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:12.097+0000 7f8cf4a8b700 1 --2- >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f8cf01223a0 0x7f8cf0120690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:36:12.116 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:12.097+0000 7f8ceffff700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8cf009c220 0x7f8cf011fc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:48228/0 (socket says 172.21.15.88:48228) 2023-09-30T01:36:12.116 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:12.097+0000 7f8ceffff700 1 -- 172.21.15.88:0/1100992925 learned_addr learned my addr 172.21.15.88:0/1100992925 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-09-30T01:36:12.116 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:12.097+0000 7f8ceffff700 10 monclient(hunting): get_auth_request con 0x7f8cf009c220 auth_method 0 2023-09-30T01:36:12.116 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:12.097+0000 7f8ceffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:36:12.116 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:12.097+0000 7f8ceffff700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:36:12.116 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:12.097+0000 7f8ceffff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:36:12.116 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:12.097+0000 7f8cf4a8b700 10 monclient(hunting): get_auth_request con 0x7f8cf01223a0 auth_method 0 2023-09-30T01:36:12.116 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:12.097+0000 7f8cf4a8b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:36:12.116 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:12.097+0000 7f8cf4a8b700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:36:12.117 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:12.097+0000 7f8cf4a8b700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:36:12.117 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:12.097+0000 7f8cef7fe700 1 --2- 172.21.15.88:0/1100992925 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7f8cf0138860 0x7f8cf0120150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:36:12.117 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:12.097+0000 7f8ceffff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:36:12.117 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:12.097+0000 7f8ceffff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:36:12.117 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:12.097+0000 7f8ceffff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:36:12.117 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:12.097+0000 7f8cef7fe700 10 monclient(hunting): get_auth_request con 0x7f8cf0138860 auth_method 0 2023-09-30T01:36:12.117 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:12.097+0000 7f8cef7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:36:12.117 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:12.097+0000 7f8cef7fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:36:12.117 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:12.097+0000 7f8cef7fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:36:12.117 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:12.097+0000 7f8cf4a8b700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:36:12.117 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:12.097+0000 7f8cf4a8b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:36:12.117 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:12.097+0000 7f8cf4a8b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:36:12.117 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:12.097+0000 7f8cef7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:36:12.118 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:12.097+0000 7f8cef7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:36:12.118 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:12.097+0000 7f8cef7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:36:12.118 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:12.097+0000 7f8cf4a8b700 10 monclient(hunting): handle_auth_done global_id 4724 payload 824 2023-09-30T01:36:12.118 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:12.097+0000 7f8cf4a8b700 1 -- 172.21.15.88:0/1100992925 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7f8cf0138860 msgr2=0x7f8cf0120150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:36:12.118 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:12.097+0000 7f8cf4a8b700 1 --2- 172.21.15.88:0/1100992925 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7f8cf0138860 0x7f8cf0120150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:36:12.118 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:12.097+0000 7f8cf4a8b700 1 -- 172.21.15.88:0/1100992925 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8cf009c220 msgr2=0x7f8cf011fc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:36:12.118 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:12.097+0000 7f8cf4a8b700 1 --2- 172.21.15.88:0/1100992925 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8cf009c220 0x7f8cf011fc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:36:12.118 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:12.097+0000 7f8cf4a8b700 10 monclient: _finish_hunting 0 2023-09-30T01:36:12.118 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:12.097+0000 7f8cf4a8b700 1 monclient: found mon.noname-e 2023-09-30T01:36:12.118 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:12.097+0000 7f8cf4a8b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:36:12.118 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:12.097+0000 7f8cf4a8b700 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.88:3304/0 2023-09-30T01:36:12.118 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:12.097+0000 7f8cf4a8b700 1 -- 172.21.15.88:0/1100992925 --> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ce0009420 con 0x7f8cf01223a0 2023-09-30T01:36:12.119 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:12.101+0000 7f8cf4a8b700 1 --2- 172.21.15.88:0/1100992925 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f8cf01223a0 0x7f8cf0120690 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f8ce8009e20 tx=0x7f8ce800b040 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:36:12.119 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:12.101+0000 7f8cd7fff700 1 -- 172.21.15.88:0/1100992925 <== mon.8 v2:172.21.15.88:3304/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f8ce8014480 con 0x7f8cf01223a0 2023-09-30T01:36:12.119 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:12.101+0000 7f8cd7fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:36:12.119 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:12.101+0000 7f8cd7fff700 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2023-09-30T01:36:12.119 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:12.101+0000 7f8cd7fff700 10 monclient: dump: 2023-09-30T01:36:12.119 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:36:12.119 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:36:12.119 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:36:12.119 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:36:12.119 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:36:12.119 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:36:12.119 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:36:12.120 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:36:12.120 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:36:12.120 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:36:12.120 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:36:12.120 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:36:12.120 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:36:12.120 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:36:12.120 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:36:12.120 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:36:12.120 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:12.101+0000 7f8cd7fff700 10 monclient: _finish_auth 0 2023-09-30T01:36:12.120 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:12.101+0000 7f8cd7fff700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:36:12.120 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:12.101+0000 7f8cd7fff700 1 -- 172.21.15.88:0/1100992925 <== mon.8 v2:172.21.15.88:3304/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ce800f040 con 0x7f8cf01223a0 2023-09-30T01:36:12.120 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:12.101+0000 7f8cf528c700 5 monclient: authenticate success, global_id 4724 2023-09-30T01:36:12.120 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:12.101+0000 7f8cf528c700 10 monclient: _renew_subs 2023-09-30T01:36:12.121 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:12.101+0000 7f8cf528c700 10 monclient: _send_mon_message to mon.e at v2:172.21.15.88:3304/0 2023-09-30T01:36:12.121 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:12.101+0000 7f8cf528c700 1 -- 172.21.15.88:0/1100992925 --> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8cf0123570 con 0x7f8cf01223a0 2023-09-30T01:36:12.121 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:12.101+0000 7f8cd7fff700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:36:12.121 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:12.101+0000 7f8cd7fff700 1 -- 172.21.15.88:0/1100992925 <== mon.8 v2:172.21.15.88:3304/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f8ce8014aa0 con 0x7f8cf01223a0 2023-09-30T01:36:12.121 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:12.101+0000 7f8cd7fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:36:12.121 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:12.101+0000 7f8cd7fff700 10 monclient: got monmap 1 from mon.e (according to old e1) 2023-09-30T01:36:12.121 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:12.101+0000 7f8cd7fff700 10 monclient: dump: 2023-09-30T01:36:12.121 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:36:12.121 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:36:12.121 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:36:12.121 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:36:12.121 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:36:12.122 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:36:12.122 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:36:12.122 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:36:12.122 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:36:12.122 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:36:12.122 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:36:12.122 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:36:12.122 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:36:12.122 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:36:12.122 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:36:12.122 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:36:12.122 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:12.101+0000 7f8cee7fc700 4 set_mon_vals no callback set 2023-09-30T01:36:12.122 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:12.101+0000 7f8cf528c700 10 monclient: _renew_subs 2023-09-30T01:36:12.123 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:12.101+0000 7f8cf528c700 10 monclient: _send_mon_message to mon.e at v2:172.21.15.88:3304/0 2023-09-30T01:36:12.123 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:12.101+0000 7f8cf528c700 1 -- 172.21.15.88:0/1100992925 --> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8cf0121760 con 0x7f8cf01223a0 2023-09-30T01:36:12.123 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:12.101+0000 7f8cf528c700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:36:12.123 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:12.101+0000 7f8cf528c700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:36:12.123 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:12.101+0000 7f8cf528c700 10 monclient: _send_mon_message to mon.e at v2:172.21.15.88:3304/0 2023-09-30T01:36:12.123 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:12.101+0000 7f8cf528c700 1 -- 172.21.15.88:0/1100992925 --> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8cf0123700 con 0x7f8cf01223a0 2023-09-30T01:36:12.123 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:12.101+0000 7f8cd7fff700 1 -- 172.21.15.88:0/1100992925 <== mon.8 v2:172.21.15.88:3304/0 4 ==== mgrmap(e 5) v1 ==== 40144+0+0 (secure 0 0 0) 0x7f8ce8014c00 con 0x7f8cf01223a0 2023-09-30T01:36:12.123 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:12.101+0000 7f8cd7fff700 1 --2- 172.21.15.88:0/1100992925 >> v2:172.21.15.107:6812/29926 conn(0x7f8cdc033390 0x7f8cdc035840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:36:12.123 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:12.101+0000 7f8cd7fff700 1 -- 172.21.15.88:0/1100992925 <== mon.8 v2:172.21.15.88:3304/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 3502+0+0 (secure 0 0 0) 0x7f8ce8043fd0 con 0x7f8cf01223a0 2023-09-30T01:36:12.123 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:12.101+0000 7f8ceffff700 1 --2- 172.21.15.88:0/1100992925 >> v2:172.21.15.107:6812/29926 conn(0x7f8cdc033390 0x7f8cdc035840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:36:12.123 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:12.101+0000 7f8ceffff700 10 monclient: get_auth_request con 0x7f8cdc033390 auth_method 0 2023-09-30T01:36:12.123 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:12.105+0000 7f8ceffff700 1 --2- 172.21.15.88:0/1100992925 >> v2:172.21.15.107:6812/29926 conn(0x7f8cdc033390 0x7f8cdc035840 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f8ce00097b0 tx=0x7f8ce0017000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:36:12.123 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:12.109+0000 7f8cd7fff700 1 -- 172.21.15.88:0/1100992925 <== mon.8 v2:172.21.15.88:3304/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f8ce800caa0 con 0x7f8cf01223a0 2023-09-30T01:36:12.124 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:12.109+0000 7f8cd7fff700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:36:12.124 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:12.109+0000 7f8cd7fff700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:36:12.302 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:12.297+0000 7f8cf528c700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T01:36:12.302 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:12.297+0000 7f8cf528c700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:36:12.303 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:12.297+0000 7f8cf528c700 10 monclient: _send_mon_message to mon.e at v2:172.21.15.88:3304/0 2023-09-30T01:36:12.303 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:12.297+0000 7f8cf528c700 1 -- 172.21.15.88:0/1100992925 --> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f8cf0071960 con 0x7f8cf01223a0 2023-09-30T01:36:12.303 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:12.297+0000 7f8cd7fff700 1 -- 172.21.15.88:0/1100992925 <== mon.8 v2:172.21.15.88:3304/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f8ce8018020 con 0x7f8cf01223a0 2023-09-30T01:36:12.303 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:12.297+0000 7f8cd7fff700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:36:12.303 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:12.297+0000 7f8cd7fff700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:36:12.304 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-09-30T01:36:12.304 INFO:teuthology.orchestra.run.smithi088.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:36:12.305 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:12.301+0000 7f8cf528c700 1 -- 172.21.15.88:0/1100992925 >> v2:172.21.15.107:6812/29926 conn(0x7f8cdc033390 msgr2=0x7f8cdc035840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:36:12.306 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:12.301+0000 7f8cf528c700 1 --2- 172.21.15.88:0/1100992925 >> v2:172.21.15.107:6812/29926 conn(0x7f8cdc033390 0x7f8cdc035840 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f8ce00097b0 tx=0x7f8ce0017000 comp rx=0 tx=0).stop 2023-09-30T01:36:12.306 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:12.301+0000 7f8cf528c700 10 monclient: shutdown 2023-09-30T01:36:12.306 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:12.301+0000 7f8cf528c700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:36:12.306 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:12.301+0000 7f8cf528c700 1 -- 172.21.15.88:0/1100992925 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f8cf01223a0 msgr2=0x7f8cf0120690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:36:12.306 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:12.301+0000 7f8cf528c700 1 --2- 172.21.15.88:0/1100992925 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f8cf01223a0 0x7f8cf0120690 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f8ce8009e20 tx=0x7f8ce800b040 comp rx=0 tx=0).stop 2023-09-30T01:36:12.306 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:12.301+0000 7f8cf528c700 1 -- 172.21.15.88:0/1100992925 shutdown_connections 2023-09-30T01:36:12.306 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:12.301+0000 7f8cf528c700 1 --2- 172.21.15.88:0/1100992925 >> v2:172.21.15.107:6812/29926 conn(0x7f8cdc033390 0x7f8cdc035840 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:36:12.306 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:12.301+0000 7f8cf528c700 1 --2- 172.21.15.88:0/1100992925 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f8cf01223a0 0x7f8cf0120690 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:36:12.306 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:12.301+0000 7f8cf528c700 1 --2- 172.21.15.88:0/1100992925 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7f8cf0138860 0x7f8cf0120150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:36:12.306 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:12.301+0000 7f8cf528c700 1 --2- 172.21.15.88:0/1100992925 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8cf009c220 0x7f8cf011fc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:36:12.306 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:12.301+0000 7f8cf528c700 1 -- 172.21.15.88:0/1100992925 >> 172.21.15.88:0/1100992925 conn(0x7f8cf0063610 msgr2=0x7f8cf00a21b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:36:12.306 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:12.301+0000 7f8cf528c700 1 -- 172.21.15.88:0/1100992925 shutdown_connections 2023-09-30T01:36:12.307 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:12.301+0000 7f8cf528c700 1 -- 172.21.15.88:0/1100992925 wait complete. 2023-09-30T01:36:15.318 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:36:15.431 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:15.425+0000 7fe573fff700 10 monclient: get_monmap_and_config 2023-09-30T01:36:15.431 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:15.425+0000 7fe573fff700 10 monclient: build_initial_monmap 2023-09-30T01:36:15.431 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:15.425+0000 7fe573fff700 10 monclient: monmap: 2023-09-30T01:36:15.431 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2023-09-30T01:36:15.431 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:36:15.431 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:36:15.431038+0000 2023-09-30T01:36:15.431 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:36:15.431038+0000 2023-09-30T01:36:15.432 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2023-09-30T01:36:15.432 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:36:15.432 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a-legacy 2023-09-30T01:36:15.432 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.107:6789/0 mon.noname-f-legacy 2023-09-30T01:36:15.432 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.noname-b 2023-09-30T01:36:15.432 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.noname-g 2023-09-30T01:36:15.432 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.noname-c 2023-09-30T01:36:15.432 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.noname-h 2023-09-30T01:36:15.432 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.noname-d 2023-09-30T01:36:15.432 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.noname-i 2023-09-30T01:36:15.432 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.noname-e 2023-09-30T01:36:15.432 INFO:teuthology.orchestra.run.smithi088.stderr:9: v2:172.21.15.88:3300/0 mon.noname-a 2023-09-30T01:36:15.433 INFO:teuthology.orchestra.run.smithi088.stderr:10: v2:172.21.15.107:3300/0 mon.noname-f 2023-09-30T01:36:15.433 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:36:15.447 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:15.429+0000 7fe573fff700 1 Processor -- start 2023-09-30T01:36:15.447 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:15.429+0000 7fe573fff700 1 -- start start 2023-09-30T01:36:15.447 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:15.429+0000 7fe573fff700 10 monclient: init 2023-09-30T01:36:15.447 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:15.429+0000 7fe573fff700 10 monclient: _reopen_session rank -1 2023-09-30T01:36:15.448 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:15.429+0000 7fe573fff700 10 monclient: _add_conns ranks=[3,7,10,4,9,1,0,2,6,5,8] 2023-09-30T01:36:15.448 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:15.429+0000 7fe573fff700 1 --2- >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7fe57409cad0 0x7fe5741314d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:36:15.448 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:15.429+0000 7fe573fff700 10 monclient(hunting): picked mon.noname-g con 0x7fe57409cad0 addr [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] 2023-09-30T01:36:15.448 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:15.429+0000 7fe573fff700 1 --2- >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7fe574131a10 0x7fe57409cea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:36:15.448 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:15.429+0000 7fe573fff700 10 monclient(hunting): picked mon.noname-i con 0x7fe574131a10 addr [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] 2023-09-30T01:36:15.448 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:15.429+0000 7fe573fff700 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fe57409d3e0 0x7fe574062c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:36:15.448 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:15.429+0000 7fe573fff700 10 monclient(hunting): picked mon.noname-f con 0x7fe57409d3e0 addr v2:172.21.15.107:3300/0 2023-09-30T01:36:15.449 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:15.429+0000 7fe573fff700 10 monclient(hunting): start opening mon connection 2023-09-30T01:36:15.449 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:15.429+0000 7fe573fff700 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe57409fbf0 con 0x7fe57409d3e0 2023-09-30T01:36:15.449 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:15.429+0000 7fe573fff700 10 monclient(hunting): start opening mon connection 2023-09-30T01:36:15.449 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:15.429+0000 7fe573fff700 1 -- --> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe57409f510 con 0x7fe57409cad0 2023-09-30T01:36:15.449 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:15.429+0000 7fe573fff700 10 monclient(hunting): start opening mon connection 2023-09-30T01:36:15.449 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:15.429+0000 7fe573fff700 1 -- --> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] -- mon_getmap magic: 0 v1 -- 0x7fe57409f230 con 0x7fe574131a10 2023-09-30T01:36:15.450 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:15.429+0000 7fe573fff700 10 monclient(hunting): _renew_subs 2023-09-30T01:36:15.450 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:15.429+0000 7fe573fff700 10 monclient(hunting): authenticate will time out at 1047.469056s 2023-09-30T01:36:15.450 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:15.429+0000 7fe5737fe700 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fe57409d3e0 0x7fe574062c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:36:15.450 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:15.429+0000 7fe572ffd700 1 --2- >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7fe57409cad0 0x7fe5741314d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:36:15.450 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:15.429+0000 7fe5727fc700 1 --2- >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7fe574131a10 0x7fe57409cea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:36:15.450 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:15.429+0000 7fe5737fe700 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fe57409d3e0 0x7fe574062c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.88:33460/0 (socket says 172.21.15.88:33460) 2023-09-30T01:36:15.450 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:15.429+0000 7fe572ffd700 1 --2- >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7fe57409cad0 0x7fe5741314d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3301/0 says I am v2:172.21.15.88:53390/0 (socket says 172.21.15.88:53390) 2023-09-30T01:36:15.450 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:15.429+0000 7fe5737fe700 1 -- 172.21.15.88:0/3577546437 learned_addr learned my addr 172.21.15.88:0/3577546437 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-09-30T01:36:15.450 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:15.429+0000 7fe5737fe700 10 monclient(hunting): get_auth_request con 0x7fe57409d3e0 auth_method 0 2023-09-30T01:36:15.450 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:15.429+0000 7fe5737fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:36:15.451 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:15.429+0000 7fe5737fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:36:15.451 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:15.429+0000 7fe5737fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:36:15.451 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:15.429+0000 7fe572ffd700 10 monclient(hunting): get_auth_request con 0x7fe57409cad0 auth_method 0 2023-09-30T01:36:15.451 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:15.429+0000 7fe572ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:36:15.451 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:15.429+0000 7fe572ffd700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:36:15.451 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:15.429+0000 7fe572ffd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:36:15.451 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:15.429+0000 7fe5727fc700 10 monclient(hunting): get_auth_request con 0x7fe574131a10 auth_method 0 2023-09-30T01:36:15.451 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:15.429+0000 7fe5727fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:36:15.451 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:15.429+0000 7fe5727fc700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:36:15.451 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:15.429+0000 7fe5727fc700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:36:15.451 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:15.429+0000 7fe5737fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:36:15.452 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:15.429+0000 7fe5737fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:36:15.452 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:15.429+0000 7fe5737fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:36:15.452 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:15.429+0000 7fe572ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:36:15.452 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:15.429+0000 7fe572ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:36:15.452 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:15.429+0000 7fe572ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:36:15.452 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:15.429+0000 7fe5727fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:36:15.452 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:15.429+0000 7fe5727fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:36:15.452 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:15.429+0000 7fe5727fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:36:15.452 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:15.429+0000 7fe572ffd700 10 monclient(hunting): handle_auth_done global_id 4566 payload 274 2023-09-30T01:36:15.452 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:15.429+0000 7fe572ffd700 1 -- 172.21.15.88:0/3577546437 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7fe574131a10 msgr2=0x7fe57409cea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:36:15.452 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:15.429+0000 7fe572ffd700 1 --2- 172.21.15.88:0/3577546437 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7fe574131a10 0x7fe57409cea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:36:15.452 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:15.429+0000 7fe572ffd700 1 -- 172.21.15.88:0/3577546437 >> v2:172.21.15.107:3300/0 conn(0x7fe57409d3e0 msgr2=0x7fe574062c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:36:15.452 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:15.429+0000 7fe572ffd700 1 --2- 172.21.15.88:0/3577546437 >> v2:172.21.15.107:3300/0 conn(0x7fe57409d3e0 0x7fe574062c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:36:15.453 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:15.429+0000 7fe572ffd700 10 monclient: _finish_hunting 0 2023-09-30T01:36:15.453 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:15.429+0000 7fe572ffd700 1 monclient: found mon.noname-g 2023-09-30T01:36:15.453 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:15.429+0000 7fe572ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:36:15.453 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:15.429+0000 7fe572ffd700 10 monclient: _send_mon_message to mon.noname-g at v2:172.21.15.107:3301/0 2023-09-30T01:36:15.453 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:15.429+0000 7fe572ffd700 1 -- 172.21.15.88:0/3577546437 --> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe57404bb20 con 0x7fe57409cad0 2023-09-30T01:36:15.453 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:15.429+0000 7fe5737fe700 1 --2- 172.21.15.88:0/3577546437 >> v2:172.21.15.107:3300/0 conn(0x7fe57409d3e0 0x7fe574062c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:36:15.453 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:15.429+0000 7fe572ffd700 1 --2- 172.21.15.88:0/3577546437 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7fe57409cad0 0x7fe5741314d0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fe55c009ee0 tx=0x7fe55c009780 comp rx=0 tx=0).ready entity=mon.3 client_cookie=3150d8cb204f4190 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:36:15.453 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:15.429+0000 7fe571ffb700 1 -- 172.21.15.88:0/3577546437 <== mon.3 v2:172.21.15.107:3301/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7fe55c00e070 con 0x7fe57409cad0 2023-09-30T01:36:15.453 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:15.429+0000 7fe571ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:36:15.453 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:15.429+0000 7fe571ffb700 10 monclient: got monmap 1 from mon.noname-g (according to old e1) 2023-09-30T01:36:15.453 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:15.429+0000 7fe571ffb700 10 monclient: dump: 2023-09-30T01:36:15.453 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:36:15.454 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:36:15.454 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:36:15.454 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:36:15.454 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:36:15.454 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:36:15.454 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:36:15.454 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:36:15.454 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:36:15.455 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:36:15.455 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:36:15.455 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:36:15.455 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:36:15.455 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:36:15.455 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:36:15.455 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:36:15.455 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:15.429+0000 7fe571ffb700 10 monclient: _finish_auth 0 2023-09-30T01:36:15.455 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:15.429+0000 7fe571ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:36:15.455 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:15.429+0000 7fe571ffb700 1 -- 172.21.15.88:0/3577546437 <== mon.3 v2:172.21.15.107:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe55c012430 con 0x7fe57409cad0 2023-09-30T01:36:15.456 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:15.429+0000 7fe573fff700 5 monclient: authenticate success, global_id 4566 2023-09-30T01:36:15.456 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:15.429+0000 7fe573fff700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:36:15.456 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:15.429+0000 7fe571ffb700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:36:15.456 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:15.429+0000 7fe573fff700 10 monclient: get_monmap_and_config success 2023-09-30T01:36:15.456 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:15.429+0000 7fe573fff700 4 set_mon_vals no callback set 2023-09-30T01:36:15.456 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:15.429+0000 7fe571ffb700 1 -- 172.21.15.88:0/3577546437 <== mon.3 v2:172.21.15.107:3301/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7fe55c0125a0 con 0x7fe57409cad0 2023-09-30T01:36:15.456 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:15.429+0000 7fe571ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:36:15.456 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:15.429+0000 7fe571ffb700 10 monclient: got monmap 1 from mon.g (according to old e1) 2023-09-30T01:36:15.456 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:15.429+0000 7fe571ffb700 10 monclient: dump: 2023-09-30T01:36:15.456 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:36:15.456 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:36:15.456 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:36:15.457 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:36:15.457 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:36:15.457 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:36:15.457 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:36:15.457 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:36:15.457 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:36:15.457 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:36:15.457 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:36:15.457 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:36:15.457 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:36:15.457 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:36:15.457 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:36:15.458 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:36:15.458 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:15.429+0000 7fe573fff700 10 monclient: shutdown 2023-09-30T01:36:15.458 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:15.429+0000 7fe573fff700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:36:15.458 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:15.429+0000 7fe573fff700 1 -- 172.21.15.88:0/3577546437 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7fe57409cad0 msgr2=0x7fe5741314d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:36:15.458 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:15.429+0000 7fe573fff700 1 --2- 172.21.15.88:0/3577546437 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7fe57409cad0 0x7fe5741314d0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fe55c009ee0 tx=0x7fe55c009780 comp rx=0 tx=0).stop 2023-09-30T01:36:15.458 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:15.429+0000 7fe573fff700 1 -- 172.21.15.88:0/3577546437 shutdown_connections 2023-09-30T01:36:15.458 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:15.429+0000 7fe573fff700 1 --2- 172.21.15.88:0/3577546437 >> v2:172.21.15.107:3300/0 conn(0x7fe57409d3e0 0x7fe574062c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:36:15.458 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:15.429+0000 7fe573fff700 1 --2- 172.21.15.88:0/3577546437 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7fe574131a10 0x7fe57409cea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:36:15.458 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:15.429+0000 7fe573fff700 1 --2- 172.21.15.88:0/3577546437 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7fe57409cad0 0x7fe5741314d0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:36:15.458 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:15.429+0000 7fe573fff700 1 -- 172.21.15.88:0/3577546437 >> 172.21.15.88:0/3577546437 conn(0x7fe5740687a0 msgr2=0x7fe57409fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:36:15.458 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:15.433+0000 7fe573fff700 1 -- 172.21.15.88:0/3577546437 shutdown_connections 2023-09-30T01:36:15.459 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:15.433+0000 7fe573fff700 1 -- 172.21.15.88:0/3577546437 wait complete. 2023-09-30T01:36:15.459 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:15.433+0000 7fe573fff700 10 monclient: build_initial_monmap 2023-09-30T01:36:15.459 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:15.433+0000 7fe573fff700 10 monclient: monmap: 2023-09-30T01:36:15.459 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2023-09-30T01:36:15.459 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:36:15.459 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 0.000000 2023-09-30T01:36:15.459 INFO:teuthology.orchestra.run.smithi088.stderr:created 0.000000 2023-09-30T01:36:15.459 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2023-09-30T01:36:15.459 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 1 2023-09-30T01:36:15.459 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.noname-a 2023-09-30T01:36:15.459 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.noname-f 2023-09-30T01:36:15.459 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.noname-b 2023-09-30T01:36:15.459 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.noname-g 2023-09-30T01:36:15.460 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.noname-c 2023-09-30T01:36:15.460 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.noname-h 2023-09-30T01:36:15.460 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.noname-d 2023-09-30T01:36:15.460 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.noname-i 2023-09-30T01:36:15.460 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.noname-e 2023-09-30T01:36:15.460 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:36:15.460 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:15.433+0000 7fe573fff700 1 Processor -- start 2023-09-30T01:36:15.460 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:15.433+0000 7fe573fff700 1 -- start start 2023-09-30T01:36:15.460 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:15.433+0000 7fe573fff700 10 monclient: init 2023-09-30T01:36:15.460 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:15.433+0000 7fe573fff700 10 monclient: _reopen_session rank -1 2023-09-30T01:36:15.460 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:15.433+0000 7fe573fff700 10 monclient: _add_conns ranks=[4,6,5,2,3,7,8,1,0] 2023-09-30T01:36:15.460 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:15.433+0000 7fe573fff700 1 --2- >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7fe57409cad0 0x7fe57412d9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:36:15.460 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:15.433+0000 7fe573fff700 10 monclient(hunting): picked mon.noname-c con 0x7fe57409cad0 addr [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] 2023-09-30T01:36:15.461 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:15.433+0000 7fe573fff700 1 --2- >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7fe57409d3e0 0x7fe57411a980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:36:15.461 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:15.433+0000 7fe573fff700 10 monclient(hunting): picked mon.noname-d con 0x7fe57409d3e0 addr [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] 2023-09-30T01:36:15.461 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:15.433+0000 7fe573fff700 1 --2- >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7fe574131a10 0x7fe57411aec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:36:15.461 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:15.433+0000 7fe573fff700 10 monclient(hunting): picked mon.noname-h con 0x7fe574131a10 addr [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] 2023-09-30T01:36:15.461 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:15.433+0000 7fe573fff700 10 monclient(hunting): start opening mon connection 2023-09-30T01:36:15.461 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:15.433+0000 7fe573fff700 1 -- --> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] -- mon_getmap magic: 0 v1 -- 0x7fe57409ed10 con 0x7fe57409cad0 2023-09-30T01:36:15.461 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:15.433+0000 7fe573fff700 10 monclient(hunting): start opening mon connection 2023-09-30T01:36:15.461 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:15.433+0000 7fe573fff700 1 -- --> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] -- mon_getmap magic: 0 v1 -- 0x7fe57409e820 con 0x7fe574131a10 2023-09-30T01:36:15.461 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:15.433+0000 7fe573fff700 10 monclient(hunting): start opening mon connection 2023-09-30T01:36:15.462 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:15.433+0000 7fe573fff700 1 -- --> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] -- mon_getmap magic: 0 v1 -- 0x7fe57411de70 con 0x7fe57409d3e0 2023-09-30T01:36:15.462 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:15.433+0000 7fe573fff700 10 monclient(hunting): _renew_subs 2023-09-30T01:36:15.462 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:15.433+0000 7fe573fff700 10 monclient(hunting): authenticate will time out at 1047.474795s 2023-09-30T01:36:15.462 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:15.433+0000 7fe572ffd700 1 --2- >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7fe57409cad0 0x7fe57412d9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:36:15.462 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:15.433+0000 7fe5727fc700 1 --2- >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7fe57409d3e0 0x7fe57411a980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:36:15.462 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:15.433+0000 7fe572ffd700 1 --2- >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7fe57409cad0 0x7fe57412d9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3302/0 says I am v2:172.21.15.88:52356/0 (socket says 172.21.15.88:52356) 2023-09-30T01:36:15.462 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:15.433+0000 7fe572ffd700 1 -- 172.21.15.88:0/2474407040 learned_addr learned my addr 172.21.15.88:0/2474407040 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-09-30T01:36:15.462 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:15.433+0000 7fe572ffd700 10 monclient(hunting): get_auth_request con 0x7fe57409cad0 auth_method 0 2023-09-30T01:36:15.462 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:15.433+0000 7fe572ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:36:15.462 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:15.433+0000 7fe572ffd700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:36:15.462 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:15.433+0000 7fe572ffd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:36:15.462 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:15.433+0000 7fe5727fc700 10 monclient(hunting): get_auth_request con 0x7fe57409d3e0 auth_method 0 2023-09-30T01:36:15.463 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:15.433+0000 7fe5727fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:36:15.463 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:15.433+0000 7fe5727fc700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:36:15.463 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:15.433+0000 7fe5727fc700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:36:15.463 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:15.433+0000 7fe572ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:36:15.463 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:15.433+0000 7fe572ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:36:15.463 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:15.433+0000 7fe572ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:36:15.463 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:15.433+0000 7fe5727fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:36:15.463 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:15.433+0000 7fe5727fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:36:15.463 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:15.433+0000 7fe5737fe700 1 --2- 172.21.15.88:0/2474407040 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7fe574131a10 0x7fe57411aec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:36:15.463 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:15.433+0000 7fe5727fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:36:15.463 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:15.433+0000 7fe5737fe700 10 monclient(hunting): get_auth_request con 0x7fe574131a10 auth_method 0 2023-09-30T01:36:15.463 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:15.433+0000 7fe5737fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:36:15.464 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:15.433+0000 7fe5737fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:36:15.464 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:15.433+0000 7fe5737fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:36:15.464 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:15.433+0000 7fe5737fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:36:15.464 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:15.433+0000 7fe5737fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:36:15.464 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:15.433+0000 7fe5737fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:36:15.464 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:15.433+0000 7fe572ffd700 10 monclient(hunting): handle_auth_done global_id 4675 payload 824 2023-09-30T01:36:15.464 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:15.433+0000 7fe572ffd700 1 -- 172.21.15.88:0/2474407040 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7fe57409d3e0 msgr2=0x7fe57411a980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:36:15.464 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:15.433+0000 7fe572ffd700 1 --2- 172.21.15.88:0/2474407040 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7fe57409d3e0 0x7fe57411a980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:36:15.464 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:15.433+0000 7fe572ffd700 1 -- 172.21.15.88:0/2474407040 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7fe574131a10 msgr2=0x7fe57411aec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:36:15.464 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:15.433+0000 7fe572ffd700 1 --2- 172.21.15.88:0/2474407040 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7fe574131a10 0x7fe57411aec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:36:15.465 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:15.433+0000 7fe572ffd700 10 monclient: _finish_hunting 0 2023-09-30T01:36:15.465 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:15.433+0000 7fe572ffd700 1 monclient: found mon.noname-c 2023-09-30T01:36:15.465 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:15.433+0000 7fe572ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:36:15.465 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:15.433+0000 7fe572ffd700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.88:3302/0 2023-09-30T01:36:15.465 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:15.433+0000 7fe572ffd700 1 -- 172.21.15.88:0/2474407040 --> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe56c0094f0 con 0x7fe57409cad0 2023-09-30T01:36:15.465 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:15.433+0000 7fe5727fc700 1 --2- 172.21.15.88:0/2474407040 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7fe57409d3e0 0x7fe57411a980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:36:15.465 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:15.433+0000 7fe572ffd700 1 --2- 172.21.15.88:0/2474407040 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7fe57409cad0 0x7fe57412d9b0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fe55c0097c0 tx=0x7fe55c019000 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:36:15.465 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:15.437+0000 7fe56affd700 1 -- 172.21.15.88:0/2474407040 <== mon.4 v2:172.21.15.88:3302/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7fe55c01c410 con 0x7fe57409cad0 2023-09-30T01:36:15.465 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:15.437+0000 7fe56affd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:36:15.465 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:15.437+0000 7fe56affd700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T01:36:15.465 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:15.437+0000 7fe56affd700 10 monclient: dump: 2023-09-30T01:36:15.465 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:36:15.466 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:36:15.466 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:36:15.466 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:36:15.466 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:36:15.466 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:36:15.466 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:36:15.466 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:36:15.466 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:36:15.466 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:36:15.466 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:36:15.466 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:36:15.467 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:36:15.467 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:36:15.467 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:36:15.467 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:36:15.467 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:15.437+0000 7fe56affd700 10 monclient: _finish_auth 0 2023-09-30T01:36:15.467 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:15.437+0000 7fe56affd700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:36:15.467 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:15.437+0000 7fe56affd700 1 -- 172.21.15.88:0/2474407040 <== mon.4 v2:172.21.15.88:3302/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe55c021070 con 0x7fe57409cad0 2023-09-30T01:36:15.467 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:15.437+0000 7fe573fff700 5 monclient: authenticate success, global_id 4675 2023-09-30T01:36:15.467 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:15.437+0000 7fe573fff700 10 monclient: _renew_subs 2023-09-30T01:36:15.467 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:15.437+0000 7fe573fff700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.88:3302/0 2023-09-30T01:36:15.467 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:15.437+0000 7fe573fff700 1 -- 172.21.15.88:0/2474407040 --> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe55c009a20 con 0x7fe57409cad0 2023-09-30T01:36:15.467 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:15.437+0000 7fe56affd700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:36:15.467 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:15.437+0000 7fe56affd700 1 -- 172.21.15.88:0/2474407040 <== mon.4 v2:172.21.15.88:3302/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7fe55c01ca50 con 0x7fe57409cad0 2023-09-30T01:36:15.468 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:15.437+0000 7fe56affd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:36:15.468 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:15.437+0000 7fe5717fa700 4 set_mon_vals no callback set 2023-09-30T01:36:15.468 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:15.437+0000 7fe56affd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T01:36:15.468 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:15.437+0000 7fe56affd700 10 monclient: dump: 2023-09-30T01:36:15.468 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:36:15.468 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:36:15.468 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:36:15.468 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:36:15.468 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:36:15.468 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:36:15.468 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:36:15.468 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:36:15.469 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:36:15.469 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:36:15.469 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:36:15.469 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:36:15.469 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:36:15.469 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:36:15.469 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:36:15.469 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:36:15.469 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:15.437+0000 7fe573fff700 10 monclient: _renew_subs 2023-09-30T01:36:15.469 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:15.437+0000 7fe573fff700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.88:3302/0 2023-09-30T01:36:15.469 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:15.437+0000 7fe573fff700 1 -- 172.21.15.88:0/2474407040 --> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe57411db00 con 0x7fe57409cad0 2023-09-30T01:36:15.469 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:15.437+0000 7fe573fff700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:36:15.469 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:15.437+0000 7fe573fff700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:36:15.470 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:15.437+0000 7fe573fff700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.88:3302/0 2023-09-30T01:36:15.470 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:15.437+0000 7fe573fff700 1 -- 172.21.15.88:0/2474407040 --> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe57409ade0 con 0x7fe57409cad0 2023-09-30T01:36:15.470 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:15.437+0000 7fe56affd700 1 -- 172.21.15.88:0/2474407040 <== mon.4 v2:172.21.15.88:3302/0 4 ==== mgrmap(e 5) v1 ==== 40144+0+0 (secure 0 0 0) 0x7fe55c023030 con 0x7fe57409cad0 2023-09-30T01:36:15.470 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:15.437+0000 7fe56affd700 1 --2- 172.21.15.88:0/2474407040 >> v2:172.21.15.107:6812/29926 conn(0x7fe554033360 0x7fe554035810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:36:15.470 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:15.437+0000 7fe56affd700 1 -- 172.21.15.88:0/2474407040 <== mon.4 v2:172.21.15.88:3302/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 3502+0+0 (secure 0 0 0) 0x7fe55c045170 con 0x7fe57409cad0 2023-09-30T01:36:15.470 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:15.437+0000 7fe5727fc700 1 --2- 172.21.15.88:0/2474407040 >> v2:172.21.15.107:6812/29926 conn(0x7fe554033360 0x7fe554035810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:36:15.470 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:15.437+0000 7fe5727fc700 10 monclient: get_auth_request con 0x7fe554033360 auth_method 0 2023-09-30T01:36:15.470 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:15.437+0000 7fe5727fc700 1 --2- 172.21.15.88:0/2474407040 >> v2:172.21.15.107:6812/29926 conn(0x7fe554033360 0x7fe554035810 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fe57406ef20 tx=0x7fe5640091a0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:36:15.471 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:15.445+0000 7fe56affd700 1 -- 172.21.15.88:0/2474407040 <== mon.4 v2:172.21.15.88:3302/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fe55c025bc0 con 0x7fe57409cad0 2023-09-30T01:36:15.471 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:15.445+0000 7fe56affd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:36:15.471 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:15.445+0000 7fe56affd700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:36:15.648 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:15.645+0000 7fe573fff700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T01:36:15.648 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:15.645+0000 7fe573fff700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:36:15.648 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:15.645+0000 7fe573fff700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.88:3302/0 2023-09-30T01:36:15.648 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:15.645+0000 7fe573fff700 1 -- 172.21.15.88:0/2474407040 --> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fe57411dc90 con 0x7fe57409cad0 2023-09-30T01:36:15.649 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:15.645+0000 7fe56affd700 1 -- 172.21.15.88:0/2474407040 <== mon.4 v2:172.21.15.88:3302/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7fe55c02fe60 con 0x7fe57409cad0 2023-09-30T01:36:15.649 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:15.645+0000 7fe56affd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:36:15.649 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:15.645+0000 7fe56affd700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:36:15.649 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-09-30T01:36:15.650 INFO:teuthology.orchestra.run.smithi088.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:36:15.651 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:15.645+0000 7fe573fff700 1 -- 172.21.15.88:0/2474407040 >> v2:172.21.15.107:6812/29926 conn(0x7fe554033360 msgr2=0x7fe554035810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:36:15.651 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:15.645+0000 7fe573fff700 1 --2- 172.21.15.88:0/2474407040 >> v2:172.21.15.107:6812/29926 conn(0x7fe554033360 0x7fe554035810 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fe57406ef20 tx=0x7fe5640091a0 comp rx=0 tx=0).stop 2023-09-30T01:36:15.652 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:15.645+0000 7fe573fff700 10 monclient: shutdown 2023-09-30T01:36:15.652 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:15.645+0000 7fe573fff700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:36:15.652 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:15.645+0000 7fe573fff700 1 -- 172.21.15.88:0/2474407040 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7fe57409cad0 msgr2=0x7fe57412d9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:36:15.652 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:15.645+0000 7fe573fff700 1 --2- 172.21.15.88:0/2474407040 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7fe57409cad0 0x7fe57412d9b0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fe55c0097c0 tx=0x7fe55c019000 comp rx=0 tx=0).stop 2023-09-30T01:36:15.652 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:15.645+0000 7fe573fff700 1 -- 172.21.15.88:0/2474407040 shutdown_connections 2023-09-30T01:36:15.652 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:15.645+0000 7fe573fff700 1 --2- 172.21.15.88:0/2474407040 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7fe574131a10 0x7fe57411aec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:36:15.652 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:15.645+0000 7fe573fff700 1 --2- 172.21.15.88:0/2474407040 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7fe57409d3e0 0x7fe57411a980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:36:15.652 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:15.645+0000 7fe573fff700 1 --2- 172.21.15.88:0/2474407040 >> v2:172.21.15.107:6812/29926 conn(0x7fe554033360 0x7fe554035810 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:36:15.652 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:15.645+0000 7fe573fff700 1 --2- 172.21.15.88:0/2474407040 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7fe57409cad0 0x7fe57412d9b0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:36:15.652 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:15.645+0000 7fe573fff700 1 -- 172.21.15.88:0/2474407040 >> 172.21.15.88:0/2474407040 conn(0x7fe5740687a0 msgr2=0x7fe57409b460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:36:15.653 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:15.645+0000 7fe573fff700 1 -- 172.21.15.88:0/2474407040 shutdown_connections 2023-09-30T01:36:15.653 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:15.645+0000 7fe573fff700 1 -- 172.21.15.88:0/2474407040 wait complete. 2023-09-30T01:36:18.664 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:36:18.774 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:18.769+0000 7fe215c1d700 10 monclient: get_monmap_and_config 2023-09-30T01:36:18.774 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:18.769+0000 7fe215c1d700 10 monclient: build_initial_monmap 2023-09-30T01:36:18.774 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:18.769+0000 7fe215c1d700 10 monclient: monmap: 2023-09-30T01:36:18.774 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2023-09-30T01:36:18.774 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:36:18.774 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:36:18.774068+0000 2023-09-30T01:36:18.774 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:36:18.774068+0000 2023-09-30T01:36:18.775 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2023-09-30T01:36:18.775 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:36:18.775 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a-legacy 2023-09-30T01:36:18.775 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.107:6789/0 mon.noname-f-legacy 2023-09-30T01:36:18.775 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.noname-b 2023-09-30T01:36:18.775 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.noname-g 2023-09-30T01:36:18.775 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.noname-c 2023-09-30T01:36:18.775 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.noname-h 2023-09-30T01:36:18.775 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.noname-d 2023-09-30T01:36:18.775 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.noname-i 2023-09-30T01:36:18.775 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.noname-e 2023-09-30T01:36:18.775 INFO:teuthology.orchestra.run.smithi088.stderr:9: v2:172.21.15.88:3300/0 mon.noname-a 2023-09-30T01:36:18.776 INFO:teuthology.orchestra.run.smithi088.stderr:10: v2:172.21.15.107:3300/0 mon.noname-f 2023-09-30T01:36:18.776 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:36:18.780 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:18.769+0000 7fe215c1d700 1 Processor -- start 2023-09-30T01:36:18.780 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:18.769+0000 7fe215c1d700 1 -- start start 2023-09-30T01:36:18.780 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:18.769+0000 7fe215c1d700 10 monclient: init 2023-09-30T01:36:18.780 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:18.769+0000 7fe215c1d700 10 monclient: _reopen_session rank -1 2023-09-30T01:36:18.780 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:18.769+0000 7fe215c1d700 10 monclient: _add_conns ranks=[1,4,5,6,0,9,3,8,7,10,2] 2023-09-30T01:36:18.780 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:18.769+0000 7fe215c1d700 10 monclient(hunting): picked mon.noname-f-legacy con 0x7fe2100a3270 addr v1:172.21.15.107:6789/0 2023-09-30T01:36:18.781 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:18.769+0000 7fe215c1d700 1 --2- >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7fe2100a4b00 0x7fe2100a0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:36:18.781 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:18.769+0000 7fe215c1d700 10 monclient(hunting): picked mon.noname-c con 0x7fe2100a4b00 addr [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] 2023-09-30T01:36:18.781 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:18.769+0000 7fe215c1d700 1 --2- >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7fe210138750 0x7fe21013ab30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:36:18.781 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:18.769+0000 7fe215c1d700 10 monclient(hunting): picked mon.noname-h con 0x7fe210138750 addr [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] 2023-09-30T01:36:18.781 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:18.769+0000 7fe215c1d700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe21005ecb0 con 0x7fe2100a3270 2023-09-30T01:36:18.781 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:18.769+0000 7fe215c1d700 10 monclient(hunting): start opening mon connection 2023-09-30T01:36:18.782 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:18.769+0000 7fe215c1d700 1 -- --> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] -- mon_getmap magic: 0 v1 -- 0x7fe2100a4ed0 con 0x7fe2100a4b00 2023-09-30T01:36:18.782 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:18.769+0000 7fe215c1d700 10 monclient(hunting): start opening mon connection 2023-09-30T01:36:18.782 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:18.769+0000 7fe215c1d700 1 -- --> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] -- mon_getmap magic: 0 v1 -- 0x7fe2100a6330 con 0x7fe210138750 2023-09-30T01:36:18.782 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:18.769+0000 7fe215c1d700 10 monclient(hunting): _renew_subs 2023-09-30T01:36:18.782 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:18.769+0000 7fe215c1d700 10 monclient(hunting): authenticate will time out at 1050.811833s 2023-09-30T01:36:18.782 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:18.773+0000 7fe20effd700 1 --2- >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7fe2100a4b00 0x7fe2100a0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:36:18.782 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:18.773+0000 7fe20effd700 1 --2- >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7fe2100a4b00 0x7fe2100a0870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3302/0 says I am v2:172.21.15.88:52372/0 (socket says 172.21.15.88:52372) 2023-09-30T01:36:18.782 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:18.773+0000 7fe20effd700 1 -- 172.21.15.88:0/2982810478 learned_addr learned my addr 172.21.15.88:0/2982810478 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-09-30T01:36:18.783 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:18.773+0000 7fe20effd700 10 monclient(hunting): get_auth_request con 0x7fe2100a4b00 auth_method 0 2023-09-30T01:36:18.783 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:18.773+0000 7fe20effd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:36:18.783 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:18.773+0000 7fe20effd700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:36:18.783 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:18.773+0000 7fe20f7fe700 1 --1- 172.21.15.88:0/2982810478 >> v1:172.21.15.107:6789/0 conn(0x7fe2100a3270 0x7fe2100a0160 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.88:57994/0 (socket says 172.21.15.88:57994) 2023-09-30T01:36:18.783 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:18.773+0000 7fe20effd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:36:18.783 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:18.773+0000 7fe20ffff700 1 --2- 172.21.15.88:0/2982810478 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7fe210138750 0x7fe21013ab30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:36:18.783 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:18.773+0000 7fe20effd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:36:18.783 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:18.773+0000 7fe20effd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:36:18.783 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:18.773+0000 7fe20effd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:36:18.783 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:18.773+0000 7fe20ffff700 10 monclient(hunting): get_auth_request con 0x7fe210138750 auth_method 0 2023-09-30T01:36:18.783 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:18.773+0000 7fe20ffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:36:18.784 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:18.773+0000 7fe20ffff700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:36:18.784 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:18.773+0000 7fe20ffff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:36:18.784 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:18.773+0000 7fe20ffff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:36:18.784 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:18.773+0000 7fe20ffff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:36:18.784 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:18.773+0000 7fe20ffff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:36:18.784 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:18.773+0000 7fe20effd700 10 monclient(hunting): handle_auth_done global_id 4684 payload 274 2023-09-30T01:36:18.784 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:18.773+0000 7fe20effd700 1 -- 172.21.15.88:0/2982810478 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7fe210138750 msgr2=0x7fe21013ab30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:36:18.784 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:18.773+0000 7fe20effd700 1 --2- 172.21.15.88:0/2982810478 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7fe210138750 0x7fe21013ab30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:36:18.785 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:18.773+0000 7fe20effd700 1 -- 172.21.15.88:0/2982810478 >> v1:172.21.15.107:6789/0 conn(0x7fe2100a3270 legacy=0x7fe2100a0160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:36:18.785 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:18.773+0000 7fe20effd700 10 monclient: _finish_hunting 0 2023-09-30T01:36:18.785 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:18.773+0000 7fe20effd700 1 monclient: found mon.noname-c 2023-09-30T01:36:18.785 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:18.773+0000 7fe20effd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:36:18.785 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:18.773+0000 7fe20effd700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.88:3302/0 2023-09-30T01:36:18.785 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:18.773+0000 7fe20effd700 1 -- 172.21.15.88:0/2982810478 --> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe210098610 con 0x7fe2100a4b00 2023-09-30T01:36:18.785 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:18.773+0000 7fe20effd700 1 --2- 172.21.15.88:0/2982810478 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7fe2100a4b00 0x7fe2100a0870 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fe200009f50 tx=0x7fe200009840 comp rx=0 tx=0).ready entity=mon.4 client_cookie=b9b695ea1d8120f9 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:36:18.785 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:18.773+0000 7fe20e7fc700 1 -- 172.21.15.88:0/2982810478 <== mon.4 v2:172.21.15.88:3302/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7fe20000e070 con 0x7fe2100a4b00 2023-09-30T01:36:18.785 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:18.773+0000 7fe20e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:36:18.785 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:18.773+0000 7fe20e7fc700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T01:36:18.785 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:18.773+0000 7fe20e7fc700 10 monclient: dump: 2023-09-30T01:36:18.786 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:36:18.786 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:36:18.786 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:36:18.786 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:36:18.786 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:36:18.786 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:36:18.786 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:36:18.786 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:36:18.786 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:36:18.786 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:36:18.786 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:36:18.786 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:36:18.787 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:36:18.787 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:36:18.787 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:36:18.787 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:36:18.787 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:18.773+0000 7fe20e7fc700 10 monclient: _finish_auth 0 2023-09-30T01:36:18.787 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:18.773+0000 7fe20e7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:36:18.787 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:18.773+0000 7fe215c1d700 5 monclient: authenticate success, global_id 4684 2023-09-30T01:36:18.787 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:18.773+0000 7fe215c1d700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:36:18.787 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:18.773+0000 7fe20e7fc700 1 -- 172.21.15.88:0/2982810478 <== mon.4 v2:172.21.15.88:3302/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe20000be80 con 0x7fe2100a4b00 2023-09-30T01:36:18.787 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:18.773+0000 7fe20e7fc700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:36:18.787 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:18.773+0000 7fe215c1d700 10 monclient: get_monmap_and_config success 2023-09-30T01:36:18.788 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:18.773+0000 7fe215c1d700 4 set_mon_vals no callback set 2023-09-30T01:36:18.788 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:18.773+0000 7fe20e7fc700 1 -- 172.21.15.88:0/2982810478 <== mon.4 v2:172.21.15.88:3302/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7fe2000092e0 con 0x7fe2100a4b00 2023-09-30T01:36:18.788 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:18.773+0000 7fe20e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:36:18.788 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:18.773+0000 7fe20e7fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T01:36:18.788 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:18.773+0000 7fe20e7fc700 10 monclient: dump: 2023-09-30T01:36:18.788 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:36:18.788 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:36:18.788 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:36:18.788 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:36:18.788 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:36:18.788 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:36:18.789 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:36:18.789 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:36:18.789 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:36:18.789 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:36:18.789 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:36:18.789 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:36:18.789 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:36:18.789 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:36:18.789 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:36:18.789 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:36:18.789 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:18.773+0000 7fe215c1d700 10 monclient: shutdown 2023-09-30T01:36:18.789 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:18.773+0000 7fe215c1d700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:36:18.790 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:18.773+0000 7fe215c1d700 1 -- 172.21.15.88:0/2982810478 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7fe2100a4b00 msgr2=0x7fe2100a0870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:36:18.790 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:18.773+0000 7fe215c1d700 1 --2- 172.21.15.88:0/2982810478 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7fe2100a4b00 0x7fe2100a0870 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fe200009f50 tx=0x7fe200009840 comp rx=0 tx=0).stop 2023-09-30T01:36:18.790 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:18.773+0000 7fe215c1d700 1 -- 172.21.15.88:0/2982810478 shutdown_connections 2023-09-30T01:36:18.790 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:18.773+0000 7fe215c1d700 1 --2- 172.21.15.88:0/2982810478 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7fe210138750 0x7fe21013ab30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:36:18.790 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:18.773+0000 7fe215c1d700 1 --2- 172.21.15.88:0/2982810478 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7fe2100a4b00 0x7fe2100a0870 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:36:18.790 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:18.773+0000 7fe215c1d700 1 -- 172.21.15.88:0/2982810478 >> 172.21.15.88:0/2982810478 conn(0x7fe2100a67f0 msgr2=0x7fe2100a6bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:36:18.790 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:18.773+0000 7fe215c1d700 1 -- 172.21.15.88:0/2982810478 shutdown_connections 2023-09-30T01:36:18.790 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:18.773+0000 7fe215c1d700 1 -- 172.21.15.88:0/2982810478 wait complete. 2023-09-30T01:36:18.790 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:18.773+0000 7fe215c1d700 10 monclient: build_initial_monmap 2023-09-30T01:36:18.790 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:18.773+0000 7fe215c1d700 10 monclient: monmap: 2023-09-30T01:36:18.791 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2023-09-30T01:36:18.791 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:36:18.791 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 0.000000 2023-09-30T01:36:18.791 INFO:teuthology.orchestra.run.smithi088.stderr:created 0.000000 2023-09-30T01:36:18.791 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2023-09-30T01:36:18.791 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 1 2023-09-30T01:36:18.791 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.noname-a 2023-09-30T01:36:18.791 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.noname-f 2023-09-30T01:36:18.791 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.noname-b 2023-09-30T01:36:18.791 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.noname-g 2023-09-30T01:36:18.791 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.noname-c 2023-09-30T01:36:18.791 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.noname-h 2023-09-30T01:36:18.792 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.noname-d 2023-09-30T01:36:18.792 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.noname-i 2023-09-30T01:36:18.792 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.noname-e 2023-09-30T01:36:18.792 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:36:18.792 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:18.777+0000 7fe215c1d700 1 Processor -- start 2023-09-30T01:36:18.792 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:18.777+0000 7fe215c1d700 1 -- start start 2023-09-30T01:36:18.792 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:18.777+0000 7fe215c1d700 10 monclient: init 2023-09-30T01:36:18.798 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:18.777+0000 7fe215c1d700 10 monclient: _reopen_session rank -1 2023-09-30T01:36:18.798 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:18.777+0000 7fe215c1d700 10 monclient: _add_conns ranks=[6,1,7,3,2,4,0,8,5] 2023-09-30T01:36:18.798 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:18.777+0000 7fe215c1d700 1 --2- >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7fe2100a3270 0x7fe2101300c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:36:18.798 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:18.777+0000 7fe215c1d700 10 monclient(hunting): picked mon.noname-d con 0x7fe2100a3270 addr [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] 2023-09-30T01:36:18.798 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:18.777+0000 7fe215c1d700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe2100a4b00 0x7fe21009bdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:36:18.798 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:18.777+0000 7fe215c1d700 10 monclient(hunting): picked mon.noname-f con 0x7fe2100a4b00 addr [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] 2023-09-30T01:36:18.798 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:18.777+0000 7fe215c1d700 1 --2- >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7fe210138750 0x7fe21009c300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:36:18.799 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:18.777+0000 7fe215c1d700 10 monclient(hunting): picked mon.noname-i con 0x7fe210138750 addr [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] 2023-09-30T01:36:18.799 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:18.777+0000 7fe215c1d700 10 monclient(hunting): start opening mon connection 2023-09-30T01:36:18.799 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:18.777+0000 7fe215c1d700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe210131090 con 0x7fe2100a4b00 2023-09-30T01:36:18.799 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:18.777+0000 7fe215c1d700 10 monclient(hunting): start opening mon connection 2023-09-30T01:36:18.799 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:18.777+0000 7fe215c1d700 1 -- --> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] -- mon_getmap magic: 0 v1 -- 0x7fe210097c70 con 0x7fe2100a3270 2023-09-30T01:36:18.799 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:18.777+0000 7fe215c1d700 10 monclient(hunting): start opening mon connection 2023-09-30T01:36:18.799 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:18.777+0000 7fe215c1d700 1 -- --> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] -- mon_getmap magic: 0 v1 -- 0x7fe2100a0e80 con 0x7fe210138750 2023-09-30T01:36:18.799 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:18.777+0000 7fe215c1d700 10 monclient(hunting): _renew_subs 2023-09-30T01:36:18.799 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:18.777+0000 7fe215c1d700 10 monclient(hunting): authenticate will time out at 1050.817949s 2023-09-30T01:36:18.799 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:18.777+0000 7fe20f7fe700 1 --2- >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7fe2100a3270 0x7fe2101300c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:36:18.799 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:18.777+0000 7fe20f7fe700 1 --2- >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7fe2100a3270 0x7fe2101300c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3303/0 says I am v2:172.21.15.88:56618/0 (socket says 172.21.15.88:56618) 2023-09-30T01:36:18.800 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:18.777+0000 7fe20f7fe700 1 -- 172.21.15.88:0/3741515969 learned_addr learned my addr 172.21.15.88:0/3741515969 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-09-30T01:36:18.800 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:18.777+0000 7fe20f7fe700 10 monclient(hunting): get_auth_request con 0x7fe2100a3270 auth_method 0 2023-09-30T01:36:18.800 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:18.777+0000 7fe20f7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:36:18.800 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:18.777+0000 7fe20f7fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:36:18.800 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:18.777+0000 7fe20f7fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:36:18.800 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:18.777+0000 7fe20effd700 1 --2- 172.21.15.88:0/3741515969 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe2100a4b00 0x7fe21009bdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:36:18.800 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:18.777+0000 7fe20ffff700 1 --2- 172.21.15.88:0/3741515969 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7fe210138750 0x7fe21009c300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:36:18.800 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:18.777+0000 7fe20effd700 10 monclient(hunting): get_auth_request con 0x7fe2100a4b00 auth_method 0 2023-09-30T01:36:18.800 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:18.777+0000 7fe20effd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:36:18.800 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:18.777+0000 7fe20effd700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:36:18.800 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:18.777+0000 7fe20effd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:36:18.801 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:18.777+0000 7fe20f7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:36:18.801 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:18.777+0000 7fe20f7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:36:18.801 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:18.777+0000 7fe20f7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:36:18.801 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:18.777+0000 7fe20ffff700 10 monclient(hunting): get_auth_request con 0x7fe210138750 auth_method 0 2023-09-30T01:36:18.801 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:18.777+0000 7fe20ffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:36:18.801 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:18.777+0000 7fe20ffff700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:36:18.801 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:18.777+0000 7fe20ffff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:36:18.801 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:18.777+0000 7fe20effd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:36:18.801 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:18.777+0000 7fe20effd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:36:18.801 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:18.777+0000 7fe20effd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:36:18.801 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:18.777+0000 7fe20ffff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:36:18.802 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:18.777+0000 7fe20ffff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:36:18.802 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:18.777+0000 7fe20ffff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:36:18.802 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:18.777+0000 7fe20f7fe700 10 monclient(hunting): handle_auth_done global_id 4587 payload 824 2023-09-30T01:36:18.802 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:18.777+0000 7fe20f7fe700 1 -- 172.21.15.88:0/3741515969 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7fe210138750 msgr2=0x7fe21009c300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:36:18.802 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:18.777+0000 7fe20f7fe700 1 --2- 172.21.15.88:0/3741515969 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7fe210138750 0x7fe21009c300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:36:18.802 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:18.777+0000 7fe20f7fe700 1 -- 172.21.15.88:0/3741515969 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe2100a4b00 msgr2=0x7fe21009bdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:36:18.802 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:18.777+0000 7fe20f7fe700 1 --2- 172.21.15.88:0/3741515969 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe2100a4b00 0x7fe21009bdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:36:18.802 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:18.777+0000 7fe20f7fe700 10 monclient: _finish_hunting 0 2023-09-30T01:36:18.802 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:18.777+0000 7fe20f7fe700 1 monclient: found mon.noname-d 2023-09-30T01:36:18.802 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:18.777+0000 7fe20f7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:36:18.802 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:18.777+0000 7fe20f7fe700 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.88:3303/0 2023-09-30T01:36:18.803 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:18.777+0000 7fe20f7fe700 1 -- 172.21.15.88:0/3741515969 --> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe200009ae0 con 0x7fe2100a3270 2023-09-30T01:36:18.803 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:18.777+0000 7fe20f7fe700 1 --2- 172.21.15.88:0/3741515969 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7fe2100a3270 0x7fe2101300c0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fe20400c7f0 tx=0x7fe204007f70 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:36:18.803 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:18.777+0000 7fe1f77fe700 1 -- 172.21.15.88:0/3741515969 <== mon.6 v2:172.21.15.88:3303/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7fe2040011f0 con 0x7fe2100a3270 2023-09-30T01:36:18.803 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:18.777+0000 7fe1f77fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:36:18.803 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:18.777+0000 7fe1f77fe700 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2023-09-30T01:36:18.803 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:18.777+0000 7fe1f77fe700 10 monclient: dump: 2023-09-30T01:36:18.803 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:36:18.803 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:36:18.803 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:36:18.803 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:36:18.803 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:36:18.804 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:36:18.804 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:36:18.804 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:36:18.804 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:36:18.804 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:36:18.804 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:36:18.804 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:36:18.804 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:36:18.804 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:36:18.804 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:36:18.804 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:36:18.804 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:18.777+0000 7fe1f77fe700 10 monclient: _finish_auth 0 2023-09-30T01:36:18.805 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:18.777+0000 7fe1f77fe700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:36:18.805 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:18.777+0000 7fe215c1d700 5 monclient: authenticate success, global_id 4587 2023-09-30T01:36:18.805 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:18.777+0000 7fe1f77fe700 1 -- 172.21.15.88:0/3741515969 <== mon.6 v2:172.21.15.88:3303/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe2040070d0 con 0x7fe2100a3270 2023-09-30T01:36:18.805 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:18.777+0000 7fe1f77fe700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:36:18.805 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:18.777+0000 7fe215c1d700 10 monclient: _renew_subs 2023-09-30T01:36:18.805 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:18.777+0000 7fe1f77fe700 1 -- 172.21.15.88:0/3741515969 <== mon.6 v2:172.21.15.88:3303/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7fe2040076c0 con 0x7fe2100a3270 2023-09-30T01:36:18.805 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:18.777+0000 7fe215c1d700 10 monclient: _send_mon_message to mon.d at v2:172.21.15.88:3303/0 2023-09-30T01:36:18.805 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:18.777+0000 7fe215c1d700 1 -- 172.21.15.88:0/3741515969 --> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe21009e5a0 con 0x7fe2100a3270 2023-09-30T01:36:18.805 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:18.777+0000 7fe1f77fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:36:18.805 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:18.777+0000 7fe20d7fa700 4 set_mon_vals no callback set 2023-09-30T01:36:18.805 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:18.777+0000 7fe1f77fe700 10 monclient: got monmap 1 from mon.d (according to old e1) 2023-09-30T01:36:18.806 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:18.777+0000 7fe1f77fe700 10 monclient: dump: 2023-09-30T01:36:18.806 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:36:18.806 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:36:18.806 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:36:18.806 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:36:18.806 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:36:18.806 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:36:18.806 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:36:18.806 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:36:18.806 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:36:18.806 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:36:18.807 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:36:18.807 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:36:18.807 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:36:18.807 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:36:18.807 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:36:18.807 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:36:18.807 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:18.777+0000 7fe215c1d700 10 monclient: _renew_subs 2023-09-30T01:36:18.807 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:18.777+0000 7fe215c1d700 10 monclient: _send_mon_message to mon.d at v2:172.21.15.88:3303/0 2023-09-30T01:36:18.807 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:18.777+0000 7fe215c1d700 1 -- 172.21.15.88:0/3741515969 --> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe210064b80 con 0x7fe2100a3270 2023-09-30T01:36:18.807 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:18.781+0000 7fe1f77fe700 1 -- 172.21.15.88:0/3741515969 <== mon.6 v2:172.21.15.88:3303/0 4 ==== mgrmap(e 5) v1 ==== 40144+0+0 (secure 0 0 0) 0x7fe204007240 con 0x7fe2100a3270 2023-09-30T01:36:18.807 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:18.781+0000 7fe215c1d700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:36:18.808 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:18.781+0000 7fe215c1d700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:36:18.808 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:18.781+0000 7fe215c1d700 10 monclient: _send_mon_message to mon.d at v2:172.21.15.88:3303/0 2023-09-30T01:36:18.808 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:18.781+0000 7fe215c1d700 1 -- 172.21.15.88:0/3741515969 --> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe210071960 con 0x7fe2100a3270 2023-09-30T01:36:18.808 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:18.781+0000 7fe1f77fe700 1 --2- 172.21.15.88:0/3741515969 >> v2:172.21.15.107:6812/29926 conn(0x7fe1fc033390 0x7fe1fc035840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:36:18.808 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:18.781+0000 7fe1f77fe700 1 -- 172.21.15.88:0/3741515969 <== mon.6 v2:172.21.15.88:3303/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 3502+0+0 (secure 0 0 0) 0x7fe2040423f0 con 0x7fe2100a3270 2023-09-30T01:36:18.808 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:18.781+0000 7fe20effd700 1 --2- 172.21.15.88:0/3741515969 >> v2:172.21.15.107:6812/29926 conn(0x7fe1fc033390 0x7fe1fc035840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:36:18.808 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:18.781+0000 7fe20effd700 10 monclient: get_auth_request con 0x7fe1fc033390 auth_method 0 2023-09-30T01:36:18.808 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:18.781+0000 7fe20effd700 1 --2- 172.21.15.88:0/3741515969 >> v2:172.21.15.107:6812/29926 conn(0x7fe1fc033390 0x7fe1fc035840 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fe200009f50 tx=0x7fe200018000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:36:18.808 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:18.789+0000 7fe1f77fe700 1 -- 172.21.15.88:0/3741515969 <== mon.6 v2:172.21.15.88:3303/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fe204018860 con 0x7fe2100a3270 2023-09-30T01:36:18.808 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:18.789+0000 7fe1f77fe700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:36:18.808 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:18.789+0000 7fe1f77fe700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:36:18.990 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:18.985+0000 7fe215c1d700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T01:36:18.991 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:18.985+0000 7fe215c1d700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:36:18.991 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:18.985+0000 7fe215c1d700 10 monclient: _send_mon_message to mon.d at v2:172.21.15.88:3303/0 2023-09-30T01:36:18.991 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:18.985+0000 7fe215c1d700 1 -- 172.21.15.88:0/3741515969 --> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fe210137f70 con 0x7fe2100a3270 2023-09-30T01:36:18.991 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:18.985+0000 7fe1f77fe700 1 -- 172.21.15.88:0/3741515969 <== mon.6 v2:172.21.15.88:3303/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7fe204017020 con 0x7fe2100a3270 2023-09-30T01:36:18.991 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:18.985+0000 7fe1f77fe700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:36:18.992 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:18.985+0000 7fe1f77fe700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:36:18.992 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-09-30T01:36:18.992 INFO:teuthology.orchestra.run.smithi088.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:36:18.994 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:18.989+0000 7fe215c1d700 1 -- 172.21.15.88:0/3741515969 >> v2:172.21.15.107:6812/29926 conn(0x7fe1fc033390 msgr2=0x7fe1fc035840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:36:18.994 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:18.989+0000 7fe215c1d700 1 --2- 172.21.15.88:0/3741515969 >> v2:172.21.15.107:6812/29926 conn(0x7fe1fc033390 0x7fe1fc035840 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fe200009f50 tx=0x7fe200018000 comp rx=0 tx=0).stop 2023-09-30T01:36:18.994 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:18.989+0000 7fe215c1d700 10 monclient: shutdown 2023-09-30T01:36:18.994 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:18.989+0000 7fe215c1d700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:36:18.995 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:18.989+0000 7fe215c1d700 1 -- 172.21.15.88:0/3741515969 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7fe2100a3270 msgr2=0x7fe2101300c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:36:18.995 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:18.989+0000 7fe215c1d700 1 --2- 172.21.15.88:0/3741515969 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7fe2100a3270 0x7fe2101300c0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fe20400c7f0 tx=0x7fe204007f70 comp rx=0 tx=0).stop 2023-09-30T01:36:18.995 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:18.989+0000 7fe215c1d700 1 -- 172.21.15.88:0/3741515969 shutdown_connections 2023-09-30T01:36:18.995 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:18.989+0000 7fe215c1d700 1 --2- 172.21.15.88:0/3741515969 >> v2:172.21.15.107:6812/29926 conn(0x7fe1fc033390 0x7fe1fc035840 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:36:18.995 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:18.989+0000 7fe215c1d700 1 --2- 172.21.15.88:0/3741515969 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7fe210138750 0x7fe21009c300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:36:18.995 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:18.989+0000 7fe215c1d700 1 --2- 172.21.15.88:0/3741515969 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe2100a4b00 0x7fe21009bdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:36:18.995 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:18.989+0000 7fe215c1d700 1 --2- 172.21.15.88:0/3741515969 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7fe2100a3270 0x7fe2101300c0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:36:18.996 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:18.989+0000 7fe215c1d700 1 -- 172.21.15.88:0/3741515969 >> 172.21.15.88:0/3741515969 conn(0x7fe2100a67f0 msgr2=0x7fe210134210 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:36:18.996 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:18.989+0000 7fe215c1d700 1 -- 172.21.15.88:0/3741515969 shutdown_connections 2023-09-30T01:36:18.996 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:18.989+0000 7fe215c1d700 1 -- 172.21.15.88:0/3741515969 wait complete. 2023-09-30T01:36:22.005 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:36:22.119 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:22.113+0000 7ff7ab59e700 10 monclient: get_monmap_and_config 2023-09-30T01:36:22.119 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:22.113+0000 7ff7ab59e700 10 monclient: build_initial_monmap 2023-09-30T01:36:22.120 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:22.113+0000 7ff7ab59e700 10 monclient: monmap: 2023-09-30T01:36:22.120 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2023-09-30T01:36:22.120 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:36:22.120 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:36:22.119531+0000 2023-09-30T01:36:22.120 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:36:22.119531+0000 2023-09-30T01:36:22.120 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2023-09-30T01:36:22.120 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:36:22.120 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a-legacy 2023-09-30T01:36:22.120 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.107:6789/0 mon.noname-f-legacy 2023-09-30T01:36:22.120 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.noname-b 2023-09-30T01:36:22.120 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.noname-g 2023-09-30T01:36:22.121 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.noname-c 2023-09-30T01:36:22.121 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.noname-h 2023-09-30T01:36:22.121 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.noname-d 2023-09-30T01:36:22.121 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.noname-i 2023-09-30T01:36:22.121 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.noname-e 2023-09-30T01:36:22.121 INFO:teuthology.orchestra.run.smithi088.stderr:9: v2:172.21.15.88:3300/0 mon.noname-a 2023-09-30T01:36:22.121 INFO:teuthology.orchestra.run.smithi088.stderr:10: v2:172.21.15.107:3300/0 mon.noname-f 2023-09-30T01:36:22.121 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:36:22.129 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:22.117+0000 7ff7ab59e700 1 Processor -- start 2023-09-30T01:36:22.129 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:22.117+0000 7ff7ab59e700 1 -- start start 2023-09-30T01:36:22.129 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:22.117+0000 7ff7ab59e700 10 monclient: init 2023-09-30T01:36:22.129 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:22.117+0000 7ff7ab59e700 10 monclient: _reopen_session rank -1 2023-09-30T01:36:22.129 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:22.117+0000 7ff7ab59e700 10 monclient: _add_conns ranks=[3,9,2,7,6,5,0,10,4,8,1] 2023-09-30T01:36:22.130 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:22.117+0000 7ff7ab59e700 1 --2- >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7ff7ac09cf30 0x7ff7ac09d300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:36:22.130 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:22.117+0000 7ff7ab59e700 10 monclient(hunting): picked mon.noname-g con 0x7ff7ac09cf30 addr [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] 2023-09-30T01:36:22.130 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:22.117+0000 7ff7ab59e700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7ff7ac06f480 0x7ff7ac09d840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:36:22.130 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:22.117+0000 7ff7ab59e700 10 monclient(hunting): picked mon.noname-a con 0x7ff7ac06f480 addr v2:172.21.15.88:3300/0 2023-09-30T01:36:22.130 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:22.117+0000 7ff7ab59e700 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff7ac097a80 0x7ff7ac1316b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:36:22.130 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:22.117+0000 7ff7ab59e700 10 monclient(hunting): picked mon.noname-b con 0x7ff7ac097a80 addr [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] 2023-09-30T01:36:22.130 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:22.117+0000 7ff7ab59e700 10 monclient(hunting): start opening mon connection 2023-09-30T01:36:22.130 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:22.117+0000 7ff7ab59e700 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff7ac096710 con 0x7ff7ac06f480 2023-09-30T01:36:22.130 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:22.117+0000 7ff7ab59e700 10 monclient(hunting): start opening mon connection 2023-09-30T01:36:22.130 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:22.117+0000 7ff7ab59e700 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff7ac09dd80 con 0x7ff7ac097a80 2023-09-30T01:36:22.130 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:22.117+0000 7ff7ab59e700 10 monclient(hunting): start opening mon connection 2023-09-30T01:36:22.131 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:22.117+0000 7ff7ab59e700 1 -- --> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff7ac096960 con 0x7ff7ac09cf30 2023-09-30T01:36:22.131 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:22.117+0000 7ff7ab59e700 10 monclient(hunting): _renew_subs 2023-09-30T01:36:22.131 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:22.117+0000 7ff7ab59e700 10 monclient(hunting): authenticate will time out at 1054.157229s 2023-09-30T01:36:22.131 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:22.117+0000 7ff7a1d9b700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7ff7ac06f480 0x7ff7ac09d840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:36:22.131 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:22.117+0000 7ff7a1d9b700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7ff7ac06f480 0x7ff7ac09d840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:56054/0 (socket says 172.21.15.88:56054) 2023-09-30T01:36:22.131 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:22.117+0000 7ff7a1d9b700 1 -- 172.21.15.88:0/1003486397 learned_addr learned my addr 172.21.15.88:0/1003486397 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-09-30T01:36:22.131 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:22.117+0000 7ff7aad9d700 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff7ac097a80 0x7ff7ac1316b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:36:22.131 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:22.117+0000 7ff7a1d9b700 10 monclient(hunting): get_auth_request con 0x7ff7ac06f480 auth_method 0 2023-09-30T01:36:22.131 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:22.117+0000 7ff7a1d9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:36:22.131 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:22.117+0000 7ff7a1d9b700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:36:22.131 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:22.117+0000 7ff7a1d9b700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:36:22.131 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:22.117+0000 7ff7aad9d700 10 monclient(hunting): get_auth_request con 0x7ff7ac097a80 auth_method 0 2023-09-30T01:36:22.132 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:22.117+0000 7ff7aad9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:36:22.132 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:22.117+0000 7ff7aad9d700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:36:22.132 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:22.117+0000 7ff7aad9d700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:36:22.132 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:22.117+0000 7ff7a1d9b700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:36:22.132 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:22.117+0000 7ff7a1d9b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:36:22.132 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:22.117+0000 7ff7a1d9b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:36:22.132 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:22.117+0000 7ff7aa59c700 1 --2- 172.21.15.88:0/1003486397 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7ff7ac09cf30 0x7ff7ac09d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:36:22.132 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:22.117+0000 7ff7aad9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:36:22.132 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:22.117+0000 7ff7aad9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:36:22.132 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:22.117+0000 7ff7aad9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:36:22.132 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:22.117+0000 7ff7aa59c700 10 monclient(hunting): get_auth_request con 0x7ff7ac09cf30 auth_method 0 2023-09-30T01:36:22.132 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:22.117+0000 7ff7aa59c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:36:22.133 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:22.117+0000 7ff7aa59c700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:36:22.133 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:22.117+0000 7ff7aa59c700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:36:22.133 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:22.117+0000 7ff7a1d9b700 10 monclient(hunting): handle_auth_done global_id 4941 payload 274 2023-09-30T01:36:22.133 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:22.117+0000 7ff7a1d9b700 1 -- 172.21.15.88:0/1003486397 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7ff7ac09cf30 msgr2=0x7ff7ac09d300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:36:22.133 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:22.117+0000 7ff7a1d9b700 1 --2- 172.21.15.88:0/1003486397 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7ff7ac09cf30 0x7ff7ac09d300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:36:22.133 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:22.117+0000 7ff7a1d9b700 1 -- 172.21.15.88:0/1003486397 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff7ac097a80 msgr2=0x7ff7ac1316b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:36:22.133 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:22.117+0000 7ff7a1d9b700 1 --2- 172.21.15.88:0/1003486397 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff7ac097a80 0x7ff7ac1316b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:36:22.133 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:22.117+0000 7ff7a1d9b700 10 monclient: _finish_hunting 0 2023-09-30T01:36:22.133 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:22.117+0000 7ff7a1d9b700 1 monclient: found mon.noname-a 2023-09-30T01:36:22.133 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:22.117+0000 7ff7a1d9b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:36:22.133 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:22.117+0000 7ff7a1d9b700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.88:3300/0 2023-09-30T01:36:22.133 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:22.117+0000 7ff7a1d9b700 1 -- 172.21.15.88:0/1003486397 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7ac04bb20 con 0x7ff7ac06f480 2023-09-30T01:36:22.134 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:22.117+0000 7ff7a1d9b700 1 --2- 172.21.15.88:0/1003486397 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff7ac06f480 0x7ff7ac09d840 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7ff7940099b0 tx=0x7ff794009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=54721ed4cec545ca server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:36:22.134 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:22.117+0000 7ff7a9d9b700 1 -- 172.21.15.88:0/1003486397 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7ff794009ea0 con 0x7ff7ac06f480 2023-09-30T01:36:22.134 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:22.117+0000 7ff7a9d9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:36:22.134 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:22.117+0000 7ff7a9d9b700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T01:36:22.134 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:22.117+0000 7ff7a9d9b700 10 monclient: dump: 2023-09-30T01:36:22.134 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:36:22.134 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:36:22.134 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:36:22.134 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:36:22.134 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:36:22.134 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:36:22.134 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:36:22.135 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:36:22.135 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:36:22.135 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:36:22.135 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:36:22.135 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:36:22.135 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:36:22.135 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:36:22.135 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:36:22.135 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:36:22.135 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:22.117+0000 7ff7a9d9b700 10 monclient: _finish_auth 0 2023-09-30T01:36:22.135 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:22.117+0000 7ff7a9d9b700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:36:22.135 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:22.117+0000 7ff7ab59e700 5 monclient: authenticate success, global_id 4941 2023-09-30T01:36:22.135 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:22.117+0000 7ff7ab59e700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:36:22.136 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:22.117+0000 7ff7a9d9b700 1 -- 172.21.15.88:0/1003486397 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff79400bce0 con 0x7ff7ac06f480 2023-09-30T01:36:22.136 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:22.117+0000 7ff7a9d9b700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:36:22.136 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:22.117+0000 7ff7ab59e700 10 monclient: get_monmap_and_config success 2023-09-30T01:36:22.136 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:22.117+0000 7ff7a9d9b700 1 -- 172.21.15.88:0/1003486397 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7ff79400be50 con 0x7ff7ac06f480 2023-09-30T01:36:22.136 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:22.117+0000 7ff7ab59e700 4 set_mon_vals no callback set 2023-09-30T01:36:22.136 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:22.117+0000 7ff7a9d9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:36:22.136 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:22.117+0000 7ff7a9d9b700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:36:22.136 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:22.117+0000 7ff7a9d9b700 10 monclient: dump: 2023-09-30T01:36:22.136 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:36:22.136 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:36:22.136 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:36:22.136 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:36:22.137 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:36:22.137 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:36:22.137 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:36:22.137 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:36:22.137 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:36:22.137 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:36:22.137 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:36:22.137 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:36:22.137 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:36:22.137 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:36:22.137 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:36:22.137 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:36:22.137 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:22.117+0000 7ff7ab59e700 10 monclient: shutdown 2023-09-30T01:36:22.138 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:22.117+0000 7ff7ab59e700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:36:22.138 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:22.117+0000 7ff7ab59e700 1 -- 172.21.15.88:0/1003486397 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff7ac06f480 msgr2=0x7ff7ac09d840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:36:22.138 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:22.117+0000 7ff7ab59e700 1 --2- 172.21.15.88:0/1003486397 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff7ac06f480 0x7ff7ac09d840 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7ff7940099b0 tx=0x7ff794009250 comp rx=0 tx=0).stop 2023-09-30T01:36:22.138 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:22.117+0000 7ff7ab59e700 1 -- 172.21.15.88:0/1003486397 shutdown_connections 2023-09-30T01:36:22.138 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:22.117+0000 7ff7ab59e700 1 --2- 172.21.15.88:0/1003486397 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff7ac097a80 0x7ff7ac1316b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:36:22.138 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:22.117+0000 7ff7ab59e700 1 --2- 172.21.15.88:0/1003486397 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff7ac06f480 0x7ff7ac09d840 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:36:22.138 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:22.117+0000 7ff7ab59e700 1 --2- 172.21.15.88:0/1003486397 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7ff7ac09cf30 0x7ff7ac09d300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:36:22.138 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:22.117+0000 7ff7ab59e700 1 -- 172.21.15.88:0/1003486397 >> 172.21.15.88:0/1003486397 conn(0x7ff7ac0a2540 msgr2=0x7ff7ac0a2910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:36:22.138 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:22.117+0000 7ff7ab59e700 1 -- 172.21.15.88:0/1003486397 shutdown_connections 2023-09-30T01:36:22.138 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:22.117+0000 7ff7ab59e700 1 -- 172.21.15.88:0/1003486397 wait complete. 2023-09-30T01:36:22.138 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:22.117+0000 7ff7ab59e700 10 monclient: build_initial_monmap 2023-09-30T01:36:22.138 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:22.117+0000 7ff7ab59e700 10 monclient: monmap: 2023-09-30T01:36:22.139 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2023-09-30T01:36:22.139 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:36:22.139 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 0.000000 2023-09-30T01:36:22.139 INFO:teuthology.orchestra.run.smithi088.stderr:created 0.000000 2023-09-30T01:36:22.139 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2023-09-30T01:36:22.139 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 1 2023-09-30T01:36:22.139 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.noname-a 2023-09-30T01:36:22.139 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.noname-f 2023-09-30T01:36:22.139 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.noname-b 2023-09-30T01:36:22.139 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.noname-g 2023-09-30T01:36:22.139 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.noname-c 2023-09-30T01:36:22.139 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.noname-h 2023-09-30T01:36:22.139 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.noname-d 2023-09-30T01:36:22.140 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.noname-i 2023-09-30T01:36:22.140 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.noname-e 2023-09-30T01:36:22.140 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:36:22.140 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:22.121+0000 7ff7ab59e700 1 Processor -- start 2023-09-30T01:36:22.140 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:22.121+0000 7ff7ab59e700 1 -- start start 2023-09-30T01:36:22.140 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:22.121+0000 7ff7ab59e700 10 monclient: init 2023-09-30T01:36:22.140 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:22.121+0000 7ff7ab59e700 10 monclient: _reopen_session rank -1 2023-09-30T01:36:22.140 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:22.121+0000 7ff7ab59e700 10 monclient: _add_conns ranks=[7,2,6,1,0,5,3,4,8] 2023-09-30T01:36:22.140 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:22.121+0000 7ff7ab59e700 1 --2- >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7ff7ac06f480 0x7ff7ac1427c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:36:22.140 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:22.121+0000 7ff7ab59e700 10 monclient(hunting): picked mon.noname-i con 0x7ff7ac06f480 addr [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] 2023-09-30T01:36:22.140 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:22.121+0000 7ff7ab59e700 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff7ac097a80 0x7ff7ac142d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:36:22.140 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:22.121+0000 7ff7ab59e700 10 monclient(hunting): picked mon.noname-b con 0x7ff7ac097a80 addr [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] 2023-09-30T01:36:22.141 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:22.121+0000 7ff7ab59e700 1 --2- >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7ff7ac09cf30 0x7ff7ac143240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:36:22.141 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:22.121+0000 7ff7ab59e700 10 monclient(hunting): picked mon.noname-d con 0x7ff7ac09cf30 addr [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] 2023-09-30T01:36:22.141 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:22.121+0000 7ff7ab59e700 10 monclient(hunting): start opening mon connection 2023-09-30T01:36:22.141 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:22.121+0000 7ff7ab59e700 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff7ac091240 con 0x7ff7ac097a80 2023-09-30T01:36:22.141 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:22.121+0000 7ff7ab59e700 10 monclient(hunting): start opening mon connection 2023-09-30T01:36:22.141 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:22.121+0000 7ff7ab59e700 1 -- --> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] -- mon_getmap magic: 0 v1 -- 0x7ff7ac099240 con 0x7ff7ac09cf30 2023-09-30T01:36:22.141 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:22.121+0000 7ff7ab59e700 10 monclient(hunting): start opening mon connection 2023-09-30T01:36:22.141 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:22.121+0000 7ff7ab59e700 1 -- --> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] -- mon_getmap magic: 0 v1 -- 0x7ff7ac092310 con 0x7ff7ac06f480 2023-09-30T01:36:22.141 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:22.121+0000 7ff7ab59e700 10 monclient(hunting): _renew_subs 2023-09-30T01:36:22.141 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:22.121+0000 7ff7ab59e700 10 monclient(hunting): authenticate will time out at 1054.161358s 2023-09-30T01:36:22.141 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:22.121+0000 7ff7aad9d700 1 --2- >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7ff7ac09cf30 0x7ff7ac143240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:36:22.142 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:22.121+0000 7ff7aad9d700 1 --2- >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7ff7ac09cf30 0x7ff7ac143240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3303/0 says I am v2:172.21.15.88:37878/0 (socket says 172.21.15.88:37878) 2023-09-30T01:36:22.142 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:22.121+0000 7ff7aad9d700 1 -- 172.21.15.88:0/3543222787 learned_addr learned my addr 172.21.15.88:0/3543222787 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-09-30T01:36:22.142 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:22.121+0000 7ff7aad9d700 10 monclient(hunting): get_auth_request con 0x7ff7ac09cf30 auth_method 0 2023-09-30T01:36:22.142 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:22.121+0000 7ff7aad9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:36:22.142 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:22.121+0000 7ff7aad9d700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:36:22.142 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:22.121+0000 7ff7aad9d700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:36:22.142 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:22.121+0000 7ff7a1d9b700 1 --2- 172.21.15.88:0/3543222787 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff7ac097a80 0x7ff7ac142d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:36:22.142 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:22.121+0000 7ff7aa59c700 1 --2- 172.21.15.88:0/3543222787 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7ff7ac06f480 0x7ff7ac1427c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:36:22.142 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:22.121+0000 7ff7aa59c700 10 monclient(hunting): get_auth_request con 0x7ff7ac06f480 auth_method 0 2023-09-30T01:36:22.142 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:22.121+0000 7ff7aa59c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:36:22.142 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:22.121+0000 7ff7aa59c700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:36:22.143 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:22.121+0000 7ff7aa59c700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:36:22.143 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:22.121+0000 7ff7a1d9b700 10 monclient(hunting): get_auth_request con 0x7ff7ac097a80 auth_method 0 2023-09-30T01:36:22.143 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:22.121+0000 7ff7a1d9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:36:22.143 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:22.121+0000 7ff7a1d9b700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:36:22.143 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:22.121+0000 7ff7a1d9b700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:36:22.143 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:22.121+0000 7ff7aad9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:36:22.143 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:22.121+0000 7ff7aad9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:36:22.143 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:22.121+0000 7ff7aad9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:36:22.143 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:22.121+0000 7ff7a1d9b700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:36:22.143 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:22.121+0000 7ff7a1d9b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:36:22.143 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:22.121+0000 7ff7a1d9b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:36:22.143 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:22.121+0000 7ff7aa59c700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:36:22.143 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:22.121+0000 7ff7aa59c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:36:22.144 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:22.121+0000 7ff7aa59c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:36:22.144 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:22.121+0000 7ff7aad9d700 10 monclient(hunting): handle_auth_done global_id 4596 payload 824 2023-09-30T01:36:22.144 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:22.121+0000 7ff7aad9d700 1 -- 172.21.15.88:0/3543222787 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7ff7ac06f480 msgr2=0x7ff7ac1427c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:36:22.144 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:22.121+0000 7ff7aad9d700 1 --2- 172.21.15.88:0/3543222787 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7ff7ac06f480 0x7ff7ac1427c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:36:22.144 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:22.121+0000 7ff7aad9d700 1 -- 172.21.15.88:0/3543222787 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff7ac097a80 msgr2=0x7ff7ac142d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:36:22.144 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:22.121+0000 7ff7aad9d700 1 --2- 172.21.15.88:0/3543222787 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff7ac097a80 0x7ff7ac142d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:36:22.144 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:22.121+0000 7ff7aad9d700 10 monclient: _finish_hunting 0 2023-09-30T01:36:22.144 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:22.121+0000 7ff7aad9d700 1 monclient: found mon.noname-d 2023-09-30T01:36:22.144 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:22.121+0000 7ff7aad9d700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:36:22.144 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:22.121+0000 7ff7aad9d700 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.88:3303/0 2023-09-30T01:36:22.144 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:22.121+0000 7ff7aad9d700 1 -- 172.21.15.88:0/3543222787 --> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7940094f0 con 0x7ff7ac09cf30 2023-09-30T01:36:22.144 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:22.121+0000 7ff7aad9d700 1 --2- 172.21.15.88:0/3543222787 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7ff7ac09cf30 0x7ff7ac143240 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7ff7a4009ec0 tx=0x7ff7a400b040 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:36:22.145 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:22.121+0000 7ff7a2ffd700 1 -- 172.21.15.88:0/3543222787 <== mon.6 v2:172.21.15.88:3303/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7ff7a4014480 con 0x7ff7ac09cf30 2023-09-30T01:36:22.145 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:22.121+0000 7ff7a2ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:36:22.145 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:22.121+0000 7ff7a2ffd700 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2023-09-30T01:36:22.145 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:22.121+0000 7ff7a2ffd700 10 monclient: dump: 2023-09-30T01:36:22.145 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:36:22.145 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:36:22.145 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:36:22.145 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:36:22.145 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:36:22.145 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:36:22.145 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:36:22.145 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:36:22.145 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:36:22.146 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:36:22.146 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:36:22.146 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:36:22.146 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:36:22.146 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:36:22.146 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:36:22.146 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:36:22.146 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:22.121+0000 7ff7a2ffd700 10 monclient: _finish_auth 0 2023-09-30T01:36:22.146 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:22.121+0000 7ff7a2ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:36:22.146 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:22.121+0000 7ff7a2ffd700 1 -- 172.21.15.88:0/3543222787 <== mon.6 v2:172.21.15.88:3303/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff7a400f040 con 0x7ff7ac09cf30 2023-09-30T01:36:22.146 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:22.121+0000 7ff7ab59e700 5 monclient: authenticate success, global_id 4596 2023-09-30T01:36:22.146 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:22.121+0000 7ff7ab59e700 10 monclient: _renew_subs 2023-09-30T01:36:22.147 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:22.121+0000 7ff7ab59e700 10 monclient: _send_mon_message to mon.d at v2:172.21.15.88:3303/0 2023-09-30T01:36:22.147 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:22.121+0000 7ff7ab59e700 1 -- 172.21.15.88:0/3543222787 --> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff7ac09b6c0 con 0x7ff7ac09cf30 2023-09-30T01:36:22.147 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:22.121+0000 7ff7a2ffd700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:36:22.147 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:22.121+0000 7ff7a2ffd700 1 -- 172.21.15.88:0/3543222787 <== mon.6 v2:172.21.15.88:3303/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7ff7a4014aa0 con 0x7ff7ac09cf30 2023-09-30T01:36:22.147 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:22.121+0000 7ff7a2ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:36:22.147 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:22.121+0000 7ff7a2ffd700 10 monclient: got monmap 1 from mon.d (according to old e1) 2023-09-30T01:36:22.147 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:22.121+0000 7ff7a2ffd700 10 monclient: dump: 2023-09-30T01:36:22.147 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:36:22.147 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:36:22.147 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:36:22.147 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:36:22.147 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:36:22.147 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:36:22.148 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:36:22.148 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:36:22.148 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:36:22.148 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:36:22.148 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:36:22.148 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:36:22.148 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:36:22.148 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:36:22.148 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:36:22.148 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:36:22.148 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:22.121+0000 7ff7a959a700 4 set_mon_vals no callback set 2023-09-30T01:36:22.148 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:22.121+0000 7ff7ab59e700 10 monclient: _renew_subs 2023-09-30T01:36:22.148 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:22.121+0000 7ff7ab59e700 10 monclient: _send_mon_message to mon.d at v2:172.21.15.88:3303/0 2023-09-30T01:36:22.149 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:22.121+0000 7ff7ab59e700 1 -- 172.21.15.88:0/3543222787 --> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff7ac0932c0 con 0x7ff7ac09cf30 2023-09-30T01:36:22.149 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:22.125+0000 7ff7ab59e700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:36:22.149 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:22.125+0000 7ff7ab59e700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:36:22.149 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:22.125+0000 7ff7ab59e700 10 monclient: _send_mon_message to mon.d at v2:172.21.15.88:3303/0 2023-09-30T01:36:22.149 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:22.125+0000 7ff7ab59e700 1 -- 172.21.15.88:0/3543222787 --> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff7ac0710d0 con 0x7ff7ac09cf30 2023-09-30T01:36:22.149 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:22.125+0000 7ff7a2ffd700 1 -- 172.21.15.88:0/3543222787 <== mon.6 v2:172.21.15.88:3303/0 4 ==== mgrmap(e 5) v1 ==== 40144+0+0 (secure 0 0 0) 0x7ff7a4014c20 con 0x7ff7ac09cf30 2023-09-30T01:36:22.149 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:22.125+0000 7ff7a2ffd700 1 --2- 172.21.15.88:0/3543222787 >> v2:172.21.15.107:6812/29926 conn(0x7ff78c033390 0x7ff78c035840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:36:22.149 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:22.125+0000 7ff7a2ffd700 1 -- 172.21.15.88:0/3543222787 <== mon.6 v2:172.21.15.88:3303/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 3502+0+0 (secure 0 0 0) 0x7ff7a4043cd0 con 0x7ff7ac09cf30 2023-09-30T01:36:22.149 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:22.125+0000 7ff7aa59c700 1 --2- 172.21.15.88:0/3543222787 >> v2:172.21.15.107:6812/29926 conn(0x7ff78c033390 0x7ff78c035840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:36:22.149 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:22.125+0000 7ff7aa59c700 10 monclient: get_auth_request con 0x7ff78c033390 auth_method 0 2023-09-30T01:36:22.149 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:22.125+0000 7ff7aa59c700 1 --2- 172.21.15.88:0/3543222787 >> v2:172.21.15.107:6812/29926 conn(0x7ff78c033390 0x7ff78c035840 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7ff79c006fd0 tx=0x7ff79c008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:36:22.150 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:22.133+0000 7ff7a2ffd700 1 -- 172.21.15.88:0/3543222787 <== mon.6 v2:172.21.15.88:3303/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7ff7a4014900 con 0x7ff7ac09cf30 2023-09-30T01:36:22.150 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:22.133+0000 7ff7a2ffd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:36:22.150 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:22.133+0000 7ff7a2ffd700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:36:22.323 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:22.317+0000 7ff7ab59e700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T01:36:22.323 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:22.317+0000 7ff7ab59e700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:36:22.323 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:22.317+0000 7ff7ab59e700 10 monclient: _send_mon_message to mon.d at v2:172.21.15.88:3303/0 2023-09-30T01:36:22.324 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:22.317+0000 7ff7ab59e700 1 -- 172.21.15.88:0/3543222787 --> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7ff7ac09eda0 con 0x7ff7ac09cf30 2023-09-30T01:36:22.324 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:22.321+0000 7ff7a2ffd700 1 -- 172.21.15.88:0/3543222787 <== mon.6 v2:172.21.15.88:3303/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7ff7a4018020 con 0x7ff7ac09cf30 2023-09-30T01:36:22.324 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:22.321+0000 7ff7a2ffd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:36:22.324 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:22.321+0000 7ff7a2ffd700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:36:22.324 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-09-30T01:36:22.325 INFO:teuthology.orchestra.run.smithi088.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:36:22.326 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:22.321+0000 7ff7ab59e700 1 -- 172.21.15.88:0/3543222787 >> v2:172.21.15.107:6812/29926 conn(0x7ff78c033390 msgr2=0x7ff78c035840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:36:22.326 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:22.321+0000 7ff7ab59e700 1 --2- 172.21.15.88:0/3543222787 >> v2:172.21.15.107:6812/29926 conn(0x7ff78c033390 0x7ff78c035840 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7ff79c006fd0 tx=0x7ff79c008040 comp rx=0 tx=0).stop 2023-09-30T01:36:22.326 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:22.321+0000 7ff7ab59e700 10 monclient: shutdown 2023-09-30T01:36:22.327 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:22.321+0000 7ff7ab59e700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:36:22.327 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:22.321+0000 7ff7ab59e700 1 -- 172.21.15.88:0/3543222787 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7ff7ac09cf30 msgr2=0x7ff7ac143240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:36:22.327 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:22.321+0000 7ff7ab59e700 1 --2- 172.21.15.88:0/3543222787 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7ff7ac09cf30 0x7ff7ac143240 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7ff7a4009ec0 tx=0x7ff7a400b040 comp rx=0 tx=0).stop 2023-09-30T01:36:22.327 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:22.321+0000 7ff7ab59e700 1 -- 172.21.15.88:0/3543222787 shutdown_connections 2023-09-30T01:36:22.327 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:22.321+0000 7ff7ab59e700 1 --2- 172.21.15.88:0/3543222787 >> v2:172.21.15.107:6812/29926 conn(0x7ff78c033390 0x7ff78c035840 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:36:22.327 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:22.321+0000 7ff7ab59e700 1 --2- 172.21.15.88:0/3543222787 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7ff7ac09cf30 0x7ff7ac143240 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:36:22.327 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:22.321+0000 7ff7ab59e700 1 --2- 172.21.15.88:0/3543222787 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff7ac097a80 0x7ff7ac142d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:36:22.327 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:22.321+0000 7ff7ab59e700 1 --2- 172.21.15.88:0/3543222787 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7ff7ac06f480 0x7ff7ac1427c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:36:22.327 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:22.321+0000 7ff7ab59e700 1 -- 172.21.15.88:0/3543222787 >> 172.21.15.88:0/3543222787 conn(0x7ff7ac0a2540 msgr2=0x7ff7ac097f00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:36:22.327 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:22.321+0000 7ff7ab59e700 1 -- 172.21.15.88:0/3543222787 shutdown_connections 2023-09-30T01:36:22.327 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:22.321+0000 7ff7ab59e700 1 -- 172.21.15.88:0/3543222787 wait complete. 2023-09-30T01:36:25.339 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:36:25.445 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:25.441+0000 7f0c55849700 10 monclient: get_monmap_and_config 2023-09-30T01:36:25.445 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:25.441+0000 7f0c55849700 10 monclient: build_initial_monmap 2023-09-30T01:36:25.445 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:25.441+0000 7f0c55849700 10 monclient: monmap: 2023-09-30T01:36:25.445 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2023-09-30T01:36:25.445 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:36:25.445 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:36:25.445043+0000 2023-09-30T01:36:25.445 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:36:25.445043+0000 2023-09-30T01:36:25.445 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2023-09-30T01:36:25.446 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:36:25.446 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a-legacy 2023-09-30T01:36:25.446 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.107:6789/0 mon.noname-f-legacy 2023-09-30T01:36:25.446 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.noname-b 2023-09-30T01:36:25.446 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.noname-g 2023-09-30T01:36:25.446 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.noname-c 2023-09-30T01:36:25.446 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.noname-h 2023-09-30T01:36:25.446 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.noname-d 2023-09-30T01:36:25.446 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.noname-i 2023-09-30T01:36:25.446 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.noname-e 2023-09-30T01:36:25.446 INFO:teuthology.orchestra.run.smithi088.stderr:9: v2:172.21.15.88:3300/0 mon.noname-a 2023-09-30T01:36:25.446 INFO:teuthology.orchestra.run.smithi088.stderr:10: v2:172.21.15.107:3300/0 mon.noname-f 2023-09-30T01:36:25.447 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:36:25.457 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:25.441+0000 7f0c55849700 1 Processor -- start 2023-09-30T01:36:25.457 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:25.441+0000 7f0c55849700 1 -- start start 2023-09-30T01:36:25.457 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:25.441+0000 7f0c55849700 10 monclient: init 2023-09-30T01:36:25.457 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:25.441+0000 7f0c55849700 10 monclient: _reopen_session rank -1 2023-09-30T01:36:25.457 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:25.441+0000 7f0c55849700 10 monclient: _add_conns ranks=[3,7,0,6,10,8,4,2,9,1,5] 2023-09-30T01:36:25.457 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:25.441+0000 7f0c55849700 1 --2- >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7f0c500976f0 0x7f0c50098de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:36:25.457 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:25.441+0000 7f0c55849700 10 monclient(hunting): picked mon.noname-g con 0x7f0c500976f0 addr [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] 2023-09-30T01:36:25.457 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:25.441+0000 7f0c55849700 1 --2- >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7f0c5009ed80 0x7f0c5009f150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:36:25.458 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:25.441+0000 7f0c55849700 10 monclient(hunting): picked mon.noname-i con 0x7f0c5009ed80 addr [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] 2023-09-30T01:36:25.458 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:25.441+0000 7f0c55849700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f0c5009f690 addr v1:172.21.15.88:6789/0 2023-09-30T01:36:25.458 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:25.441+0000 7f0c55849700 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c5004bb20 con 0x7f0c5009f690 2023-09-30T01:36:25.458 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:25.441+0000 7f0c55849700 10 monclient(hunting): start opening mon connection 2023-09-30T01:36:25.458 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:25.441+0000 7f0c55849700 1 -- --> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c50099320 con 0x7f0c500976f0 2023-09-30T01:36:25.458 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:25.441+0000 7f0c55849700 10 monclient(hunting): start opening mon connection 2023-09-30T01:36:25.458 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:25.441+0000 7f0c55849700 1 -- --> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] -- mon_getmap magic: 0 v1 -- 0x7f0c50098400 con 0x7f0c5009ed80 2023-09-30T01:36:25.458 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:25.441+0000 7f0c55849700 10 monclient(hunting): _renew_subs 2023-09-30T01:36:25.458 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:25.441+0000 7f0c55849700 10 monclient(hunting): authenticate will time out at 1057.482987s 2023-09-30T01:36:25.458 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:25.441+0000 7f0c55048700 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f0c5009f690 0x7f0c50130570 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:34730/0 (socket says 172.21.15.88:34730) 2023-09-30T01:36:25.458 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:25.441+0000 7f0c55048700 1 -- 172.21.15.88:0/524572716 learned_addr learned my addr 172.21.15.88:0/524572716 (peer_addr_for_me v1:172.21.15.88:0/0) 2023-09-30T01:36:25.459 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:25.441+0000 7f0c4ffff700 1 --2- 172.21.15.88:0/524572716 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7f0c5009ed80 0x7f0c5009f150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:36:25.459 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:25.441+0000 7f0c54847700 1 --2- 172.21.15.88:0/524572716 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7f0c500976f0 0x7f0c50098de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:36:25.459 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:25.441+0000 7f0c4ffff700 10 monclient(hunting): get_auth_request con 0x7f0c5009ed80 auth_method 0 2023-09-30T01:36:25.459 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:25.441+0000 7f0c4ffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:36:25.459 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:25.441+0000 7f0c4ffff700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:36:25.459 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:25.441+0000 7f0c4ffff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:36:25.459 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:25.441+0000 7f0c54847700 10 monclient(hunting): get_auth_request con 0x7f0c500976f0 auth_method 0 2023-09-30T01:36:25.459 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:25.441+0000 7f0c54847700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:36:25.459 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:25.441+0000 7f0c54847700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:36:25.459 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:25.441+0000 7f0c54847700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:36:25.459 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:25.441+0000 7f0c4f7fe700 1 -- 172.21.15.88:0/524572716 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2532766533 0 0) 0x7f0c5004bb20 con 0x7f0c5009f690 2023-09-30T01:36:25.459 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:25.441+0000 7f0c4f7fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:36:25.460 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:25.441+0000 7f0c4f7fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:36:25.460 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:25.441+0000 7f0c4f7fe700 10 monclient(hunting): my global_id is 4950 2023-09-30T01:36:25.460 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:25.441+0000 7f0c4f7fe700 1 -- 172.21.15.88:0/524572716 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0c40003570 con 0x7f0c5009f690 2023-09-30T01:36:25.460 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:25.441+0000 7f0c4ffff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:36:25.460 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:25.441+0000 7f0c4ffff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:36:25.460 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:25.441+0000 7f0c4ffff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:36:25.460 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:25.441+0000 7f0c54847700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:36:25.460 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:25.441+0000 7f0c54847700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:36:25.460 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:25.441+0000 7f0c54847700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:36:25.460 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:25.445+0000 7f0c4f7fe700 1 -- 172.21.15.88:0/524572716 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2029614892 0 0) 0x7f0c40003570 con 0x7f0c5009f690 2023-09-30T01:36:25.460 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:25.445+0000 7f0c4f7fe700 1 -- 172.21.15.88:0/524572716 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7f0c5009ed80 msgr2=0x7f0c5009f150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:36:25.460 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:25.445+0000 7f0c4f7fe700 1 --2- 172.21.15.88:0/524572716 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7f0c5009ed80 0x7f0c5009f150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:36:25.461 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:25.445+0000 7f0c4f7fe700 1 -- 172.21.15.88:0/524572716 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7f0c500976f0 msgr2=0x7f0c50098de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:36:25.461 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:25.445+0000 7f0c4f7fe700 1 --2- 172.21.15.88:0/524572716 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7f0c500976f0 0x7f0c50098de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:36:25.461 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:25.445+0000 7f0c4f7fe700 10 monclient: _finish_hunting 0 2023-09-30T01:36:25.461 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:25.445+0000 7f0c4f7fe700 1 monclient: found mon.noname-a-legacy 2023-09-30T01:36:25.461 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:25.445+0000 7f0c4f7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:36:25.461 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:25.445+0000 7f0c4f7fe700 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.88:6789/0 2023-09-30T01:36:25.461 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:25.445+0000 7f0c4f7fe700 1 -- 172.21.15.88:0/524572716 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c50098a30 con 0x7f0c5009f690 2023-09-30T01:36:25.461 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:25.445+0000 7f0c4f7fe700 10 monclient: _finish_auth 0 2023-09-30T01:36:25.461 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:25.445+0000 7f0c4f7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:36:25.461 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:25.445+0000 7f0c4f7fe700 1 -- 172.21.15.88:0/524572716 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (unknown 176869241 0 0) 0x7f0c48001c70 con 0x7f0c5009f690 2023-09-30T01:36:25.461 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:25.445+0000 7f0c55849700 5 monclient: authenticate success, global_id 4950 2023-09-30T01:36:25.461 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:25.445+0000 7f0c55849700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:36:25.462 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:25.445+0000 7f0c4f7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:36:25.462 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:25.445+0000 7f0c4f7fe700 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2023-09-30T01:36:25.462 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:25.445+0000 7f0c4f7fe700 10 monclient: dump: 2023-09-30T01:36:25.462 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:36:25.462 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:36:25.462 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:36:25.462 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:36:25.462 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:36:25.462 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:36:25.462 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:36:25.462 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:36:25.462 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:36:25.463 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:36:25.463 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:36:25.463 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:36:25.463 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:36:25.463 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:36:25.463 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:36:25.463 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:36:25.463 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:25.445+0000 7f0c4f7fe700 1 monclient: mon.a has (v2) addrs [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] but i'm connected to v1:172.21.15.88:6789/0, reconnecting 2023-09-30T01:36:25.463 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:25.445+0000 7f0c4f7fe700 10 monclient: _reopen_session rank -1 2023-09-30T01:36:25.463 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:25.445+0000 7f0c4f7fe700 1 -- 172.21.15.88:0/524572716 >> v1:172.21.15.88:6789/0 conn(0x7f0c5009f690 legacy=0x7f0c50130570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:36:25.463 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:25.445+0000 7f0c4f7fe700 10 monclient: _add_conns ranks=[0,8,4,7,5,6,2,1,3] 2023-09-30T01:36:25.463 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:25.445+0000 7f0c4f7fe700 1 --2- 172.21.15.88:0/524572716 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0c400042b0 0x7f0c40004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:36:25.464 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:25.445+0000 7f0c4f7fe700 10 monclient(hunting): picked mon.a con 0x7f0c400042b0 addr [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] 2023-09-30T01:36:25.464 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:25.445+0000 7f0c4f7fe700 1 --2- 172.21.15.88:0/524572716 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f0c40004da0 0x7f0c40005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:36:25.464 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:25.445+0000 7f0c4f7fe700 10 monclient(hunting): picked mon.e con 0x7f0c40004da0 addr [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] 2023-09-30T01:36:25.464 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:25.445+0000 7f0c4f7fe700 1 --2- 172.21.15.88:0/524572716 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f0c40009aa0 0x7f0c4000bf00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:36:25.464 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:25.445+0000 7f0c4f7fe700 10 monclient(hunting): picked mon.c con 0x7f0c40009aa0 addr [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] 2023-09-30T01:36:25.464 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:25.445+0000 7f0c4f7fe700 10 monclient(hunting): start opening mon connection 2023-09-30T01:36:25.464 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:25.445+0000 7f0c4f7fe700 1 -- 172.21.15.88:0/524572716 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c50099320 con 0x7f0c400042b0 2023-09-30T01:36:25.464 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:25.445+0000 7f0c4f7fe700 10 monclient(hunting): start opening mon connection 2023-09-30T01:36:25.464 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:25.445+0000 7f0c4f7fe700 1 -- 172.21.15.88:0/524572716 --> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] -- mon_getmap magic: 0 v1 -- 0x7f0c50098400 con 0x7f0c40009aa0 2023-09-30T01:36:25.464 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:25.445+0000 7f0c4f7fe700 10 monclient(hunting): start opening mon connection 2023-09-30T01:36:25.464 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:25.445+0000 7f0c4f7fe700 1 -- 172.21.15.88:0/524572716 --> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] -- mon_getmap magic: 0 v1 -- 0x7f0c4000c810 con 0x7f0c40004da0 2023-09-30T01:36:25.464 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:25.445+0000 7f0c4f7fe700 10 monclient(hunting): _renew_subs 2023-09-30T01:36:25.464 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:25.445+0000 7f0c4f7fe700 10 monclient(hunting): _finish_auth 0 2023-09-30T01:36:25.465 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:25.445+0000 7f0c55849700 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2023-09-30T01:36:25.465 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:25.445+0000 7f0c55048700 1 --2- 172.21.15.88:0/524572716 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f0c40009aa0 0x7f0c4000bf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:36:25.465 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:25.445+0000 7f0c4ffff700 1 --2- 172.21.15.88:0/524572716 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f0c40004da0 0x7f0c40005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:36:25.465 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:25.445+0000 7f0c54847700 1 --2- 172.21.15.88:0/524572716 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0c400042b0 0x7f0c40004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:36:25.465 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:25.445+0000 7f0c55048700 10 monclient(hunting): get_auth_request con 0x7f0c40009aa0 auth_method 0 2023-09-30T01:36:25.465 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:25.445+0000 7f0c55048700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:36:25.465 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:25.445+0000 7f0c55048700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:36:25.465 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:25.445+0000 7f0c55048700 10 monclient(hunting): _init_auth already have auth, reseting 2023-09-30T01:36:25.465 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:25.445+0000 7f0c4ffff700 10 monclient(hunting): get_auth_request con 0x7f0c40004da0 auth_method 0 2023-09-30T01:36:25.465 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:25.445+0000 7f0c4ffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:36:25.465 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:25.445+0000 7f0c4ffff700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:36:25.466 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:25.445+0000 7f0c4ffff700 10 monclient(hunting): _init_auth already have auth, reseting 2023-09-30T01:36:25.466 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:25.445+0000 7f0c54847700 10 monclient(hunting): get_auth_request con 0x7f0c400042b0 auth_method 0 2023-09-30T01:36:25.466 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:25.445+0000 7f0c54847700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:36:25.466 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:25.445+0000 7f0c54847700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:36:25.466 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:25.445+0000 7f0c54847700 10 monclient(hunting): _init_auth already have auth, reseting 2023-09-30T01:36:25.466 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:25.445+0000 7f0c55048700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:36:25.466 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:25.445+0000 7f0c55048700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:36:25.466 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:25.445+0000 7f0c55048700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-09-30T01:36:25.466 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:25.445+0000 7f0c4ffff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:36:25.466 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:25.445+0000 7f0c4ffff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:36:25.466 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:25.445+0000 7f0c4ffff700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-09-30T01:36:25.466 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:25.445+0000 7f0c54847700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:36:25.466 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:25.445+0000 7f0c54847700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:36:25.467 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:25.445+0000 7f0c54847700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-09-30T01:36:25.467 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:25.445+0000 7f0c55048700 10 monclient(hunting): handle_auth_done global_id 4950 payload 293 2023-09-30T01:36:25.467 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:25.445+0000 7f0c55048700 1 -- 172.21.15.88:0/524572716 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f0c40004da0 msgr2=0x7f0c40005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:36:25.467 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:25.445+0000 7f0c55048700 1 --2- 172.21.15.88:0/524572716 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f0c40004da0 0x7f0c40005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:36:25.467 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:25.445+0000 7f0c55048700 1 -- 172.21.15.88:0/524572716 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0c400042b0 msgr2=0x7f0c40004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:36:25.467 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:25.445+0000 7f0c55048700 1 --2- 172.21.15.88:0/524572716 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0c400042b0 0x7f0c40004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:36:25.467 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:25.445+0000 7f0c55048700 10 monclient: _finish_hunting 0 2023-09-30T01:36:25.467 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:25.445+0000 7f0c55048700 1 monclient: found mon.c 2023-09-30T01:36:25.467 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:25.445+0000 7f0c55048700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:36:25.467 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:25.445+0000 7f0c55048700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.88:3302/0 2023-09-30T01:36:25.467 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:25.445+0000 7f0c55048700 1 -- 172.21.15.88:0/524572716 --> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c40003570 con 0x7f0c40009aa0 2023-09-30T01:36:25.467 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:25.445+0000 7f0c55048700 10 monclient: _finish_auth 0 2023-09-30T01:36:25.468 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:25.445+0000 7f0c55048700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:36:25.468 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:25.445+0000 7f0c54847700 1 --2- 172.21.15.88:0/524572716 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0c400042b0 0x7f0c40004810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:36:25.468 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:25.445+0000 7f0c55048700 1 -- 172.21.15.88:0/524572716 reap_dead start 2023-09-30T01:36:25.468 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:25.445+0000 7f0c55048700 1 --2- 172.21.15.88:0/524572716 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f0c40009aa0 0x7f0c4000bf00 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f0c4800d860 tx=0x7f0c50097f90 comp rx=0 tx=0).ready entity=mon.4 client_cookie=ba014991ed07c630 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:36:25.468 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:25.445+0000 7f0c4f7fe700 1 -- 172.21.15.88:0/524572716 <== mon.4 v2:172.21.15.88:3302/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f0c4800f9e0 con 0x7f0c40009aa0 2023-09-30T01:36:25.468 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:25.445+0000 7f0c4f7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:36:25.468 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:25.445+0000 7f0c4f7fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T01:36:25.468 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:25.445+0000 7f0c4f7fe700 10 monclient: dump: 2023-09-30T01:36:25.468 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:36:25.468 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:36:25.468 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:36:25.468 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:36:25.469 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:36:25.469 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:36:25.469 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:36:25.469 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:36:25.469 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:36:25.469 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:36:25.469 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:36:25.469 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:36:25.469 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:36:25.469 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:36:25.469 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:36:25.469 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:36:25.469 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:25.445+0000 7f0c55849700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:36:25.470 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:25.445+0000 7f0c4f7fe700 1 -- 172.21.15.88:0/524572716 <== mon.4 v2:172.21.15.88:3302/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c4800fba0 con 0x7f0c40009aa0 2023-09-30T01:36:25.470 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:25.445+0000 7f0c4f7fe700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:36:25.470 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:25.445+0000 7f0c55849700 10 monclient: get_monmap_and_config success 2023-09-30T01:36:25.470 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:25.445+0000 7f0c4f7fe700 1 -- 172.21.15.88:0/524572716 <== mon.4 v2:172.21.15.88:3302/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f0c4800fd10 con 0x7f0c40009aa0 2023-09-30T01:36:25.470 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:25.445+0000 7f0c55849700 4 set_mon_vals no callback set 2023-09-30T01:36:25.470 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:25.445+0000 7f0c4f7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:36:25.470 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:25.445+0000 7f0c4f7fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T01:36:25.470 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:25.445+0000 7f0c4f7fe700 10 monclient: dump: 2023-09-30T01:36:25.470 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:36:25.470 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:36:25.470 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:36:25.470 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:36:25.471 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:36:25.471 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:36:25.471 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:36:25.471 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:36:25.471 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:36:25.471 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:36:25.471 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:36:25.471 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:36:25.471 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:36:25.471 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:36:25.471 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:36:25.471 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:36:25.472 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:25.445+0000 7f0c55849700 10 monclient: shutdown 2023-09-30T01:36:25.472 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:25.445+0000 7f0c55849700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:36:25.472 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:25.445+0000 7f0c55849700 1 -- 172.21.15.88:0/524572716 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f0c40009aa0 msgr2=0x7f0c4000bf00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:36:25.472 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:25.445+0000 7f0c55849700 1 --2- 172.21.15.88:0/524572716 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f0c40009aa0 0x7f0c4000bf00 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f0c4800d860 tx=0x7f0c50097f90 comp rx=0 tx=0).stop 2023-09-30T01:36:25.472 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:25.445+0000 7f0c55849700 1 -- 172.21.15.88:0/524572716 shutdown_connections 2023-09-30T01:36:25.472 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:25.445+0000 7f0c55849700 1 --2- 172.21.15.88:0/524572716 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f0c40009aa0 0x7f0c4000bf00 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:36:25.472 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:25.445+0000 7f0c55849700 1 -- 172.21.15.88:0/524572716 >> 172.21.15.88:0/524572716 conn(0x7f0c500a0910 msgr2=0x7f0c500a33c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:36:25.472 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:25.445+0000 7f0c55849700 1 -- 172.21.15.88:0/524572716 shutdown_connections 2023-09-30T01:36:25.472 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:25.445+0000 7f0c55849700 1 -- 172.21.15.88:0/524572716 wait complete. 2023-09-30T01:36:25.472 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:25.445+0000 7f0c55849700 10 monclient: build_initial_monmap 2023-09-30T01:36:25.472 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:25.445+0000 7f0c55849700 10 monclient: monmap: 2023-09-30T01:36:25.472 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2023-09-30T01:36:25.473 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:36:25.473 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 0.000000 2023-09-30T01:36:25.473 INFO:teuthology.orchestra.run.smithi088.stderr:created 0.000000 2023-09-30T01:36:25.473 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2023-09-30T01:36:25.473 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 1 2023-09-30T01:36:25.473 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.noname-a 2023-09-30T01:36:25.473 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.noname-f 2023-09-30T01:36:25.473 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.noname-b 2023-09-30T01:36:25.473 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.noname-g 2023-09-30T01:36:25.473 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.noname-c 2023-09-30T01:36:25.473 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.noname-h 2023-09-30T01:36:25.473 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.noname-d 2023-09-30T01:36:25.474 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.noname-i 2023-09-30T01:36:25.474 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.noname-e 2023-09-30T01:36:25.474 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:36:25.474 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:25.445+0000 7f0c55849700 1 Processor -- start 2023-09-30T01:36:25.474 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:25.445+0000 7f0c55849700 1 -- start start 2023-09-30T01:36:25.474 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:25.445+0000 7f0c55849700 10 monclient: init 2023-09-30T01:36:25.474 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:25.449+0000 7f0c55849700 10 monclient: _reopen_session rank -1 2023-09-30T01:36:25.474 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:25.449+0000 7f0c55849700 10 monclient: _add_conns ranks=[4,2,5,8,3,6,7,0,1] 2023-09-30T01:36:25.474 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:25.449+0000 7f0c55849700 1 --2- >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f0c40009aa0 0x7f0c5012ac50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:36:25.474 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:25.449+0000 7f0c55849700 10 monclient(hunting): picked mon.noname-c con 0x7f0c40009aa0 addr [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] 2023-09-30T01:36:25.474 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:25.449+0000 7f0c55849700 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0c50096330 0x7f0c50096700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:36:25.474 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:25.449+0000 7f0c55849700 10 monclient(hunting): picked mon.noname-b con 0x7f0c50096330 addr [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] 2023-09-30T01:36:25.474 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:25.449+0000 7f0c55849700 1 --2- >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7f0c50096c40 0x7f0c5006d580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:36:25.475 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:25.449+0000 7f0c55849700 10 monclient(hunting): picked mon.noname-h con 0x7f0c50096c40 addr [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] 2023-09-30T01:36:25.475 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:25.449+0000 7f0c55849700 10 monclient(hunting): start opening mon connection 2023-09-30T01:36:25.475 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:25.449+0000 7f0c55849700 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c5012a490 con 0x7f0c50096330 2023-09-30T01:36:25.475 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:25.449+0000 7f0c55849700 10 monclient(hunting): start opening mon connection 2023-09-30T01:36:25.475 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:25.449+0000 7f0c55849700 1 -- --> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] -- mon_getmap magic: 0 v1 -- 0x7f0c5012a140 con 0x7f0c40009aa0 2023-09-30T01:36:25.475 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:25.449+0000 7f0c55849700 10 monclient(hunting): start opening mon connection 2023-09-30T01:36:25.475 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:25.449+0000 7f0c55849700 1 -- --> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] -- mon_getmap magic: 0 v1 -- 0x7f0c5006f0d0 con 0x7f0c50096c40 2023-09-30T01:36:25.475 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:25.449+0000 7f0c55849700 10 monclient(hunting): _renew_subs 2023-09-30T01:36:25.475 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:25.449+0000 7f0c55849700 10 monclient(hunting): authenticate will time out at 1057.488429s 2023-09-30T01:36:25.475 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:25.449+0000 7f0c54847700 1 --2- >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f0c40009aa0 0x7f0c5012ac50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:36:25.475 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:25.449+0000 7f0c4ffff700 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0c50096330 0x7f0c50096700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:36:25.475 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:25.449+0000 7f0c54847700 1 --2- >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f0c40009aa0 0x7f0c5012ac50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3302/0 says I am v2:172.21.15.88:53932/0 (socket says 172.21.15.88:53932) 2023-09-30T01:36:25.476 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:25.449+0000 7f0c54847700 1 -- 172.21.15.88:0/3289623775 learned_addr learned my addr 172.21.15.88:0/3289623775 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-09-30T01:36:25.476 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:25.449+0000 7f0c54847700 10 monclient(hunting): get_auth_request con 0x7f0c40009aa0 auth_method 0 2023-09-30T01:36:25.476 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:25.449+0000 7f0c54847700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:36:25.476 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:25.449+0000 7f0c54847700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:36:25.476 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:25.449+0000 7f0c54847700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:36:25.476 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:25.449+0000 7f0c4ffff700 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0c50096330 0x7f0c50096700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3301/0 says I am v2:172.21.15.88:57122/0 (socket says 172.21.15.88:57122) 2023-09-30T01:36:25.476 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:25.449+0000 7f0c4ffff700 10 monclient(hunting): get_auth_request con 0x7f0c50096330 auth_method 0 2023-09-30T01:36:25.476 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:25.449+0000 7f0c4ffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:36:25.476 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:25.449+0000 7f0c4ffff700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:36:25.476 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:25.449+0000 7f0c4ffff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:36:25.476 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:25.449+0000 7f0c4ffff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:36:25.477 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:25.449+0000 7f0c4ffff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:36:25.477 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:25.449+0000 7f0c55048700 1 --2- 172.21.15.88:0/3289623775 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7f0c50096c40 0x7f0c5006d580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:36:25.477 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:25.449+0000 7f0c4ffff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:36:25.477 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:25.449+0000 7f0c54847700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:36:25.477 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:25.449+0000 7f0c54847700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:36:25.477 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:25.449+0000 7f0c54847700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:36:25.477 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:25.449+0000 7f0c55048700 10 monclient(hunting): get_auth_request con 0x7f0c50096c40 auth_method 0 2023-09-30T01:36:25.477 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:25.449+0000 7f0c55048700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:36:25.477 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:25.449+0000 7f0c55048700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:36:25.477 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:25.449+0000 7f0c55048700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:36:25.477 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:25.449+0000 7f0c55048700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:36:25.477 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:25.449+0000 7f0c55048700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:36:25.478 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:25.449+0000 7f0c55048700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:36:25.478 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:25.449+0000 7f0c4ffff700 10 monclient(hunting): handle_auth_done global_id 4664 payload 824 2023-09-30T01:36:25.478 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:25.449+0000 7f0c4ffff700 1 -- 172.21.15.88:0/3289623775 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7f0c50096c40 msgr2=0x7f0c5006d580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:36:25.478 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:25.449+0000 7f0c4ffff700 1 --2- 172.21.15.88:0/3289623775 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7f0c50096c40 0x7f0c5006d580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:36:25.478 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:25.449+0000 7f0c4ffff700 1 -- 172.21.15.88:0/3289623775 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f0c40009aa0 msgr2=0x7f0c5012ac50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:36:25.478 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:25.449+0000 7f0c4ffff700 1 --2- 172.21.15.88:0/3289623775 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f0c40009aa0 0x7f0c5012ac50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:36:25.478 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:25.449+0000 7f0c4ffff700 10 monclient: _finish_hunting 0 2023-09-30T01:36:25.478 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:25.449+0000 7f0c4ffff700 1 monclient: found mon.noname-b 2023-09-30T01:36:25.478 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:25.449+0000 7f0c4ffff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:36:25.478 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:25.449+0000 7f0c4ffff700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.88:3301/0 2023-09-30T01:36:25.478 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:25.449+0000 7f0c4ffff700 1 -- 172.21.15.88:0/3289623775 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c5012b190 con 0x7f0c50096330 2023-09-30T01:36:25.478 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:25.449+0000 7f0c4ffff700 1 --2- 172.21.15.88:0/3289623775 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0c50096330 0x7f0c50096700 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f0c3c009c10 tx=0x7f0c3c012040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:36:25.479 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:25.449+0000 7f0c4cff9700 1 -- 172.21.15.88:0/3289623775 <== mon.2 v2:172.21.15.88:3301/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f0c3c015d90 con 0x7f0c50096330 2023-09-30T01:36:25.479 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:25.449+0000 7f0c4cff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:36:25.479 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:25.449+0000 7f0c4cff9700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T01:36:25.479 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:25.449+0000 7f0c4cff9700 10 monclient: dump: 2023-09-30T01:36:25.479 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:36:25.479 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:36:25.479 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:36:25.479 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:36:25.479 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:36:25.479 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:36:25.479 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:36:25.479 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:36:25.480 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:36:25.480 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:36:25.480 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:36:25.480 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:36:25.480 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:36:25.480 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:36:25.480 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:36:25.480 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:36:25.480 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:25.449+0000 7f0c4cff9700 10 monclient: _finish_auth 0 2023-09-30T01:36:25.480 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:25.449+0000 7f0c4cff9700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:36:25.480 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:25.449+0000 7f0c55849700 5 monclient: authenticate success, global_id 4664 2023-09-30T01:36:25.480 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:25.449+0000 7f0c4cff9700 1 -- 172.21.15.88:0/3289623775 <== mon.2 v2:172.21.15.88:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c3c016040 con 0x7f0c50096330 2023-09-30T01:36:25.481 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:25.449+0000 7f0c55849700 10 monclient: _renew_subs 2023-09-30T01:36:25.481 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:25.449+0000 7f0c55849700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.88:3301/0 2023-09-30T01:36:25.481 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:25.449+0000 7f0c55849700 1 -- 172.21.15.88:0/3289623775 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c5012b380 con 0x7f0c50096330 2023-09-30T01:36:25.481 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:25.449+0000 7f0c4cff9700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:36:25.481 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:25.449+0000 7f0c4cff9700 1 -- 172.21.15.88:0/3289623775 <== mon.2 v2:172.21.15.88:3301/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f0c3c01b900 con 0x7f0c50096330 2023-09-30T01:36:25.481 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:25.449+0000 7f0c4cff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:36:25.481 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:25.449+0000 7f0c4cff9700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T01:36:25.481 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:25.449+0000 7f0c4cff9700 10 monclient: dump: 2023-09-30T01:36:25.481 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:36:25.481 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:36:25.481 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:36:25.481 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:36:25.482 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:36:25.482 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:36:25.482 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:36:25.482 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:36:25.482 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:36:25.482 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:36:25.482 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:36:25.482 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:36:25.482 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:36:25.482 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:36:25.482 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:36:25.482 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:36:25.483 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:25.449+0000 7f0c4effd700 4 set_mon_vals no callback set 2023-09-30T01:36:25.483 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:25.449+0000 7f0c55849700 10 monclient: _renew_subs 2023-09-30T01:36:25.483 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:25.449+0000 7f0c55849700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.88:3301/0 2023-09-30T01:36:25.483 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:25.449+0000 7f0c55849700 1 -- 172.21.15.88:0/3289623775 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c5006dc60 con 0x7f0c50096330 2023-09-30T01:36:25.483 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:25.449+0000 7f0c55849700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:36:25.483 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:25.449+0000 7f0c55849700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:36:25.483 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:25.449+0000 7f0c55849700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.88:3301/0 2023-09-30T01:36:25.483 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:25.449+0000 7f0c55849700 1 -- 172.21.15.88:0/3289623775 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0c50071960 con 0x7f0c50096330 2023-09-30T01:36:25.483 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:25.449+0000 7f0c4cff9700 1 -- 172.21.15.88:0/3289623775 <== mon.2 v2:172.21.15.88:3301/0 4 ==== mgrmap(e 5) v1 ==== 40144+0+0 (secure 0 0 0) 0x7f0c3c01b480 con 0x7f0c50096330 2023-09-30T01:36:25.483 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:25.449+0000 7f0c4cff9700 1 --2- 172.21.15.88:0/3289623775 >> v2:172.21.15.107:6812/29926 conn(0x7f0c34033360 0x7f0c34035810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:36:25.483 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:25.449+0000 7f0c4cff9700 1 -- 172.21.15.88:0/3289623775 <== mon.2 v2:172.21.15.88:3301/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 3502+0+0 (secure 0 0 0) 0x7f0c3c047a30 con 0x7f0c50096330 2023-09-30T01:36:25.483 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:25.449+0000 7f0c54847700 1 --2- 172.21.15.88:0/3289623775 >> v2:172.21.15.107:6812/29926 conn(0x7f0c34033360 0x7f0c34035810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:36:25.483 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:25.449+0000 7f0c54847700 10 monclient: get_auth_request con 0x7f0c34033360 auth_method 0 2023-09-30T01:36:25.484 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:25.453+0000 7f0c54847700 1 --2- 172.21.15.88:0/3289623775 >> v2:172.21.15.107:6812/29926 conn(0x7f0c34033360 0x7f0c34035810 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f0c44006e50 tx=0x7f0c44009120 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:36:25.484 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:25.457+0000 7f0c4cff9700 1 -- 172.21.15.88:0/3289623775 <== mon.2 v2:172.21.15.88:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f0c3c01ba60 con 0x7f0c50096330 2023-09-30T01:36:25.484 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:25.457+0000 7f0c4cff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:36:25.484 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:25.457+0000 7f0c4cff9700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:36:25.660 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:25.657+0000 7f0c55849700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T01:36:25.660 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:25.657+0000 7f0c55849700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:36:25.660 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:25.657+0000 7f0c55849700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.88:3301/0 2023-09-30T01:36:25.660 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:25.657+0000 7f0c55849700 1 -- 172.21.15.88:0/3289623775 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f0c50129f20 con 0x7f0c50096330 2023-09-30T01:36:25.661 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:25.657+0000 7f0c4cff9700 1 -- 172.21.15.88:0/3289623775 <== mon.2 v2:172.21.15.88:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f0c3c019340 con 0x7f0c50096330 2023-09-30T01:36:25.661 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:25.657+0000 7f0c4cff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:36:25.661 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:25.657+0000 7f0c4cff9700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:36:25.661 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-09-30T01:36:25.661 INFO:teuthology.orchestra.run.smithi088.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:36:25.663 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:25.657+0000 7f0c55849700 1 -- 172.21.15.88:0/3289623775 >> v2:172.21.15.107:6812/29926 conn(0x7f0c34033360 msgr2=0x7f0c34035810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:36:25.663 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:25.657+0000 7f0c55849700 1 --2- 172.21.15.88:0/3289623775 >> v2:172.21.15.107:6812/29926 conn(0x7f0c34033360 0x7f0c34035810 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f0c44006e50 tx=0x7f0c44009120 comp rx=0 tx=0).stop 2023-09-30T01:36:25.663 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:25.657+0000 7f0c55849700 10 monclient: shutdown 2023-09-30T01:36:25.663 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:25.657+0000 7f0c55849700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:36:25.663 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:25.657+0000 7f0c55849700 1 -- 172.21.15.88:0/3289623775 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0c50096330 msgr2=0x7f0c50096700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:36:25.663 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:25.657+0000 7f0c55849700 1 --2- 172.21.15.88:0/3289623775 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0c50096330 0x7f0c50096700 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f0c3c009c10 tx=0x7f0c3c012040 comp rx=0 tx=0).stop 2023-09-30T01:36:25.663 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:25.657+0000 7f0c55849700 1 -- 172.21.15.88:0/3289623775 shutdown_connections 2023-09-30T01:36:25.663 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:25.657+0000 7f0c55849700 1 --2- 172.21.15.88:0/3289623775 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7f0c50096c40 0x7f0c5006d580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:36:25.664 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:25.657+0000 7f0c55849700 1 --2- 172.21.15.88:0/3289623775 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0c50096330 0x7f0c50096700 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:36:25.664 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:25.657+0000 7f0c55849700 1 --2- 172.21.15.88:0/3289623775 >> v2:172.21.15.107:6812/29926 conn(0x7f0c34033360 0x7f0c34035810 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:36:25.664 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:25.657+0000 7f0c55849700 1 --2- 172.21.15.88:0/3289623775 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f0c40009aa0 0x7f0c5012ac50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:36:25.664 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:25.657+0000 7f0c55849700 1 -- 172.21.15.88:0/3289623775 >> 172.21.15.88:0/3289623775 conn(0x7f0c500a0910 msgr2=0x7f0c50131760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:36:25.664 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:25.657+0000 7f0c55849700 1 -- 172.21.15.88:0/3289623775 shutdown_connections 2023-09-30T01:36:25.664 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:25.657+0000 7f0c55849700 1 -- 172.21.15.88:0/3289623775 wait complete. 2023-09-30T01:36:28.676 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:36:28.786 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:28.781+0000 7f6848afd700 10 monclient: get_monmap_and_config 2023-09-30T01:36:28.786 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:28.781+0000 7f6848afd700 10 monclient: build_initial_monmap 2023-09-30T01:36:28.786 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:28.781+0000 7f6848afd700 10 monclient: monmap: 2023-09-30T01:36:28.786 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2023-09-30T01:36:28.786 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:36:28.786 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:36:28.785832+0000 2023-09-30T01:36:28.786 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:36:28.785832+0000 2023-09-30T01:36:28.786 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2023-09-30T01:36:28.786 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:36:28.787 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a-legacy 2023-09-30T01:36:28.787 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.107:6789/0 mon.noname-f-legacy 2023-09-30T01:36:28.787 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.noname-b 2023-09-30T01:36:28.787 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.noname-g 2023-09-30T01:36:28.787 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.noname-c 2023-09-30T01:36:28.787 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.noname-h 2023-09-30T01:36:28.787 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.noname-d 2023-09-30T01:36:28.787 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.noname-i 2023-09-30T01:36:28.787 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.noname-e 2023-09-30T01:36:28.787 INFO:teuthology.orchestra.run.smithi088.stderr:9: v2:172.21.15.88:3300/0 mon.noname-a 2023-09-30T01:36:28.787 INFO:teuthology.orchestra.run.smithi088.stderr:10: v2:172.21.15.107:3300/0 mon.noname-f 2023-09-30T01:36:28.787 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:36:28.790 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:28.781+0000 7f6848afd700 1 Processor -- start 2023-09-30T01:36:28.790 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:28.781+0000 7f6848afd700 1 -- start start 2023-09-30T01:36:28.790 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:28.781+0000 7f6848afd700 10 monclient: init 2023-09-30T01:36:28.790 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:28.785+0000 7f6848afd700 10 monclient: _reopen_session rank -1 2023-09-30T01:36:28.790 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:28.785+0000 7f6848afd700 10 monclient: _add_conns ranks=[1,5,4,10,3,2,8,7,9,0,6] 2023-09-30T01:36:28.790 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:28.785+0000 7f6848afd700 10 monclient(hunting): picked mon.noname-f-legacy con 0x7f68440966a0 addr v1:172.21.15.107:6789/0 2023-09-30T01:36:28.790 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:28.785+0000 7f6848afd700 1 --2- >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7f6844097e40 0x7f6844138650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:36:28.791 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:28.785+0000 7f6848afd700 10 monclient(hunting): picked mon.noname-h con 0x7f6844097e40 addr [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] 2023-09-30T01:36:28.791 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:28.785+0000 7f6848afd700 1 --2- >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f6844098210 0x7f684413aba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:36:28.791 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:28.785+0000 7f6848afd700 10 monclient(hunting): picked mon.noname-c con 0x7f6844098210 addr [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] 2023-09-30T01:36:28.791 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:28.785+0000 7f6848afd700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f684405ecb0 con 0x7f68440966a0 2023-09-30T01:36:28.791 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:28.785+0000 7f6848afd700 10 monclient(hunting): start opening mon connection 2023-09-30T01:36:28.791 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:28.785+0000 7f6848afd700 1 -- --> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] -- mon_getmap magic: 0 v1 -- 0x7f6844097180 con 0x7f6844098210 2023-09-30T01:36:28.791 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:28.785+0000 7f6848afd700 10 monclient(hunting): start opening mon connection 2023-09-30T01:36:28.791 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:28.785+0000 7f6848afd700 1 -- --> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] -- mon_getmap magic: 0 v1 -- 0x7f6844097740 con 0x7f6844097e40 2023-09-30T01:36:28.791 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:28.785+0000 7f6848afd700 10 monclient(hunting): _renew_subs 2023-09-30T01:36:28.791 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:28.785+0000 7f6848afd700 10 monclient(hunting): authenticate will time out at 1060.824419s 2023-09-30T01:36:28.791 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:28.785+0000 7f6842d9d700 1 --2- >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f6844098210 0x7f684413aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:36:28.791 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:28.785+0000 7f6842d9d700 1 --2- >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f6844098210 0x7f684413aba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3302/0 says I am v2:172.21.15.88:53944/0 (socket says 172.21.15.88:53944) 2023-09-30T01:36:28.792 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:28.785+0000 7f6842d9d700 1 -- 172.21.15.88:0/3609744771 learned_addr learned my addr 172.21.15.88:0/3609744771 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-09-30T01:36:28.792 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:28.785+0000 7f6842d9d700 10 monclient(hunting): get_auth_request con 0x7f6844098210 auth_method 0 2023-09-30T01:36:28.792 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:28.785+0000 7f6842d9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:36:28.792 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:28.785+0000 7f6842d9d700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:36:28.792 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:28.785+0000 7f6842d9d700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:36:28.792 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:28.785+0000 7f6841d9b700 1 --2- 172.21.15.88:0/3609744771 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7f6844097e40 0x7f6844138650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:36:28.792 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:28.785+0000 7f6841d9b700 10 monclient(hunting): get_auth_request con 0x7f6844097e40 auth_method 0 2023-09-30T01:36:28.792 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:28.785+0000 7f6841d9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:36:28.792 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:28.785+0000 7f6841d9b700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:36:28.792 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:28.785+0000 7f6841d9b700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:36:28.792 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:28.785+0000 7f6842d9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:36:28.792 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:28.785+0000 7f6842d9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:36:28.793 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:28.785+0000 7f6842d9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:36:28.793 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:28.785+0000 7f6841d9b700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:36:28.793 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:28.785+0000 7f6841d9b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:36:28.793 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:28.785+0000 7f6841d9b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:36:28.793 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:28.785+0000 7f6842d9d700 10 monclient(hunting): handle_auth_done global_id 4702 payload 274 2023-09-30T01:36:28.793 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:28.785+0000 7f6842d9d700 1 -- 172.21.15.88:0/3609744771 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7f6844097e40 msgr2=0x7f6844138650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:36:28.793 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:28.785+0000 7f6842d9d700 1 --2- 172.21.15.88:0/3609744771 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7f6844097e40 0x7f6844138650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:36:28.793 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:28.785+0000 7f6842d9d700 1 -- 172.21.15.88:0/3609744771 >> v1:172.21.15.107:6789/0 conn(0x7f68440966a0 legacy=0x7f6844096a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:36:28.793 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:28.785+0000 7f6842d9d700 10 monclient: _finish_hunting 0 2023-09-30T01:36:28.793 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:28.785+0000 7f6842d9d700 1 monclient: found mon.noname-c 2023-09-30T01:36:28.793 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:28.785+0000 7f6842d9d700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:36:28.793 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:28.785+0000 7f6842d9d700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.88:3302/0 2023-09-30T01:36:28.793 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:28.785+0000 7f6842d9d700 1 -- 172.21.15.88:0/3609744771 --> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6844063510 con 0x7f6844098210 2023-09-30T01:36:28.794 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:28.785+0000 7f6842d9d700 1 --2- 172.21.15.88:0/3609744771 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f6844098210 0x7f684413aba0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f683c009f50 tx=0x7f683c009840 comp rx=0 tx=0).ready entity=mon.4 client_cookie=13f62d4e3e589a52 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:36:28.794 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:28.785+0000 7f684159a700 1 -- 172.21.15.88:0/3609744771 <== mon.4 v2:172.21.15.88:3302/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f683c00e070 con 0x7f6844098210 2023-09-30T01:36:28.794 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:28.785+0000 7f684159a700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:36:28.794 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:28.785+0000 7f684159a700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T01:36:28.794 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:28.785+0000 7f684159a700 10 monclient: dump: 2023-09-30T01:36:28.794 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:36:28.794 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:36:28.794 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:36:28.794 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:36:28.794 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:36:28.794 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:36:28.794 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:36:28.795 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:36:28.795 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:36:28.795 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:36:28.795 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:36:28.795 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:36:28.795 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:36:28.795 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:36:28.795 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:36:28.795 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:36:28.795 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:28.785+0000 7f684159a700 10 monclient: _finish_auth 0 2023-09-30T01:36:28.795 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:28.785+0000 7f684159a700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:36:28.795 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:28.785+0000 7f684159a700 1 -- 172.21.15.88:0/3609744771 <== mon.4 v2:172.21.15.88:3302/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f683c00be80 con 0x7f6844098210 2023-09-30T01:36:28.795 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:28.785+0000 7f6848afd700 5 monclient: authenticate success, global_id 4702 2023-09-30T01:36:28.796 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:28.785+0000 7f6848afd700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:36:28.796 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:28.785+0000 7f684159a700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:36:28.796 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:28.785+0000 7f6848afd700 10 monclient: get_monmap_and_config success 2023-09-30T01:36:28.796 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:28.785+0000 7f684159a700 1 -- 172.21.15.88:0/3609744771 <== mon.4 v2:172.21.15.88:3302/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f683c0092e0 con 0x7f6844098210 2023-09-30T01:36:28.796 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:28.785+0000 7f684159a700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:36:28.796 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:28.785+0000 7f6848afd700 4 set_mon_vals no callback set 2023-09-30T01:36:28.796 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:28.785+0000 7f684159a700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T01:36:28.796 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:28.785+0000 7f684159a700 10 monclient: dump: 2023-09-30T01:36:28.796 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:36:28.796 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:36:28.796 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:36:28.796 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:36:28.796 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:36:28.797 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:36:28.797 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:36:28.797 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:36:28.797 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:36:28.797 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:36:28.797 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:36:28.797 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:36:28.797 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:36:28.797 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:36:28.797 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:36:28.797 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:36:28.802 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:28.785+0000 7f6848afd700 10 monclient: shutdown 2023-09-30T01:36:28.802 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:28.785+0000 7f6848afd700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:36:28.802 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:28.785+0000 7f6848afd700 1 -- 172.21.15.88:0/3609744771 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f6844098210 msgr2=0x7f684413aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:36:28.802 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:28.785+0000 7f6848afd700 1 --2- 172.21.15.88:0/3609744771 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f6844098210 0x7f684413aba0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f683c009f50 tx=0x7f683c009840 comp rx=0 tx=0).stop 2023-09-30T01:36:28.802 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:28.785+0000 7f6848afd700 1 -- 172.21.15.88:0/3609744771 shutdown_connections 2023-09-30T01:36:28.803 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:28.785+0000 7f6848afd700 1 --2- 172.21.15.88:0/3609744771 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f6844098210 0x7f684413aba0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:36:28.803 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:28.785+0000 7f6848afd700 1 --2- 172.21.15.88:0/3609744771 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7f6844097e40 0x7f6844138650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:36:28.803 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:28.785+0000 7f6848afd700 1 -- 172.21.15.88:0/3609744771 >> 172.21.15.88:0/3609744771 conn(0x7f68440a3440 msgr2=0x7f68440a3840 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:36:28.803 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:28.785+0000 7f6848afd700 1 -- 172.21.15.88:0/3609744771 shutdown_connections 2023-09-30T01:36:28.803 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:28.785+0000 7f6848afd700 1 -- 172.21.15.88:0/3609744771 wait complete. 2023-09-30T01:36:28.803 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:28.789+0000 7f6848afd700 10 monclient: build_initial_monmap 2023-09-30T01:36:28.803 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:28.789+0000 7f6848afd700 10 monclient: monmap: 2023-09-30T01:36:28.803 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2023-09-30T01:36:28.803 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:36:28.803 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 0.000000 2023-09-30T01:36:28.803 INFO:teuthology.orchestra.run.smithi088.stderr:created 0.000000 2023-09-30T01:36:28.803 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2023-09-30T01:36:28.803 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 1 2023-09-30T01:36:28.804 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.noname-a 2023-09-30T01:36:28.804 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.noname-f 2023-09-30T01:36:28.804 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.noname-b 2023-09-30T01:36:28.804 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.noname-g 2023-09-30T01:36:28.804 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.noname-c 2023-09-30T01:36:28.804 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.noname-h 2023-09-30T01:36:28.804 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.noname-d 2023-09-30T01:36:28.804 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.noname-i 2023-09-30T01:36:28.804 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.noname-e 2023-09-30T01:36:28.804 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:36:28.804 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:28.789+0000 7f6848afd700 1 Processor -- start 2023-09-30T01:36:28.804 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:28.789+0000 7f6848afd700 1 -- start start 2023-09-30T01:36:28.805 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:28.789+0000 7f6848afd700 10 monclient: init 2023-09-30T01:36:28.805 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:28.789+0000 7f6848afd700 10 monclient: _reopen_session rank -1 2023-09-30T01:36:28.805 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:28.789+0000 7f6848afd700 10 monclient: _add_conns ranks=[5,0,7,4,2,3,8,1,6] 2023-09-30T01:36:28.805 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:28.789+0000 7f6848afd700 1 --2- >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7f68440966a0 0x7f68441357e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:36:28.805 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:28.789+0000 7f6848afd700 10 monclient(hunting): picked mon.noname-h con 0x7f68440966a0 addr [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] 2023-09-30T01:36:28.805 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:28.789+0000 7f6848afd700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6844097e40 0x7f6844135d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:36:28.805 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:28.789+0000 7f6848afd700 10 monclient(hunting): picked mon.noname-a con 0x7f6844097e40 addr [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] 2023-09-30T01:36:28.805 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:28.789+0000 7f6848afd700 1 --2- >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7f6844098210 0x7f684408d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:36:28.805 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:28.789+0000 7f6848afd700 10 monclient(hunting): picked mon.noname-i con 0x7f6844098210 addr [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] 2023-09-30T01:36:28.805 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:28.789+0000 7f6848afd700 10 monclient(hunting): start opening mon connection 2023-09-30T01:36:28.805 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:28.789+0000 7f6848afd700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6844099830 con 0x7f6844097e40 2023-09-30T01:36:28.805 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:28.789+0000 7f6848afd700 10 monclient(hunting): start opening mon connection 2023-09-30T01:36:28.806 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:28.789+0000 7f6848afd700 1 -- --> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] -- mon_getmap magic: 0 v1 -- 0x7f684409f780 con 0x7f68440966a0 2023-09-30T01:36:28.806 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:28.789+0000 7f6848afd700 10 monclient(hunting): start opening mon connection 2023-09-30T01:36:28.806 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:28.789+0000 7f6848afd700 1 -- --> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] -- mon_getmap magic: 0 v1 -- 0x7f684408f7b0 con 0x7f6844098210 2023-09-30T01:36:28.806 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:28.789+0000 7f6848afd700 10 monclient(hunting): _renew_subs 2023-09-30T01:36:28.806 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:28.789+0000 7f6848afd700 10 monclient(hunting): authenticate will time out at 1060.830912s 2023-09-30T01:36:28.806 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:28.789+0000 7f6841d9b700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6844097e40 0x7f6844135d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:36:28.806 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:28.789+0000 7f6841d9b700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6844097e40 0x7f6844135d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:56082/0 (socket says 172.21.15.88:56082) 2023-09-30T01:36:28.806 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:28.789+0000 7f6841d9b700 1 -- 172.21.15.88:0/953599096 learned_addr learned my addr 172.21.15.88:0/953599096 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-09-30T01:36:28.806 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:28.789+0000 7f6841d9b700 10 monclient(hunting): get_auth_request con 0x7f6844097e40 auth_method 0 2023-09-30T01:36:28.806 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:28.789+0000 7f6841d9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:36:28.806 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:28.789+0000 7f6841d9b700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:36:28.806 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:28.789+0000 7f6841d9b700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:36:28.806 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:28.789+0000 7f684259c700 1 --2- 172.21.15.88:0/953599096 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7f68440966a0 0x7f68441357e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:36:28.807 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:28.789+0000 7f6842d9d700 1 --2- 172.21.15.88:0/953599096 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7f6844098210 0x7f684408d3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:36:28.807 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:28.789+0000 7f6841d9b700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:36:28.807 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:28.789+0000 7f6841d9b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:36:28.807 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:28.789+0000 7f6841d9b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:36:28.807 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:28.789+0000 7f684259c700 10 monclient(hunting): get_auth_request con 0x7f68440966a0 auth_method 0 2023-09-30T01:36:28.807 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:28.789+0000 7f684259c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:36:28.807 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:28.789+0000 7f684259c700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:36:28.807 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:28.789+0000 7f684259c700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:36:28.807 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:28.789+0000 7f6842d9d700 10 monclient(hunting): get_auth_request con 0x7f6844098210 auth_method 0 2023-09-30T01:36:28.807 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:28.789+0000 7f6842d9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:36:28.807 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:28.789+0000 7f6842d9d700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:36:28.807 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:28.789+0000 7f6842d9d700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:36:28.807 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:28.789+0000 7f684259c700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:36:28.808 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:28.789+0000 7f684259c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:36:28.808 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:28.789+0000 7f684259c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:36:28.808 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:28.789+0000 7f6842d9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:36:28.808 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:28.789+0000 7f6842d9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:36:28.808 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:28.789+0000 7f6842d9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:36:28.808 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:28.789+0000 7f6841d9b700 10 monclient(hunting): handle_auth_done global_id 4959 payload 824 2023-09-30T01:36:28.808 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:28.789+0000 7f6841d9b700 1 -- 172.21.15.88:0/953599096 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7f6844098210 msgr2=0x7f684408d3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:36:28.808 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:28.789+0000 7f6841d9b700 1 --2- 172.21.15.88:0/953599096 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7f6844098210 0x7f684408d3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:36:28.808 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:28.789+0000 7f6841d9b700 1 -- 172.21.15.88:0/953599096 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7f68440966a0 msgr2=0x7f68441357e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:36:28.808 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:28.789+0000 7f6841d9b700 1 --2- 172.21.15.88:0/953599096 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7f68440966a0 0x7f68441357e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:36:28.808 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:28.789+0000 7f6841d9b700 10 monclient: _finish_hunting 0 2023-09-30T01:36:28.808 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:28.789+0000 7f6841d9b700 1 monclient: found mon.noname-a 2023-09-30T01:36:28.809 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:28.789+0000 7f6841d9b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:36:28.809 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:28.789+0000 7f6841d9b700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.88:3300/0 2023-09-30T01:36:28.809 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:28.789+0000 7f6841d9b700 1 -- 172.21.15.88:0/953599096 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f683c009ae0 con 0x7f6844097e40 2023-09-30T01:36:28.809 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:28.789+0000 7f6841d9b700 1 --2- 172.21.15.88:0/953599096 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6844097e40 0x7f6844135d20 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f683800ec80 tx=0x7f68380091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:36:28.809 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:28.793+0000 7f68327fc700 1 -- 172.21.15.88:0/953599096 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f6838009690 con 0x7f6844097e40 2023-09-30T01:36:28.809 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:28.793+0000 7f68327fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:36:28.809 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:28.793+0000 7f68327fc700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T01:36:28.809 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:28.793+0000 7f68327fc700 10 monclient: dump: 2023-09-30T01:36:28.809 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:36:28.809 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:36:28.809 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:36:28.809 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:36:28.810 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:36:28.810 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:36:28.810 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:36:28.810 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:36:28.810 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:36:28.810 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:36:28.810 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:36:28.810 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:36:28.810 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:36:28.810 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:36:28.810 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:36:28.810 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:36:28.810 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:28.793+0000 7f68327fc700 10 monclient: _finish_auth 0 2023-09-30T01:36:28.811 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:28.793+0000 7f68327fc700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:36:28.811 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:28.793+0000 7f68327fc700 1 -- 172.21.15.88:0/953599096 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f683800ccd0 con 0x7f6844097e40 2023-09-30T01:36:28.811 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:28.793+0000 7f68327fc700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:36:28.811 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:28.793+0000 7f68327fc700 1 -- 172.21.15.88:0/953599096 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f683800ce40 con 0x7f6844097e40 2023-09-30T01:36:28.811 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:28.793+0000 7f6848afd700 5 monclient: authenticate success, global_id 4959 2023-09-30T01:36:28.811 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:28.793+0000 7f6848afd700 10 monclient: _renew_subs 2023-09-30T01:36:28.811 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:28.793+0000 7f6840d99700 4 set_mon_vals no callback set 2023-09-30T01:36:28.811 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:28.793+0000 7f6848afd700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.88:3300/0 2023-09-30T01:36:28.811 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:28.793+0000 7f6848afd700 1 -- 172.21.15.88:0/953599096 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f68440906a0 con 0x7f6844097e40 2023-09-30T01:36:28.811 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:28.793+0000 7f68327fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:36:28.811 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:28.793+0000 7f68327fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:36:28.811 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:28.793+0000 7f68327fc700 10 monclient: dump: 2023-09-30T01:36:28.812 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:36:28.812 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:36:28.812 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:36:28.812 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:36:28.812 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:36:28.812 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:36:28.812 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:36:28.812 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:36:28.812 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:36:28.812 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:36:28.812 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:36:28.812 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:36:28.812 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:36:28.813 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:36:28.813 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:36:28.813 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:36:28.813 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:28.793+0000 7f6848afd700 10 monclient: _renew_subs 2023-09-30T01:36:28.813 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:28.793+0000 7f6848afd700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.88:3300/0 2023-09-30T01:36:28.813 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:28.793+0000 7f6848afd700 1 -- 172.21.15.88:0/953599096 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f684408eb70 con 0x7f6844097e40 2023-09-30T01:36:28.813 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:28.793+0000 7f68327fc700 1 -- 172.21.15.88:0/953599096 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 40144+0+0 (secure 0 0 0) 0x7f683800c850 con 0x7f6844097e40 2023-09-30T01:36:28.813 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:28.793+0000 7f6848afd700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:36:28.813 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:28.793+0000 7f68327fc700 1 --2- 172.21.15.88:0/953599096 >> v2:172.21.15.107:6812/29926 conn(0x7f6824033360 0x7f6824035810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:36:28.813 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:28.793+0000 7f6848afd700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:36:28.813 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:28.793+0000 7f6848afd700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.88:3300/0 2023-09-30T01:36:28.813 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:28.793+0000 7f68327fc700 1 -- 172.21.15.88:0/953599096 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 3502+0+0 (secure 0 0 0) 0x7f68380436c0 con 0x7f6844097e40 2023-09-30T01:36:28.813 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:28.793+0000 7f6848afd700 1 -- 172.21.15.88:0/953599096 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f68441375a0 con 0x7f6844097e40 2023-09-30T01:36:28.814 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:28.793+0000 7f684259c700 1 --2- 172.21.15.88:0/953599096 >> v2:172.21.15.107:6812/29926 conn(0x7f6824033360 0x7f6824035810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:36:28.814 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:28.793+0000 7f684259c700 10 monclient: get_auth_request con 0x7f6824033360 auth_method 0 2023-09-30T01:36:28.814 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:28.793+0000 7f684259c700 1 --2- 172.21.15.88:0/953599096 >> v2:172.21.15.107:6812/29926 conn(0x7f6824033360 0x7f6824035810 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f6834003aa0 tx=0x7f6834000bc0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:36:28.814 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:28.801+0000 7f68327fc700 1 -- 172.21.15.88:0/953599096 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f68441375a0 con 0x7f6844097e40 2023-09-30T01:36:28.814 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:28.801+0000 7f68327fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:36:28.814 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:28.801+0000 7f68327fc700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:36:29.000 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:28.997+0000 7f6848afd700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T01:36:29.001 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:28.997+0000 7f6848afd700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:36:29.001 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:28.997+0000 7f6848afd700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.88:3300/0 2023-09-30T01:36:29.001 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:28.997+0000 7f6848afd700 1 -- 172.21.15.88:0/953599096 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f68441375a0 con 0x7f6844097e40 2023-09-30T01:36:29.001 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:28.997+0000 7f68327fc700 1 -- 172.21.15.88:0/953599096 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f6838029100 con 0x7f6844097e40 2023-09-30T01:36:29.001 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:28.997+0000 7f68327fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:36:29.001 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:28.997+0000 7f68327fc700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:36:29.002 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-09-30T01:36:29.002 INFO:teuthology.orchestra.run.smithi088.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:36:29.003 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:28.997+0000 7f6848afd700 1 -- 172.21.15.88:0/953599096 >> v2:172.21.15.107:6812/29926 conn(0x7f6824033360 msgr2=0x7f6824035810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:36:29.003 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:28.997+0000 7f6848afd700 1 --2- 172.21.15.88:0/953599096 >> v2:172.21.15.107:6812/29926 conn(0x7f6824033360 0x7f6824035810 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f6834003aa0 tx=0x7f6834000bc0 comp rx=0 tx=0).stop 2023-09-30T01:36:29.003 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:28.997+0000 7f6848afd700 10 monclient: shutdown 2023-09-30T01:36:29.003 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:28.997+0000 7f6848afd700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:36:29.003 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:28.997+0000 7f6848afd700 1 -- 172.21.15.88:0/953599096 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6844097e40 msgr2=0x7f6844135d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:36:29.004 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:28.997+0000 7f6848afd700 1 --2- 172.21.15.88:0/953599096 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6844097e40 0x7f6844135d20 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f683800ec80 tx=0x7f68380091a0 comp rx=0 tx=0).stop 2023-09-30T01:36:29.004 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:28.997+0000 7f6848afd700 1 -- 172.21.15.88:0/953599096 shutdown_connections 2023-09-30T01:36:29.004 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:28.997+0000 7f6848afd700 1 --2- 172.21.15.88:0/953599096 >> v2:172.21.15.107:6812/29926 conn(0x7f6824033360 0x7f6824035810 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:36:29.004 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:28.997+0000 7f6848afd700 1 --2- 172.21.15.88:0/953599096 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7f6844098210 0x7f684408d3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:36:29.004 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:28.997+0000 7f6848afd700 1 --2- 172.21.15.88:0/953599096 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6844097e40 0x7f6844135d20 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:36:29.004 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:28.997+0000 7f6848afd700 1 --2- 172.21.15.88:0/953599096 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7f68440966a0 0x7f68441357e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:36:29.004 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:28.997+0000 7f6848afd700 1 -- 172.21.15.88:0/953599096 >> 172.21.15.88:0/953599096 conn(0x7f68440a3440 msgr2=0x7f68441366c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:36:29.004 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:28.997+0000 7f6848afd700 1 -- 172.21.15.88:0/953599096 shutdown_connections 2023-09-30T01:36:29.004 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:28.997+0000 7f6848afd700 1 -- 172.21.15.88:0/953599096 wait complete. 2023-09-30T01:36:32.020 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:36:32.135 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:32.129+0000 7fde9ffff700 10 monclient: get_monmap_and_config 2023-09-30T01:36:32.135 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:32.129+0000 7fde9ffff700 10 monclient: build_initial_monmap 2023-09-30T01:36:32.135 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:32.129+0000 7fde9ffff700 10 monclient: monmap: 2023-09-30T01:36:32.135 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2023-09-30T01:36:32.135 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:36:32.135 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:36:32.134978+0000 2023-09-30T01:36:32.135 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:36:32.134978+0000 2023-09-30T01:36:32.136 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2023-09-30T01:36:32.136 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:36:32.136 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a-legacy 2023-09-30T01:36:32.136 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.107:6789/0 mon.noname-f-legacy 2023-09-30T01:36:32.136 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.noname-b 2023-09-30T01:36:32.136 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.noname-g 2023-09-30T01:36:32.136 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.noname-c 2023-09-30T01:36:32.136 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.noname-h 2023-09-30T01:36:32.136 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.noname-d 2023-09-30T01:36:32.136 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.noname-i 2023-09-30T01:36:32.136 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.noname-e 2023-09-30T01:36:32.136 INFO:teuthology.orchestra.run.smithi088.stderr:9: v2:172.21.15.88:3300/0 mon.noname-a 2023-09-30T01:36:32.137 INFO:teuthology.orchestra.run.smithi088.stderr:10: v2:172.21.15.107:3300/0 mon.noname-f 2023-09-30T01:36:32.137 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:36:32.140 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:32.133+0000 7fde9ffff700 1 Processor -- start 2023-09-30T01:36:32.140 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:32.133+0000 7fde9ffff700 1 -- start start 2023-09-30T01:36:32.140 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:32.133+0000 7fde9ffff700 10 monclient: init 2023-09-30T01:36:32.140 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:32.133+0000 7fde9ffff700 10 monclient: _reopen_session rank -1 2023-09-30T01:36:32.140 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:32.133+0000 7fde9ffff700 10 monclient: _add_conns ranks=[9,1,4,8,10,6,2,3,0,5,7] 2023-09-30T01:36:32.140 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:32.133+0000 7fde9ffff700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fdea0097a40 0x7fdea0132700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:36:32.140 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:32.133+0000 7fde9ffff700 10 monclient(hunting): picked mon.noname-a con 0x7fdea0097a40 addr v2:172.21.15.88:3300/0 2023-09-30T01:36:32.140 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:32.133+0000 7fde9ffff700 10 monclient(hunting): picked mon.noname-f-legacy con 0x7fdea01311c0 addr v1:172.21.15.107:6789/0 2023-09-30T01:36:32.140 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:32.133+0000 7fde9ffff700 1 --2- >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7fdea0132c40 0x7fdea01339f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:36:32.141 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:32.133+0000 7fde9ffff700 10 monclient(hunting): picked mon.noname-c con 0x7fdea0132c40 addr [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] 2023-09-30T01:36:32.141 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:32.133+0000 7fde9ffff700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdea004bb20 con 0x7fdea01311c0 2023-09-30T01:36:32.141 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:32.133+0000 7fde9ffff700 10 monclient(hunting): start opening mon connection 2023-09-30T01:36:32.141 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:32.133+0000 7fde9ffff700 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdea0131590 con 0x7fdea0097a40 2023-09-30T01:36:32.141 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:32.133+0000 7fde9ffff700 10 monclient(hunting): start opening mon connection 2023-09-30T01:36:32.141 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:32.133+0000 7fde9ffff700 1 -- --> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] -- mon_getmap magic: 0 v1 -- 0x7fdea009a5d0 con 0x7fdea0132c40 2023-09-30T01:36:32.141 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:32.133+0000 7fde9ffff700 10 monclient(hunting): _renew_subs 2023-09-30T01:36:32.141 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:32.133+0000 7fde9ffff700 10 monclient(hunting): authenticate will time out at 1064.173016s 2023-09-30T01:36:32.141 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:32.133+0000 7fde9f7fe700 1 --2- >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7fdea0132c40 0x7fdea01339f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:36:32.141 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:32.133+0000 7fde9effd700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fdea0097a40 0x7fdea0132700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:36:32.141 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:32.133+0000 7fde9f7fe700 1 --2- >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7fdea0132c40 0x7fdea01339f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3302/0 says I am v2:172.21.15.88:37594/0 (socket says 172.21.15.88:37594) 2023-09-30T01:36:32.141 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:32.133+0000 7fde9f7fe700 1 -- 172.21.15.88:0/1279571751 learned_addr learned my addr 172.21.15.88:0/1279571751 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-09-30T01:36:32.142 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:32.133+0000 7fde9f7fe700 10 monclient(hunting): get_auth_request con 0x7fdea0132c40 auth_method 0 2023-09-30T01:36:32.142 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:32.133+0000 7fde9f7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:36:32.142 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:32.133+0000 7fde9f7fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:36:32.142 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:32.133+0000 7fde9f7fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:36:32.142 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:32.133+0000 7fde9effd700 10 monclient(hunting): get_auth_request con 0x7fdea0097a40 auth_method 0 2023-09-30T01:36:32.142 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:32.133+0000 7fde9effd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:36:32.142 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:32.133+0000 7fde9effd700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:36:32.142 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:32.133+0000 7fde9effd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:36:32.142 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:32.133+0000 7fde9f7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:36:32.142 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:32.133+0000 7fde9f7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:36:32.142 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:32.133+0000 7fde9f7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:36:32.142 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:32.133+0000 7fde9effd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:36:32.143 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:32.133+0000 7fde9effd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:36:32.143 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:32.133+0000 7fde9effd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:36:32.143 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:32.133+0000 7fde9f7fe700 10 monclient(hunting): handle_auth_done global_id 4711 payload 274 2023-09-30T01:36:32.143 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:32.133+0000 7fde9f7fe700 1 -- 172.21.15.88:0/1279571751 >> v2:172.21.15.88:3300/0 conn(0x7fdea0097a40 msgr2=0x7fdea0132700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:36:32.143 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:32.133+0000 7fde9f7fe700 1 --2- 172.21.15.88:0/1279571751 >> v2:172.21.15.88:3300/0 conn(0x7fdea0097a40 0x7fdea0132700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:36:32.143 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:32.133+0000 7fde9f7fe700 1 -- 172.21.15.88:0/1279571751 >> v1:172.21.15.107:6789/0 conn(0x7fdea01311c0 legacy=0x7fdea01332e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:36:32.143 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:32.133+0000 7fde9f7fe700 10 monclient: _finish_hunting 0 2023-09-30T01:36:32.143 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:32.133+0000 7fde9f7fe700 1 monclient: found mon.noname-c 2023-09-30T01:36:32.143 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:32.133+0000 7fde9f7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:36:32.143 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:32.133+0000 7fde9f7fe700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.88:3302/0 2023-09-30T01:36:32.143 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:32.133+0000 7fde9f7fe700 1 -- 172.21.15.88:0/1279571751 --> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdea01330a0 con 0x7fdea0132c40 2023-09-30T01:36:32.143 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:32.133+0000 7fde9f7fe700 1 --2- 172.21.15.88:0/1279571751 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7fdea0132c40 0x7fdea01339f0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fde98009f20 tx=0x7fde98009840 comp rx=0 tx=0).ready entity=mon.4 client_cookie=bedafa183949a58b server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:36:32.143 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:32.133+0000 7fde9dffb700 1 -- 172.21.15.88:0/1279571751 <== mon.4 v2:172.21.15.88:3302/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7fde9800e070 con 0x7fdea0132c40 2023-09-30T01:36:32.144 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:32.133+0000 7fde9dffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:36:32.144 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:32.133+0000 7fde9dffb700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T01:36:32.144 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:32.133+0000 7fde9dffb700 10 monclient: dump: 2023-09-30T01:36:32.144 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:36:32.144 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:36:32.144 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:36:32.144 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:36:32.144 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:36:32.144 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:36:32.144 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:36:32.144 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:36:32.144 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:36:32.144 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:36:32.145 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:36:32.145 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:36:32.145 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:36:32.145 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:36:32.145 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:36:32.145 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:36:32.145 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:32.133+0000 7fde9dffb700 10 monclient: _finish_auth 0 2023-09-30T01:36:32.145 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:32.133+0000 7fde9dffb700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:36:32.145 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:32.133+0000 7fde9ffff700 5 monclient: authenticate success, global_id 4711 2023-09-30T01:36:32.145 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:32.133+0000 7fde9ffff700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:36:32.145 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:32.133+0000 7fde9dffb700 1 -- 172.21.15.88:0/1279571751 <== mon.4 v2:172.21.15.88:3302/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde9800be50 con 0x7fdea0132c40 2023-09-30T01:36:32.145 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:32.133+0000 7fde9dffb700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:36:32.145 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:32.133+0000 7fde9ffff700 10 monclient: get_monmap_and_config success 2023-09-30T01:36:32.146 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:32.133+0000 7fde9dffb700 1 -- 172.21.15.88:0/1279571751 <== mon.4 v2:172.21.15.88:3302/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7fde980092e0 con 0x7fdea0132c40 2023-09-30T01:36:32.146 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:32.133+0000 7fde9ffff700 4 set_mon_vals no callback set 2023-09-30T01:36:32.146 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:32.133+0000 7fde9dffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:36:32.146 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:32.133+0000 7fde9dffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T01:36:32.146 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:32.133+0000 7fde9dffb700 10 monclient: dump: 2023-09-30T01:36:32.146 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:36:32.146 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:36:32.146 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:36:32.146 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:36:32.146 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:36:32.146 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:36:32.146 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:36:32.147 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:36:32.147 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:36:32.147 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:36:32.147 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:36:32.147 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:36:32.147 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:36:32.147 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:36:32.147 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:36:32.147 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:36:32.147 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:32.133+0000 7fde9ffff700 10 monclient: shutdown 2023-09-30T01:36:32.147 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:32.133+0000 7fde9ffff700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:36:32.147 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:32.133+0000 7fde9ffff700 1 -- 172.21.15.88:0/1279571751 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7fdea0132c40 msgr2=0x7fdea01339f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:36:32.147 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:32.133+0000 7fde9ffff700 1 --2- 172.21.15.88:0/1279571751 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7fdea0132c40 0x7fdea01339f0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fde98009f20 tx=0x7fde98009840 comp rx=0 tx=0).stop 2023-09-30T01:36:32.148 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:32.133+0000 7fde9ffff700 1 -- 172.21.15.88:0/1279571751 shutdown_connections 2023-09-30T01:36:32.148 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:32.133+0000 7fde9ffff700 1 --2- 172.21.15.88:0/1279571751 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7fdea0132c40 0x7fdea01339f0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:36:32.148 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:32.133+0000 7fde9ffff700 1 --2- 172.21.15.88:0/1279571751 >> v2:172.21.15.88:3300/0 conn(0x7fdea0097a40 0x7fdea0132700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:36:32.148 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:32.133+0000 7fde9ffff700 1 -- 172.21.15.88:0/1279571751 >> 172.21.15.88:0/1279571751 conn(0x7fdea006e5e0 msgr2=0x7fdea006e9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:36:32.148 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:32.133+0000 7fde9ffff700 1 -- 172.21.15.88:0/1279571751 shutdown_connections 2023-09-30T01:36:32.148 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:32.133+0000 7fde9ffff700 1 -- 172.21.15.88:0/1279571751 wait complete. 2023-09-30T01:36:32.148 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:32.137+0000 7fde9ffff700 10 monclient: build_initial_monmap 2023-09-30T01:36:32.153 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:32.137+0000 7fde9ffff700 10 monclient: monmap: 2023-09-30T01:36:32.153 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2023-09-30T01:36:32.153 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:36:32.153 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 0.000000 2023-09-30T01:36:32.153 INFO:teuthology.orchestra.run.smithi088.stderr:created 0.000000 2023-09-30T01:36:32.153 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2023-09-30T01:36:32.153 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 1 2023-09-30T01:36:32.153 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.noname-a 2023-09-30T01:36:32.153 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.noname-f 2023-09-30T01:36:32.153 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.noname-b 2023-09-30T01:36:32.154 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.noname-g 2023-09-30T01:36:32.154 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.noname-c 2023-09-30T01:36:32.154 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.noname-h 2023-09-30T01:36:32.154 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.noname-d 2023-09-30T01:36:32.154 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.noname-i 2023-09-30T01:36:32.154 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.noname-e 2023-09-30T01:36:32.154 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:36:32.154 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:32.137+0000 7fde9ffff700 1 Processor -- start 2023-09-30T01:36:32.154 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:32.137+0000 7fde9ffff700 1 -- start start 2023-09-30T01:36:32.154 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:32.137+0000 7fde9ffff700 10 monclient: init 2023-09-30T01:36:32.154 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:32.137+0000 7fde9ffff700 10 monclient: _reopen_session rank -1 2023-09-30T01:36:32.154 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:32.137+0000 7fde9ffff700 10 monclient: _add_conns ranks=[6,5,0,8,7,2,1,3,4] 2023-09-30T01:36:32.154 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:32.137+0000 7fde9ffff700 1 --2- >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7fdea0097a40 0x7fdea011cd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:36:32.155 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:32.137+0000 7fde9ffff700 10 monclient(hunting): picked mon.noname-d con 0x7fdea0097a40 addr [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] 2023-09-30T01:36:32.155 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:32.137+0000 7fde9ffff700 1 --2- >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7fdea01311c0 0x7fdea011d290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:36:32.155 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:32.137+0000 7fde9ffff700 10 monclient(hunting): picked mon.noname-h con 0x7fdea01311c0 addr [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] 2023-09-30T01:36:32.155 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:32.137+0000 7fde9ffff700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdea0132c40 0x7fdea011d7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:36:32.155 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:32.137+0000 7fde9ffff700 10 monclient(hunting): picked mon.noname-a con 0x7fdea0132c40 addr [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] 2023-09-30T01:36:32.155 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:32.137+0000 7fde9ffff700 10 monclient(hunting): start opening mon connection 2023-09-30T01:36:32.155 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:32.137+0000 7fde9ffff700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdea011f020 con 0x7fdea0132c40 2023-09-30T01:36:32.155 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:32.137+0000 7fde9ffff700 10 monclient(hunting): start opening mon connection 2023-09-30T01:36:32.155 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:32.137+0000 7fde9ffff700 1 -- --> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] -- mon_getmap magic: 0 v1 -- 0x7fdea011f650 con 0x7fdea01311c0 2023-09-30T01:36:32.155 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:32.137+0000 7fde9ffff700 10 monclient(hunting): start opening mon connection 2023-09-30T01:36:32.155 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:32.137+0000 7fde9ffff700 1 -- --> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] -- mon_getmap magic: 0 v1 -- 0x7fdea00a2fa0 con 0x7fdea0097a40 2023-09-30T01:36:32.155 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:32.137+0000 7fde9ffff700 10 monclient(hunting): _renew_subs 2023-09-30T01:36:32.156 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:32.137+0000 7fde9ffff700 10 monclient(hunting): authenticate will time out at 1064.179341s 2023-09-30T01:36:32.156 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:32.137+0000 7fde9f7fe700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdea0132c40 0x7fdea011d7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:36:32.156 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:32.137+0000 7fde9f7fe700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdea0132c40 0x7fdea011d7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:49602/0 (socket says 172.21.15.88:49602) 2023-09-30T01:36:32.156 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:32.137+0000 7fde9f7fe700 1 -- 172.21.15.88:0/3977006579 learned_addr learned my addr 172.21.15.88:0/3977006579 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-09-30T01:36:32.156 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:32.137+0000 7fde9f7fe700 10 monclient(hunting): get_auth_request con 0x7fdea0132c40 auth_method 0 2023-09-30T01:36:32.156 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:32.137+0000 7fde9f7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:36:32.156 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:32.137+0000 7fde9f7fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:36:32.156 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:32.137+0000 7fde9f7fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:36:32.156 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:32.137+0000 7fde9effd700 1 --2- 172.21.15.88:0/3977006579 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7fdea0097a40 0x7fdea011cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:36:32.156 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:32.137+0000 7fde9effd700 10 monclient(hunting): get_auth_request con 0x7fdea0097a40 auth_method 0 2023-09-30T01:36:32.156 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:32.137+0000 7fde9effd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:36:32.156 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:32.137+0000 7fde9effd700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:36:32.156 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:32.137+0000 7fde9effd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:36:32.157 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:32.137+0000 7fde9e7fc700 1 --2- 172.21.15.88:0/3977006579 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7fdea01311c0 0x7fdea011d290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:36:32.157 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:32.137+0000 7fde9f7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:36:32.157 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:32.137+0000 7fde9f7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:36:32.157 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:32.137+0000 7fde9f7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:36:32.157 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:32.137+0000 7fde9e7fc700 10 monclient(hunting): get_auth_request con 0x7fdea01311c0 auth_method 0 2023-09-30T01:36:32.157 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:32.137+0000 7fde9e7fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:36:32.157 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:32.137+0000 7fde9e7fc700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:36:32.157 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:32.137+0000 7fde9e7fc700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:36:32.157 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:32.137+0000 7fde9effd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:36:32.157 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:32.137+0000 7fde9effd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:36:32.157 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:32.137+0000 7fde9effd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:36:32.157 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:32.137+0000 7fde9e7fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:36:32.157 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:32.137+0000 7fde9e7fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:36:32.158 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:32.137+0000 7fde9e7fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:36:32.158 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:32.141+0000 7fde9f7fe700 10 monclient(hunting): handle_auth_done global_id 4977 payload 824 2023-09-30T01:36:32.158 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:32.141+0000 7fde9f7fe700 1 -- 172.21.15.88:0/3977006579 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7fdea0097a40 msgr2=0x7fdea011cd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:36:32.158 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:32.141+0000 7fde9f7fe700 1 --2- 172.21.15.88:0/3977006579 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7fdea0097a40 0x7fdea011cd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:36:32.158 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:32.141+0000 7fde9f7fe700 1 -- 172.21.15.88:0/3977006579 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7fdea01311c0 msgr2=0x7fdea011d290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:36:32.158 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:32.141+0000 7fde9f7fe700 1 --2- 172.21.15.88:0/3977006579 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7fdea01311c0 0x7fdea011d290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:36:32.158 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:32.141+0000 7fde9f7fe700 10 monclient: _finish_hunting 0 2023-09-30T01:36:32.158 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:32.141+0000 7fde9f7fe700 1 monclient: found mon.noname-a 2023-09-30T01:36:32.158 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:32.141+0000 7fde9f7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:36:32.158 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:32.141+0000 7fde9f7fe700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.88:3300/0 2023-09-30T01:36:32.158 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:32.141+0000 7fde9f7fe700 1 -- 172.21.15.88:0/3977006579 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde98009ae0 con 0x7fdea0132c40 2023-09-30T01:36:32.158 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:32.141+0000 7fde9f7fe700 1 --2- 172.21.15.88:0/3977006579 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdea0132c40 0x7fdea011d7d0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fde98009990 tx=0x7fde98018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:36:32.159 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:32.141+0000 7fde86ffd700 1 -- 172.21.15.88:0/3977006579 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7fde98003300 con 0x7fdea0132c40 2023-09-30T01:36:32.159 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:32.141+0000 7fde86ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:36:32.159 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:32.141+0000 7fde86ffd700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T01:36:32.159 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:32.141+0000 7fde86ffd700 10 monclient: dump: 2023-09-30T01:36:32.159 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:36:32.159 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:36:32.159 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:36:32.159 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:36:32.159 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:36:32.159 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:36:32.159 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:36:32.160 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:36:32.160 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:36:32.160 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:36:32.160 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:36:32.160 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:36:32.160 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:36:32.160 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:36:32.160 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:36:32.160 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:36:32.160 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:32.141+0000 7fde86ffd700 10 monclient: _finish_auth 0 2023-09-30T01:36:32.160 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:32.141+0000 7fde86ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:36:32.160 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:32.141+0000 7fde86ffd700 1 -- 172.21.15.88:0/3977006579 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde980092e0 con 0x7fdea0132c40 2023-09-30T01:36:32.160 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:32.141+0000 7fde9ffff700 5 monclient: authenticate success, global_id 4977 2023-09-30T01:36:32.161 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:32.141+0000 7fde9ffff700 10 monclient: _renew_subs 2023-09-30T01:36:32.161 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:32.141+0000 7fde9ffff700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.88:3300/0 2023-09-30T01:36:32.161 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:32.141+0000 7fde9ffff700 1 -- 172.21.15.88:0/3977006579 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdea0099c30 con 0x7fdea0132c40 2023-09-30T01:36:32.161 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:32.141+0000 7fde86ffd700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:36:32.161 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:32.141+0000 7fde86ffd700 1 -- 172.21.15.88:0/3977006579 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7fde98007000 con 0x7fdea0132c40 2023-09-30T01:36:32.161 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:32.141+0000 7fde86ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:36:32.161 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:32.141+0000 7fde86ffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:36:32.161 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:32.141+0000 7fde86ffd700 10 monclient: dump: 2023-09-30T01:36:32.161 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:36:32.161 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:36:32.161 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:36:32.161 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:36:32.161 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:36:32.162 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:36:32.162 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:36:32.162 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:36:32.162 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:36:32.162 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:36:32.162 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:36:32.162 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:36:32.162 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:36:32.162 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:36:32.162 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:36:32.162 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:36:32.162 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:32.141+0000 7fde9d7fa700 4 set_mon_vals no callback set 2023-09-30T01:36:32.163 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:32.141+0000 7fde9ffff700 10 monclient: _renew_subs 2023-09-30T01:36:32.163 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:32.141+0000 7fde9ffff700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.88:3300/0 2023-09-30T01:36:32.163 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:32.141+0000 7fde9ffff700 1 -- 172.21.15.88:0/3977006579 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdea011e0c0 con 0x7fdea0132c40 2023-09-30T01:36:32.163 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:32.141+0000 7fde86ffd700 1 -- 172.21.15.88:0/3977006579 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 40144+0+0 (secure 0 0 0) 0x7fde98003050 con 0x7fdea0132c40 2023-09-30T01:36:32.163 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:32.141+0000 7fde9ffff700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:36:32.163 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:32.141+0000 7fde9ffff700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:36:32.163 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:32.141+0000 7fde9ffff700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.88:3300/0 2023-09-30T01:36:32.163 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:32.141+0000 7fde9ffff700 1 -- 172.21.15.88:0/3977006579 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdea0071960 con 0x7fdea0132c40 2023-09-30T01:36:32.163 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:32.141+0000 7fde86ffd700 1 --2- 172.21.15.88:0/3977006579 >> v2:172.21.15.107:6812/29926 conn(0x7fde8c033360 0x7fde8c035810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:36:32.163 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:32.141+0000 7fde86ffd700 1 -- 172.21.15.88:0/3977006579 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 3502+0+0 (secure 0 0 0) 0x7fde98044cf0 con 0x7fdea0132c40 2023-09-30T01:36:32.163 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:32.141+0000 7fde9effd700 1 --2- 172.21.15.88:0/3977006579 >> v2:172.21.15.107:6812/29926 conn(0x7fde8c033360 0x7fde8c035810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:36:32.163 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:32.141+0000 7fde9effd700 10 monclient: get_auth_request con 0x7fde8c033360 auth_method 0 2023-09-30T01:36:32.164 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:32.141+0000 7fde9effd700 1 --2- 172.21.15.88:0/3977006579 >> v2:172.21.15.107:6812/29926 conn(0x7fde8c033360 0x7fde8c035810 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fde90009db0 tx=0x7fde900091a0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:36:32.164 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:32.149+0000 7fde86ffd700 1 -- 172.21.15.88:0/3977006579 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fde98011390 con 0x7fdea0132c40 2023-09-30T01:36:32.164 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:32.149+0000 7fde86ffd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:36:32.164 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:32.149+0000 7fde86ffd700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:36:32.343 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:32.337+0000 7fde9ffff700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T01:36:32.343 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:32.337+0000 7fde9ffff700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:36:32.343 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:32.337+0000 7fde9ffff700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.88:3300/0 2023-09-30T01:36:32.343 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:32.337+0000 7fde9ffff700 1 -- 172.21.15.88:0/3977006579 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fdea0067b60 con 0x7fdea0132c40 2023-09-30T01:36:32.343 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:32.337+0000 7fde86ffd700 1 -- 172.21.15.88:0/3977006579 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7fde9801cba0 con 0x7fdea0132c40 2023-09-30T01:36:32.343 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:32.337+0000 7fde86ffd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:36:32.344 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:32.337+0000 7fde86ffd700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:36:32.344 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-09-30T01:36:32.344 INFO:teuthology.orchestra.run.smithi088.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:36:32.347 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:32.341+0000 7fde9ffff700 1 -- 172.21.15.88:0/3977006579 >> v2:172.21.15.107:6812/29926 conn(0x7fde8c033360 msgr2=0x7fde8c035810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:36:32.347 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:32.341+0000 7fde9ffff700 1 --2- 172.21.15.88:0/3977006579 >> v2:172.21.15.107:6812/29926 conn(0x7fde8c033360 0x7fde8c035810 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fde90009db0 tx=0x7fde900091a0 comp rx=0 tx=0).stop 2023-09-30T01:36:32.347 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:32.341+0000 7fde9ffff700 10 monclient: shutdown 2023-09-30T01:36:32.347 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:32.341+0000 7fde9ffff700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:36:32.347 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:32.341+0000 7fde9ffff700 1 -- 172.21.15.88:0/3977006579 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdea0132c40 msgr2=0x7fdea011d7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:36:32.347 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:32.341+0000 7fde9ffff700 1 --2- 172.21.15.88:0/3977006579 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdea0132c40 0x7fdea011d7d0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fde98009990 tx=0x7fde98018000 comp rx=0 tx=0).stop 2023-09-30T01:36:32.347 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:32.341+0000 7fde9ffff700 1 -- 172.21.15.88:0/3977006579 shutdown_connections 2023-09-30T01:36:32.348 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:32.341+0000 7fde9ffff700 1 --2- 172.21.15.88:0/3977006579 >> v2:172.21.15.107:6812/29926 conn(0x7fde8c033360 0x7fde8c035810 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:36:32.348 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:32.341+0000 7fde9ffff700 1 --2- 172.21.15.88:0/3977006579 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdea0132c40 0x7fdea011d7d0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:36:32.348 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:32.341+0000 7fde9ffff700 1 --2- 172.21.15.88:0/3977006579 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7fdea01311c0 0x7fdea011d290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:36:32.348 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:32.341+0000 7fde9ffff700 1 --2- 172.21.15.88:0/3977006579 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7fdea0097a40 0x7fdea011cd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:36:32.348 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:32.341+0000 7fde9ffff700 1 -- 172.21.15.88:0/3977006579 >> 172.21.15.88:0/3977006579 conn(0x7fdea006e5e0 msgr2=0x7fdea0133aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:36:32.348 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:32.341+0000 7fde9ffff700 1 -- 172.21.15.88:0/3977006579 shutdown_connections 2023-09-30T01:36:32.348 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:32.341+0000 7fde9ffff700 1 -- 172.21.15.88:0/3977006579 wait complete. 2023-09-30T01:36:35.363 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:36:35.475 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:35.469+0000 7f39129de700 10 monclient: get_monmap_and_config 2023-09-30T01:36:35.475 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:35.469+0000 7f39129de700 10 monclient: build_initial_monmap 2023-09-30T01:36:35.475 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:35.469+0000 7f39129de700 10 monclient: monmap: 2023-09-30T01:36:35.476 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2023-09-30T01:36:35.476 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:36:35.476 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:36:35.475491+0000 2023-09-30T01:36:35.476 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:36:35.475491+0000 2023-09-30T01:36:35.476 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2023-09-30T01:36:35.476 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:36:35.476 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a-legacy 2023-09-30T01:36:35.476 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.107:6789/0 mon.noname-f-legacy 2023-09-30T01:36:35.476 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.noname-b 2023-09-30T01:36:35.476 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.noname-g 2023-09-30T01:36:35.476 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.noname-c 2023-09-30T01:36:35.476 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.noname-h 2023-09-30T01:36:35.477 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.noname-d 2023-09-30T01:36:35.477 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.noname-i 2023-09-30T01:36:35.477 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.noname-e 2023-09-30T01:36:35.477 INFO:teuthology.orchestra.run.smithi088.stderr:9: v2:172.21.15.88:3300/0 mon.noname-a 2023-09-30T01:36:35.477 INFO:teuthology.orchestra.run.smithi088.stderr:10: v2:172.21.15.107:3300/0 mon.noname-f 2023-09-30T01:36:35.477 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:36:35.484 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:35.473+0000 7f39129de700 1 Processor -- start 2023-09-30T01:36:35.484 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:35.473+0000 7f39129de700 1 -- start start 2023-09-30T01:36:35.484 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:35.473+0000 7f39129de700 10 monclient: init 2023-09-30T01:36:35.485 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:35.473+0000 7f39129de700 10 monclient: _reopen_session rank -1 2023-09-30T01:36:35.485 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:35.473+0000 7f39129de700 10 monclient: _add_conns ranks=[10,1,3,9,5,8,2,4,7,0,6] 2023-09-30T01:36:35.485 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:35.473+0000 7f39129de700 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f390c097a40 0x7f390c132700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:36:35.485 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:35.473+0000 7f39129de700 10 monclient(hunting): picked mon.noname-f con 0x7f390c097a40 addr v2:172.21.15.107:3300/0 2023-09-30T01:36:35.485 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:35.473+0000 7f39129de700 10 monclient(hunting): picked mon.noname-f-legacy con 0x7f390c1311c0 addr v1:172.21.15.107:6789/0 2023-09-30T01:36:35.485 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:35.473+0000 7f39129de700 1 --2- >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7f390c132c40 0x7f390c1339f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:36:35.485 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:35.473+0000 7f39129de700 10 monclient(hunting): picked mon.noname-g con 0x7f390c132c40 addr [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] 2023-09-30T01:36:35.485 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:35.473+0000 7f39129de700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f390c04bb20 con 0x7f390c1311c0 2023-09-30T01:36:35.485 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:35.473+0000 7f39129de700 10 monclient(hunting): start opening mon connection 2023-09-30T01:36:35.485 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:35.473+0000 7f39129de700 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f390c131590 con 0x7f390c097a40 2023-09-30T01:36:35.485 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:35.473+0000 7f39129de700 10 monclient(hunting): start opening mon connection 2023-09-30T01:36:35.485 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:35.473+0000 7f39129de700 1 -- --> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f390c09a5d0 con 0x7f390c132c40 2023-09-30T01:36:35.486 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:35.473+0000 7f39129de700 10 monclient(hunting): _renew_subs 2023-09-30T01:36:35.486 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:35.473+0000 7f39129de700 10 monclient(hunting): authenticate will time out at 1067.513257s 2023-09-30T01:36:35.486 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:35.473+0000 7f39121dd700 1 --2- >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7f390c132c40 0x7f390c1339f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:36:35.486 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:35.473+0000 7f39119dc700 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f390c097a40 0x7f390c132700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:36:35.486 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:35.473+0000 7f39121dd700 1 --2- >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7f390c132c40 0x7f390c1339f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3301/0 says I am v2:172.21.15.88:57286/0 (socket says 172.21.15.88:57286) 2023-09-30T01:36:35.486 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:35.473+0000 7f39119dc700 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f390c097a40 0x7f390c132700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.88:49944/0 (socket says 172.21.15.88:49944) 2023-09-30T01:36:35.486 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:35.473+0000 7f39121dd700 1 -- 172.21.15.88:0/3362620200 learned_addr learned my addr 172.21.15.88:0/3362620200 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-09-30T01:36:35.486 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:35.473+0000 7f39121dd700 10 monclient(hunting): get_auth_request con 0x7f390c132c40 auth_method 0 2023-09-30T01:36:35.486 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:35.473+0000 7f39121dd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:36:35.486 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:35.473+0000 7f39121dd700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:36:35.486 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:35.473+0000 7f39121dd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:36:35.486 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:35.473+0000 7f39119dc700 10 monclient(hunting): get_auth_request con 0x7f390c097a40 auth_method 0 2023-09-30T01:36:35.487 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:35.473+0000 7f39119dc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:36:35.487 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:35.473+0000 7f39119dc700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:36:35.487 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:35.473+0000 7f39119dc700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:36:35.487 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:35.473+0000 7f39121dd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:36:35.487 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:35.473+0000 7f39121dd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:36:35.487 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:35.473+0000 7f39121dd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:36:35.487 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:35.473+0000 7f39119dc700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:36:35.487 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:35.473+0000 7f39119dc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:36:35.487 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:35.473+0000 7f39119dc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:36:35.487 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:35.473+0000 7f39121dd700 10 monclient(hunting): handle_auth_done global_id 4593 payload 274 2023-09-30T01:36:35.487 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:35.473+0000 7f39121dd700 1 -- 172.21.15.88:0/3362620200 >> v2:172.21.15.107:3300/0 conn(0x7f390c097a40 msgr2=0x7f390c132700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:36:35.487 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:35.473+0000 7f39121dd700 1 --2- 172.21.15.88:0/3362620200 >> v2:172.21.15.107:3300/0 conn(0x7f390c097a40 0x7f390c132700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:36:35.487 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:35.473+0000 7f39121dd700 1 -- 172.21.15.88:0/3362620200 >> v1:172.21.15.107:6789/0 conn(0x7f390c1311c0 legacy=0x7f390c1332e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:36:35.488 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:35.473+0000 7f39121dd700 10 monclient: _finish_hunting 0 2023-09-30T01:36:35.488 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:35.473+0000 7f39121dd700 1 monclient: found mon.noname-g 2023-09-30T01:36:35.488 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:35.473+0000 7f39121dd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:36:35.488 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:35.473+0000 7f39121dd700 10 monclient: _send_mon_message to mon.noname-g at v2:172.21.15.107:3301/0 2023-09-30T01:36:35.488 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:35.473+0000 7f39121dd700 1 -- 172.21.15.88:0/3362620200 --> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f390c1330a0 con 0x7f390c132c40 2023-09-30T01:36:35.488 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:35.473+0000 7f39121dd700 1 --2- 172.21.15.88:0/3362620200 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7f390c132c40 0x7f390c1339f0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f3904009f20 tx=0x7f3904009840 comp rx=0 tx=0).ready entity=mon.3 client_cookie=6a07922585fb2458 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:36:35.488 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:35.473+0000 7f39109da700 1 -- 172.21.15.88:0/3362620200 <== mon.3 v2:172.21.15.107:3301/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f390400e070 con 0x7f390c132c40 2023-09-30T01:36:35.488 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:35.473+0000 7f39109da700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:36:35.488 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:35.473+0000 7f39109da700 10 monclient: got monmap 1 from mon.noname-g (according to old e1) 2023-09-30T01:36:35.488 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:35.473+0000 7f39109da700 10 monclient: dump: 2023-09-30T01:36:35.488 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:36:35.488 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:36:35.489 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:36:35.489 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:36:35.489 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:36:35.489 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:36:35.489 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:36:35.489 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:36:35.489 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:36:35.489 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:36:35.489 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:36:35.489 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:36:35.489 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:36:35.489 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:36:35.489 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:36:35.490 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:36:35.490 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:35.473+0000 7f39109da700 10 monclient: _finish_auth 0 2023-09-30T01:36:35.490 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:35.473+0000 7f39109da700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:36:35.490 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:35.473+0000 7f39109da700 1 -- 172.21.15.88:0/3362620200 <== mon.3 v2:172.21.15.107:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f390400be50 con 0x7f390c132c40 2023-09-30T01:36:35.490 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:35.473+0000 7f39129de700 5 monclient: authenticate success, global_id 4593 2023-09-30T01:36:35.490 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:35.473+0000 7f39109da700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:36:35.490 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:35.473+0000 7f39129de700 10 monclient: get_monmap_and_config success 2023-09-30T01:36:35.490 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:35.473+0000 7f39109da700 1 -- 172.21.15.88:0/3362620200 <== mon.3 v2:172.21.15.107:3301/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f39040092e0 con 0x7f390c132c40 2023-09-30T01:36:35.490 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:35.473+0000 7f39109da700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:36:35.490 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:35.473+0000 7f39129de700 4 set_mon_vals no callback set 2023-09-30T01:36:35.490 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:35.473+0000 7f39109da700 10 monclient: got monmap 1 from mon.g (according to old e1) 2023-09-30T01:36:35.490 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:35.473+0000 7f39109da700 10 monclient: dump: 2023-09-30T01:36:35.491 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:36:35.491 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:36:35.491 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:36:35.491 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:36:35.491 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:36:35.491 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:36:35.491 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:36:35.491 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:36:35.491 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:36:35.491 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:36:35.491 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:36:35.491 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:36:35.491 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:36:35.492 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:36:35.492 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:36:35.492 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:36:35.492 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:35.473+0000 7f39129de700 10 monclient: shutdown 2023-09-30T01:36:35.492 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:35.473+0000 7f39129de700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:36:35.492 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:35.473+0000 7f39129de700 1 -- 172.21.15.88:0/3362620200 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7f390c132c40 msgr2=0x7f390c1339f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:36:35.492 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:35.473+0000 7f39129de700 1 --2- 172.21.15.88:0/3362620200 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7f390c132c40 0x7f390c1339f0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f3904009f20 tx=0x7f3904009840 comp rx=0 tx=0).stop 2023-09-30T01:36:35.492 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:35.473+0000 7f39129de700 1 -- 172.21.15.88:0/3362620200 shutdown_connections 2023-09-30T01:36:35.492 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:35.473+0000 7f39129de700 1 --2- 172.21.15.88:0/3362620200 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7f390c132c40 0x7f390c1339f0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:36:35.492 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:35.473+0000 7f39129de700 1 --2- 172.21.15.88:0/3362620200 >> v2:172.21.15.107:3300/0 conn(0x7f390c097a40 0x7f390c132700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:36:35.492 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:35.473+0000 7f39129de700 1 -- 172.21.15.88:0/3362620200 >> 172.21.15.88:0/3362620200 conn(0x7f390c06e5e0 msgr2=0x7f390c06e9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:36:35.492 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:35.473+0000 7f39129de700 1 -- 172.21.15.88:0/3362620200 shutdown_connections 2023-09-30T01:36:35.493 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:35.473+0000 7f39129de700 1 -- 172.21.15.88:0/3362620200 wait complete. 2023-09-30T01:36:35.493 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:35.473+0000 7f39129de700 10 monclient: build_initial_monmap 2023-09-30T01:36:35.493 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:35.473+0000 7f39129de700 10 monclient: monmap: 2023-09-30T01:36:35.493 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2023-09-30T01:36:35.493 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:36:35.493 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 0.000000 2023-09-30T01:36:35.493 INFO:teuthology.orchestra.run.smithi088.stderr:created 0.000000 2023-09-30T01:36:35.493 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2023-09-30T01:36:35.493 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 1 2023-09-30T01:36:35.493 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.noname-a 2023-09-30T01:36:35.493 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.noname-f 2023-09-30T01:36:35.494 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.noname-b 2023-09-30T01:36:35.494 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.noname-g 2023-09-30T01:36:35.494 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.noname-c 2023-09-30T01:36:35.494 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.noname-h 2023-09-30T01:36:35.494 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.noname-d 2023-09-30T01:36:35.494 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.noname-i 2023-09-30T01:36:35.494 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.noname-e 2023-09-30T01:36:35.494 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:36:35.494 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:35.477+0000 7f39129de700 1 Processor -- start 2023-09-30T01:36:35.494 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:35.477+0000 7f39129de700 1 -- start start 2023-09-30T01:36:35.494 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:35.477+0000 7f39129de700 10 monclient: init 2023-09-30T01:36:35.494 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:35.477+0000 7f39129de700 10 monclient: _reopen_session rank -1 2023-09-30T01:36:35.495 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:35.477+0000 7f39129de700 10 monclient: _add_conns ranks=[0,8,1,6,5,3,4,2,7] 2023-09-30T01:36:35.495 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:35.477+0000 7f39129de700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f390c097a40 0x7f390c11cd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:36:35.495 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:35.477+0000 7f39129de700 10 monclient(hunting): picked mon.noname-a con 0x7f390c097a40 addr [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] 2023-09-30T01:36:35.495 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:35.477+0000 7f39129de700 1 --2- >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f390c1311c0 0x7f390c11d290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:36:35.495 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:35.477+0000 7f39129de700 10 monclient(hunting): picked mon.noname-e con 0x7f390c1311c0 addr [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] 2023-09-30T01:36:35.495 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:35.477+0000 7f39129de700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f390c132c40 0x7f390c11d7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:36:35.495 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:35.477+0000 7f39129de700 10 monclient(hunting): picked mon.noname-f con 0x7f390c132c40 addr [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] 2023-09-30T01:36:35.495 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:35.477+0000 7f39129de700 10 monclient(hunting): start opening mon connection 2023-09-30T01:36:35.495 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:35.477+0000 7f39129de700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f390c11f020 con 0x7f390c097a40 2023-09-30T01:36:35.495 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:35.477+0000 7f39129de700 10 monclient(hunting): start opening mon connection 2023-09-30T01:36:35.495 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:35.477+0000 7f39129de700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f390c11f650 con 0x7f390c132c40 2023-09-30T01:36:35.495 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:35.477+0000 7f39129de700 10 monclient(hunting): start opening mon connection 2023-09-30T01:36:35.496 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:35.477+0000 7f39129de700 1 -- --> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] -- mon_getmap magic: 0 v1 -- 0x7f390c0a2fa0 con 0x7f390c1311c0 2023-09-30T01:36:35.496 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:35.477+0000 7f39129de700 10 monclient(hunting): _renew_subs 2023-09-30T01:36:35.496 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:35.477+0000 7f39129de700 10 monclient(hunting): authenticate will time out at 1067.516898s 2023-09-30T01:36:35.496 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:35.477+0000 7f39119dc700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f390c097a40 0x7f390c11cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:36:35.496 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:35.477+0000 7f39111db700 1 --2- >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f390c1311c0 0x7f390c11d290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:36:35.496 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:35.477+0000 7f39119dc700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f390c097a40 0x7f390c11cd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:49604/0 (socket says 172.21.15.88:49604) 2023-09-30T01:36:35.496 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:35.477+0000 7f39121dd700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f390c132c40 0x7f390c11d7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:36:35.496 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:35.477+0000 7f39119dc700 1 -- 172.21.15.88:0/1924317475 learned_addr learned my addr 172.21.15.88:0/1924317475 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-09-30T01:36:35.496 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:35.477+0000 7f39119dc700 10 monclient(hunting): get_auth_request con 0x7f390c097a40 auth_method 0 2023-09-30T01:36:35.496 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:35.477+0000 7f39119dc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:36:35.496 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:35.477+0000 7f39119dc700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:36:35.496 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:35.477+0000 7f39119dc700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:36:35.496 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:35.477+0000 7f39111db700 10 monclient(hunting): get_auth_request con 0x7f390c1311c0 auth_method 0 2023-09-30T01:36:35.497 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:35.477+0000 7f39111db700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:36:35.497 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:35.477+0000 7f39111db700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:36:35.497 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:35.477+0000 7f39111db700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:36:35.497 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:35.477+0000 7f39121dd700 10 monclient(hunting): get_auth_request con 0x7f390c132c40 auth_method 0 2023-09-30T01:36:35.497 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:35.477+0000 7f39121dd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:36:35.497 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:35.477+0000 7f39121dd700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:36:35.497 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:35.477+0000 7f39121dd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:36:35.497 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:35.477+0000 7f39119dc700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:36:35.497 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:35.477+0000 7f39119dc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:36:35.497 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:35.477+0000 7f39119dc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:36:35.497 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:35.477+0000 7f39111db700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:36:35.497 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:35.477+0000 7f39111db700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:36:35.498 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:35.477+0000 7f39111db700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:36:35.498 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:35.477+0000 7f39121dd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:36:35.498 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:35.477+0000 7f39121dd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:36:35.498 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:35.477+0000 7f39121dd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:36:35.498 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:35.477+0000 7f39111db700 10 monclient(hunting): handle_auth_done global_id 4733 payload 824 2023-09-30T01:36:35.498 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:35.477+0000 7f39111db700 1 -- 172.21.15.88:0/1924317475 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f390c132c40 msgr2=0x7f390c11d7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:36:35.498 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:35.477+0000 7f39111db700 1 --2- 172.21.15.88:0/1924317475 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f390c132c40 0x7f390c11d7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:36:35.498 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:35.477+0000 7f39111db700 1 -- 172.21.15.88:0/1924317475 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f390c097a40 msgr2=0x7f390c11cd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:36:35.498 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:35.477+0000 7f39111db700 1 --2- 172.21.15.88:0/1924317475 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f390c097a40 0x7f390c11cd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:36:35.498 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:35.477+0000 7f39111db700 10 monclient: _finish_hunting 0 2023-09-30T01:36:35.498 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:35.477+0000 7f39111db700 1 monclient: found mon.noname-e 2023-09-30T01:36:35.498 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:35.477+0000 7f39111db700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:36:35.499 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:35.477+0000 7f39111db700 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.88:3304/0 2023-09-30T01:36:35.499 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:35.477+0000 7f39111db700 1 -- 172.21.15.88:0/1924317475 --> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3904009ae0 con 0x7f390c1311c0 2023-09-30T01:36:35.499 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:35.477+0000 7f39121dd700 1 --2- 172.21.15.88:0/1924317475 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f390c132c40 0x7f390c11d7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:36:35.499 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:35.477+0000 7f39119dc700 1 --2- 172.21.15.88:0/1924317475 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f390c097a40 0x7f390c11cd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:36:35.499 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:35.477+0000 7f39111db700 1 --2- 172.21.15.88:0/1924317475 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f390c1311c0 0x7f390c11d290 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f390000c5f0 tx=0x7f3900007e80 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:36:35.499 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:35.477+0000 7f3909ffb700 1 -- 172.21.15.88:0/1924317475 <== mon.8 v2:172.21.15.88:3304/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f3900011a10 con 0x7f390c1311c0 2023-09-30T01:36:35.499 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:35.477+0000 7f3909ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:36:35.499 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:35.477+0000 7f3909ffb700 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2023-09-30T01:36:35.499 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:35.477+0000 7f3909ffb700 10 monclient: dump: 2023-09-30T01:36:35.499 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:36:35.499 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:36:35.499 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:36:35.499 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:36:35.500 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:36:35.500 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:36:35.500 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:36:35.500 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:36:35.500 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:36:35.500 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:36:35.500 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:36:35.500 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:36:35.500 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:36:35.500 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:36:35.500 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:36:35.500 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:36:35.500 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:35.477+0000 7f3909ffb700 10 monclient: _finish_auth 0 2023-09-30T01:36:35.501 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:35.477+0000 7f3909ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:36:35.501 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:35.477+0000 7f39129de700 5 monclient: authenticate success, global_id 4733 2023-09-30T01:36:35.501 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:35.477+0000 7f3909ffb700 1 -- 172.21.15.88:0/1924317475 <== mon.8 v2:172.21.15.88:3304/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3900011bb0 con 0x7f390c1311c0 2023-09-30T01:36:35.501 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:35.477+0000 7f3909ffb700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:36:35.501 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:35.477+0000 7f3909ffb700 1 -- 172.21.15.88:0/1924317475 <== mon.8 v2:172.21.15.88:3304/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f3900011d60 con 0x7f390c1311c0 2023-09-30T01:36:35.501 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:35.477+0000 7f39129de700 10 monclient: _renew_subs 2023-09-30T01:36:35.501 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:35.477+0000 7f39129de700 10 monclient: _send_mon_message to mon.e at v2:172.21.15.88:3304/0 2023-09-30T01:36:35.501 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:35.477+0000 7f39129de700 1 -- 172.21.15.88:0/1924317475 --> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f390c099c30 con 0x7f390c1311c0 2023-09-30T01:36:35.501 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:35.477+0000 7f390b7fe700 4 set_mon_vals no callback set 2023-09-30T01:36:35.501 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:35.477+0000 7f3909ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:36:35.501 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:35.477+0000 7f3909ffb700 10 monclient: got monmap 1 from mon.e (according to old e1) 2023-09-30T01:36:35.501 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:35.477+0000 7f3909ffb700 10 monclient: dump: 2023-09-30T01:36:35.501 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:36:35.502 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:36:35.502 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:36:35.502 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:36:35.502 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:36:35.502 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:36:35.502 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:36:35.502 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:36:35.502 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:36:35.502 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:36:35.502 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:36:35.502 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:36:35.502 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:36:35.502 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:36:35.503 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:36:35.503 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:36:35.503 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:35.477+0000 7f39129de700 10 monclient: _renew_subs 2023-09-30T01:36:35.503 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:35.477+0000 7f39129de700 10 monclient: _send_mon_message to mon.e at v2:172.21.15.88:3304/0 2023-09-30T01:36:35.503 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:35.477+0000 7f39129de700 1 -- 172.21.15.88:0/1924317475 --> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f390c11e0c0 con 0x7f390c1311c0 2023-09-30T01:36:35.503 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:35.477+0000 7f3909ffb700 1 -- 172.21.15.88:0/1924317475 <== mon.8 v2:172.21.15.88:3304/0 4 ==== mgrmap(e 5) v1 ==== 40144+0+0 (secure 0 0 0) 0x7f3900011590 con 0x7f390c1311c0 2023-09-30T01:36:35.503 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:35.477+0000 7f39129de700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:36:35.503 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:35.477+0000 7f39129de700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:36:35.503 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:35.477+0000 7f39129de700 10 monclient: _send_mon_message to mon.e at v2:172.21.15.88:3304/0 2023-09-30T01:36:35.503 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:35.477+0000 7f39129de700 1 -- 172.21.15.88:0/1924317475 --> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f390c071960 con 0x7f390c1311c0 2023-09-30T01:36:35.503 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:35.477+0000 7f3909ffb700 1 --2- 172.21.15.88:0/1924317475 >> v2:172.21.15.107:6812/29926 conn(0x7f38f8033360 0x7f38f8035810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:36:35.503 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:35.477+0000 7f3909ffb700 1 -- 172.21.15.88:0/1924317475 <== mon.8 v2:172.21.15.88:3304/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 3502+0+0 (secure 0 0 0) 0x7f3900042740 con 0x7f390c1311c0 2023-09-30T01:36:35.504 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:35.477+0000 7f39119dc700 1 --2- 172.21.15.88:0/1924317475 >> v2:172.21.15.107:6812/29926 conn(0x7f38f8033360 0x7f38f8035810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:36:35.504 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:35.477+0000 7f39119dc700 10 monclient: get_auth_request con 0x7f38f8033360 auth_method 0 2023-09-30T01:36:35.504 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:35.481+0000 7f39119dc700 1 --2- 172.21.15.88:0/1924317475 >> v2:172.21.15.107:6812/29926 conn(0x7f38f8033360 0x7f38f8035810 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f38fc009fd0 tx=0x7f38fc0091a0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:36:35.504 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:35.485+0000 7f3909ffb700 1 -- 172.21.15.88:0/1924317475 <== mon.8 v2:172.21.15.88:3304/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f3900011840 con 0x7f390c1311c0 2023-09-30T01:36:35.504 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:35.485+0000 7f3909ffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:36:35.505 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:35.485+0000 7f3909ffb700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:36:35.685 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:35.681+0000 7f39129de700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T01:36:35.685 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:35.681+0000 7f39129de700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:36:35.685 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:35.681+0000 7f39129de700 10 monclient: _send_mon_message to mon.e at v2:172.21.15.88:3304/0 2023-09-30T01:36:35.686 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:35.681+0000 7f39129de700 1 -- 172.21.15.88:0/1924317475 --> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f390c067b60 con 0x7f390c1311c0 2023-09-30T01:36:35.686 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:35.681+0000 7f3909ffb700 1 -- 172.21.15.88:0/1924317475 <== mon.8 v2:172.21.15.88:3304/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f390001a370 con 0x7f390c1311c0 2023-09-30T01:36:35.686 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:35.681+0000 7f3909ffb700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:36:35.686 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:35.681+0000 7f3909ffb700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:36:35.686 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-09-30T01:36:35.686 INFO:teuthology.orchestra.run.smithi088.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:36:35.690 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:35.685+0000 7f39129de700 1 -- 172.21.15.88:0/1924317475 >> v2:172.21.15.107:6812/29926 conn(0x7f38f8033360 msgr2=0x7f38f8035810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:36:35.690 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:35.685+0000 7f39129de700 1 --2- 172.21.15.88:0/1924317475 >> v2:172.21.15.107:6812/29926 conn(0x7f38f8033360 0x7f38f8035810 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f38fc009fd0 tx=0x7f38fc0091a0 comp rx=0 tx=0).stop 2023-09-30T01:36:35.690 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:35.685+0000 7f39129de700 10 monclient: shutdown 2023-09-30T01:36:35.690 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:35.685+0000 7f39129de700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:36:35.690 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:35.685+0000 7f39129de700 1 -- 172.21.15.88:0/1924317475 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f390c1311c0 msgr2=0x7f390c11d290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:36:35.690 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:35.685+0000 7f39129de700 1 --2- 172.21.15.88:0/1924317475 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f390c1311c0 0x7f390c11d290 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f390000c5f0 tx=0x7f3900007e80 comp rx=0 tx=0).stop 2023-09-30T01:36:35.690 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:35.685+0000 7f39129de700 1 -- 172.21.15.88:0/1924317475 shutdown_connections 2023-09-30T01:36:35.690 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:35.685+0000 7f39129de700 1 --2- 172.21.15.88:0/1924317475 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f390c132c40 0x7f390c11d7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:36:35.690 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:35.685+0000 7f39129de700 1 --2- 172.21.15.88:0/1924317475 >> v2:172.21.15.107:6812/29926 conn(0x7f38f8033360 0x7f38f8035810 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:36:35.690 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:35.685+0000 7f39129de700 1 --2- 172.21.15.88:0/1924317475 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f390c1311c0 0x7f390c11d290 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:36:35.691 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:35.685+0000 7f39129de700 1 --2- 172.21.15.88:0/1924317475 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f390c097a40 0x7f390c11cd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:36:35.691 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:35.685+0000 7f39129de700 1 -- 172.21.15.88:0/1924317475 >> 172.21.15.88:0/1924317475 conn(0x7f390c06e5e0 msgr2=0x7f390c133aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:36:35.691 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:35.685+0000 7f39129de700 1 -- 172.21.15.88:0/1924317475 shutdown_connections 2023-09-30T01:36:35.691 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:35.685+0000 7f39129de700 1 -- 172.21.15.88:0/1924317475 wait complete. 2023-09-30T01:36:38.707 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:36:38.818 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:38.813+0000 7fc85691e700 10 monclient: get_monmap_and_config 2023-09-30T01:36:38.818 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:38.813+0000 7fc85691e700 10 monclient: build_initial_monmap 2023-09-30T01:36:38.818 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:38.813+0000 7fc85691e700 10 monclient: monmap: 2023-09-30T01:36:38.818 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2023-09-30T01:36:38.818 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:36:38.818 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:36:38.817991+0000 2023-09-30T01:36:38.818 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:36:38.817991+0000 2023-09-30T01:36:38.818 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2023-09-30T01:36:38.819 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:36:38.819 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a-legacy 2023-09-30T01:36:38.819 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.107:6789/0 mon.noname-f-legacy 2023-09-30T01:36:38.819 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.noname-b 2023-09-30T01:36:38.819 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.noname-g 2023-09-30T01:36:38.819 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.noname-c 2023-09-30T01:36:38.819 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.noname-h 2023-09-30T01:36:38.819 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.noname-d 2023-09-30T01:36:38.819 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.noname-i 2023-09-30T01:36:38.819 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.noname-e 2023-09-30T01:36:38.819 INFO:teuthology.orchestra.run.smithi088.stderr:9: v2:172.21.15.88:3300/0 mon.noname-a 2023-09-30T01:36:38.819 INFO:teuthology.orchestra.run.smithi088.stderr:10: v2:172.21.15.107:3300/0 mon.noname-f 2023-09-30T01:36:38.820 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:36:38.824 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:38.813+0000 7fc85691e700 1 Processor -- start 2023-09-30T01:36:38.824 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:38.813+0000 7fc85691e700 1 -- start start 2023-09-30T01:36:38.824 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:38.813+0000 7fc85691e700 10 monclient: init 2023-09-30T01:36:38.824 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:38.813+0000 7fc85691e700 10 monclient: _reopen_session rank -1 2023-09-30T01:36:38.825 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:38.813+0000 7fc85691e700 10 monclient: _add_conns ranks=[6,2,1,9,8,4,5,3,7,0,10] 2023-09-30T01:36:38.825 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:38.813+0000 7fc85691e700 1 --2- >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7fc8500976f0 0x7fc850098de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:36:38.825 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:38.813+0000 7fc85691e700 10 monclient(hunting): picked mon.noname-d con 0x7fc8500976f0 addr [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] 2023-09-30T01:36:38.825 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:38.813+0000 7fc85691e700 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc85009ed80 0x7fc85009f150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:36:38.825 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:38.813+0000 7fc85691e700 10 monclient(hunting): picked mon.noname-b con 0x7fc85009ed80 addr [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] 2023-09-30T01:36:38.825 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:38.813+0000 7fc85691e700 10 monclient(hunting): picked mon.noname-f-legacy con 0x7fc85009f690 addr v1:172.21.15.107:6789/0 2023-09-30T01:36:38.825 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:38.813+0000 7fc85691e700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc85004bb20 con 0x7fc85009f690 2023-09-30T01:36:38.825 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:38.813+0000 7fc85691e700 10 monclient(hunting): start opening mon connection 2023-09-30T01:36:38.825 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:38.813+0000 7fc85691e700 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc850099320 con 0x7fc85009ed80 2023-09-30T01:36:38.825 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:38.813+0000 7fc85691e700 10 monclient(hunting): start opening mon connection 2023-09-30T01:36:38.825 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:38.813+0000 7fc85691e700 1 -- --> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] -- mon_getmap magic: 0 v1 -- 0x7fc850098400 con 0x7fc8500976f0 2023-09-30T01:36:38.826 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:38.813+0000 7fc85691e700 10 monclient(hunting): _renew_subs 2023-09-30T01:36:38.826 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:38.813+0000 7fc85691e700 10 monclient(hunting): authenticate will time out at 1070.855559s 2023-09-30T01:36:38.826 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:38.813+0000 7fc85511b700 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc85009ed80 0x7fc85009f150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:36:38.826 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:38.813+0000 7fc85511b700 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc85009ed80 0x7fc85009f150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3301/0 says I am v2:172.21.15.88:46730/0 (socket says 172.21.15.88:46730) 2023-09-30T01:36:38.826 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:38.813+0000 7fc85511b700 1 -- 172.21.15.88:0/46914020 learned_addr learned my addr 172.21.15.88:0/46914020 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-09-30T01:36:38.826 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:38.813+0000 7fc85591c700 1 --2- 172.21.15.88:0/46914020 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7fc8500976f0 0x7fc850098de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:36:38.826 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:38.813+0000 7fc85511b700 10 monclient(hunting): get_auth_request con 0x7fc85009ed80 auth_method 0 2023-09-30T01:36:38.826 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:38.813+0000 7fc85511b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:36:38.826 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:38.813+0000 7fc85511b700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:36:38.826 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:38.813+0000 7fc85511b700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:36:38.826 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:38.813+0000 7fc85591c700 10 monclient(hunting): get_auth_request con 0x7fc8500976f0 auth_method 0 2023-09-30T01:36:38.826 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:38.813+0000 7fc85591c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:36:38.827 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:38.813+0000 7fc85591c700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:36:38.827 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:38.813+0000 7fc85591c700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:36:38.827 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:38.813+0000 7fc85511b700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:36:38.827 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:38.813+0000 7fc85511b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:36:38.827 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:38.817+0000 7fc85511b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:36:38.827 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:38.817+0000 7fc85591c700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:36:38.827 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:38.817+0000 7fc85591c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:36:38.827 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:38.817+0000 7fc85591c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:36:38.827 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:38.817+0000 7fc85591c700 10 monclient(hunting): handle_auth_done global_id 4614 payload 274 2023-09-30T01:36:38.827 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:38.817+0000 7fc85591c700 1 -- 172.21.15.88:0/46914020 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc85009ed80 msgr2=0x7fc85009f150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:36:38.827 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:38.817+0000 7fc85591c700 1 --2- 172.21.15.88:0/46914020 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc85009ed80 0x7fc85009f150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:36:38.827 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:38.817+0000 7fc85591c700 1 -- 172.21.15.88:0/46914020 >> v1:172.21.15.107:6789/0 conn(0x7fc85009f690 legacy=0x7fc850130570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:36:38.828 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:38.817+0000 7fc85591c700 10 monclient: _finish_hunting 0 2023-09-30T01:36:38.828 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:38.817+0000 7fc85591c700 1 monclient: found mon.noname-d 2023-09-30T01:36:38.828 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:38.817+0000 7fc85591c700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:36:38.828 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:38.817+0000 7fc85591c700 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.88:3303/0 2023-09-30T01:36:38.828 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:38.817+0000 7fc85591c700 1 -- 172.21.15.88:0/46914020 --> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc850098a30 con 0x7fc8500976f0 2023-09-30T01:36:38.828 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:38.817+0000 7fc85511b700 1 --2- 172.21.15.88:0/46914020 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc85009ed80 0x7fc85009f150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:36:38.828 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:38.817+0000 7fc85591c700 1 --2- 172.21.15.88:0/46914020 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7fc8500976f0 0x7fc850098de0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fc840009900 tx=0x7fc840009120 comp rx=0 tx=0).ready entity=mon.6 client_cookie=a0881c538dd99c7e server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:36:38.828 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:38.817+0000 7fc85491a700 1 -- 172.21.15.88:0/46914020 <== mon.6 v2:172.21.15.88:3303/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7fc840005c10 con 0x7fc8500976f0 2023-09-30T01:36:38.828 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:38.817+0000 7fc85491a700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:36:38.828 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:38.817+0000 7fc85491a700 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2023-09-30T01:36:38.828 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:38.817+0000 7fc85491a700 10 monclient: dump: 2023-09-30T01:36:38.828 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:36:38.829 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:36:38.829 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:36:38.829 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:36:38.829 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:36:38.829 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:36:38.829 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:36:38.829 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:36:38.829 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:36:38.829 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:36:38.829 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:36:38.829 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:36:38.829 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:36:38.830 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:36:38.830 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:36:38.830 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:36:38.830 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:38.817+0000 7fc85491a700 10 monclient: _finish_auth 0 2023-09-30T01:36:38.830 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:38.817+0000 7fc85491a700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:36:38.830 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:38.817+0000 7fc85691e700 5 monclient: authenticate success, global_id 4614 2023-09-30T01:36:38.830 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:38.817+0000 7fc85491a700 1 -- 172.21.15.88:0/46914020 <== mon.6 v2:172.21.15.88:3303/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc840005d70 con 0x7fc8500976f0 2023-09-30T01:36:38.830 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:38.817+0000 7fc85491a700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:36:38.830 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:38.817+0000 7fc85691e700 10 monclient: get_monmap_and_config success 2023-09-30T01:36:38.830 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:38.817+0000 7fc85691e700 4 set_mon_vals no callback set 2023-09-30T01:36:38.830 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:38.817+0000 7fc85491a700 1 -- 172.21.15.88:0/46914020 <== mon.6 v2:172.21.15.88:3303/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7fc8400064b0 con 0x7fc8500976f0 2023-09-30T01:36:38.830 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:38.817+0000 7fc85491a700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:36:38.831 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:38.817+0000 7fc85491a700 10 monclient: got monmap 1 from mon.d (according to old e1) 2023-09-30T01:36:38.831 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:38.817+0000 7fc85491a700 10 monclient: dump: 2023-09-30T01:36:38.831 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:36:38.831 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:36:38.831 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:36:38.831 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:36:38.831 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:36:38.831 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:36:38.831 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:36:38.831 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:36:38.831 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:36:38.831 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:36:38.832 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:36:38.832 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:36:38.832 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:36:38.832 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:36:38.832 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:36:38.832 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:36:38.832 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:38.817+0000 7fc85691e700 10 monclient: shutdown 2023-09-30T01:36:38.832 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:38.817+0000 7fc85691e700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:36:38.832 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:38.817+0000 7fc85691e700 1 -- 172.21.15.88:0/46914020 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7fc8500976f0 msgr2=0x7fc850098de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:36:38.832 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:38.817+0000 7fc85691e700 1 --2- 172.21.15.88:0/46914020 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7fc8500976f0 0x7fc850098de0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fc840009900 tx=0x7fc840009120 comp rx=0 tx=0).stop 2023-09-30T01:36:38.832 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:38.817+0000 7fc85691e700 1 -- 172.21.15.88:0/46914020 shutdown_connections 2023-09-30T01:36:38.832 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:38.817+0000 7fc85691e700 1 --2- 172.21.15.88:0/46914020 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc85009ed80 0x7fc85009f150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:36:38.832 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:38.817+0000 7fc85691e700 1 --2- 172.21.15.88:0/46914020 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7fc8500976f0 0x7fc850098de0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:36:38.833 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:38.817+0000 7fc85691e700 1 -- 172.21.15.88:0/46914020 >> 172.21.15.88:0/46914020 conn(0x7fc8500a0910 msgr2=0x7fc8500a33c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:36:38.833 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:38.817+0000 7fc85691e700 1 -- 172.21.15.88:0/46914020 shutdown_connections 2023-09-30T01:36:38.833 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:38.817+0000 7fc85691e700 1 -- 172.21.15.88:0/46914020 wait complete. 2023-09-30T01:36:38.833 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:38.817+0000 7fc85691e700 10 monclient: build_initial_monmap 2023-09-30T01:36:38.833 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:38.817+0000 7fc85691e700 10 monclient: monmap: 2023-09-30T01:36:38.833 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2023-09-30T01:36:38.833 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:36:38.833 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 0.000000 2023-09-30T01:36:38.833 INFO:teuthology.orchestra.run.smithi088.stderr:created 0.000000 2023-09-30T01:36:38.833 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2023-09-30T01:36:38.833 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 1 2023-09-30T01:36:38.833 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.noname-a 2023-09-30T01:36:38.834 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.noname-f 2023-09-30T01:36:38.834 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.noname-b 2023-09-30T01:36:38.834 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.noname-g 2023-09-30T01:36:38.834 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.noname-c 2023-09-30T01:36:38.834 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.noname-h 2023-09-30T01:36:38.834 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.noname-d 2023-09-30T01:36:38.834 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.noname-i 2023-09-30T01:36:38.834 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.noname-e 2023-09-30T01:36:38.834 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:36:38.834 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:38.821+0000 7fc85691e700 1 Processor -- start 2023-09-30T01:36:38.834 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:38.821+0000 7fc85691e700 1 -- start start 2023-09-30T01:36:38.834 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:38.821+0000 7fc85691e700 10 monclient: init 2023-09-30T01:36:38.839 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:38.821+0000 7fc85691e700 10 monclient: _reopen_session rank -1 2023-09-30T01:36:38.839 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:38.821+0000 7fc85691e700 10 monclient: _add_conns ranks=[0,2,4,5,7,6,3,8,1] 2023-09-30T01:36:38.839 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:38.821+0000 7fc85691e700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc8500976f0 0x7fc850096c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:36:38.839 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:38.821+0000 7fc85691e700 10 monclient(hunting): picked mon.noname-a con 0x7fc8500976f0 addr [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] 2023-09-30T01:36:38.839 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:38.821+0000 7fc85691e700 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc85009ed80 0x7fc85008b580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:36:38.839 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:38.821+0000 7fc85691e700 10 monclient(hunting): picked mon.noname-b con 0x7fc85009ed80 addr [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] 2023-09-30T01:36:38.839 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:38.821+0000 7fc85691e700 1 --2- >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7fc85009f690 0x7fc85008bac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:36:38.839 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:38.821+0000 7fc85691e700 10 monclient(hunting): picked mon.noname-c con 0x7fc85009f690 addr [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] 2023-09-30T01:36:38.839 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:38.821+0000 7fc85691e700 10 monclient(hunting): start opening mon connection 2023-09-30T01:36:38.839 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:38.821+0000 7fc85691e700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc85008eed0 con 0x7fc8500976f0 2023-09-30T01:36:38.840 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:38.821+0000 7fc85691e700 10 monclient(hunting): start opening mon connection 2023-09-30T01:36:38.840 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:38.821+0000 7fc85691e700 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc85008d640 con 0x7fc85009ed80 2023-09-30T01:36:38.840 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:38.821+0000 7fc85691e700 10 monclient(hunting): start opening mon connection 2023-09-30T01:36:38.840 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:38.821+0000 7fc85691e700 1 -- --> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] -- mon_getmap magic: 0 v1 -- 0x7fc85008ebf0 con 0x7fc85009f690 2023-09-30T01:36:38.840 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:38.821+0000 7fc85691e700 10 monclient(hunting): _renew_subs 2023-09-30T01:36:38.840 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:38.821+0000 7fc85691e700 10 monclient(hunting): authenticate will time out at 1070.861871s 2023-09-30T01:36:38.840 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:38.821+0000 7fc85591c700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc8500976f0 0x7fc850096c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:36:38.840 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:38.821+0000 7fc85511b700 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc85009ed80 0x7fc85008b580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:36:38.840 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:38.821+0000 7fc85511b700 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc85009ed80 0x7fc85008b580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3301/0 says I am v2:172.21.15.88:46740/0 (socket says 172.21.15.88:46740) 2023-09-30T01:36:38.840 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:38.821+0000 7fc85611d700 1 --2- >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7fc85009f690 0x7fc85008bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:36:38.840 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:38.821+0000 7fc85511b700 1 -- 172.21.15.88:0/852800833 learned_addr learned my addr 172.21.15.88:0/852800833 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-09-30T01:36:38.840 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:38.821+0000 7fc85511b700 10 monclient(hunting): get_auth_request con 0x7fc85009ed80 auth_method 0 2023-09-30T01:36:38.841 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:38.821+0000 7fc85511b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:36:38.841 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:38.821+0000 7fc85591c700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc8500976f0 0x7fc850096c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:49618/0 (socket says 172.21.15.88:49618) 2023-09-30T01:36:38.841 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:38.821+0000 7fc85511b700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:36:38.841 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:38.821+0000 7fc85511b700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:36:38.841 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:38.821+0000 7fc85591c700 10 monclient(hunting): get_auth_request con 0x7fc8500976f0 auth_method 0 2023-09-30T01:36:38.841 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:38.821+0000 7fc85591c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:36:38.841 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:38.821+0000 7fc85591c700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:36:38.841 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:38.821+0000 7fc85591c700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:36:38.841 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:38.821+0000 7fc85611d700 10 monclient(hunting): get_auth_request con 0x7fc85009f690 auth_method 0 2023-09-30T01:36:38.841 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:38.821+0000 7fc85611d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:36:38.841 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:38.821+0000 7fc85611d700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:36:38.841 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:38.821+0000 7fc85611d700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:36:38.842 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:38.821+0000 7fc85511b700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:36:38.842 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:38.821+0000 7fc85511b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:36:38.842 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:38.821+0000 7fc85511b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:36:38.842 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:38.821+0000 7fc85591c700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:36:38.842 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:38.821+0000 7fc85591c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:36:38.842 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:38.821+0000 7fc85591c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:36:38.842 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:38.821+0000 7fc85611d700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:36:38.842 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:38.821+0000 7fc85611d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:36:38.842 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:38.821+0000 7fc85611d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:36:38.842 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:38.821+0000 7fc85511b700 10 monclient(hunting): handle_auth_done global_id 4682 payload 824 2023-09-30T01:36:38.842 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:38.821+0000 7fc85511b700 1 -- 172.21.15.88:0/852800833 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7fc85009f690 msgr2=0x7fc85008bac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:36:38.842 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:38.821+0000 7fc85511b700 1 --2- 172.21.15.88:0/852800833 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7fc85009f690 0x7fc85008bac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:36:38.843 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:38.821+0000 7fc85511b700 1 -- 172.21.15.88:0/852800833 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc8500976f0 msgr2=0x7fc850096c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:36:38.843 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:38.821+0000 7fc85511b700 1 --2- 172.21.15.88:0/852800833 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc8500976f0 0x7fc850096c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:36:38.843 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:38.821+0000 7fc85511b700 10 monclient: _finish_hunting 0 2023-09-30T01:36:38.843 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:38.821+0000 7fc85511b700 1 monclient: found mon.noname-b 2023-09-30T01:36:38.843 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:38.821+0000 7fc85511b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:36:38.843 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:38.821+0000 7fc85511b700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.88:3301/0 2023-09-30T01:36:38.843 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:38.821+0000 7fc85511b700 1 -- 172.21.15.88:0/852800833 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc84c0094f0 con 0x7fc85009ed80 2023-09-30T01:36:38.843 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:38.821+0000 7fc85511b700 1 --2- 172.21.15.88:0/852800833 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc85009ed80 0x7fc85008b580 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fc84c00ec50 tx=0x7fc84c009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:36:38.843 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:38.821+0000 7fc845ffb700 1 -- 172.21.15.88:0/852800833 <== mon.2 v2:172.21.15.88:3301/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7fc84c0097d0 con 0x7fc85009ed80 2023-09-30T01:36:38.843 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:38.821+0000 7fc845ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:36:38.843 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:38.821+0000 7fc845ffb700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T01:36:38.843 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:38.821+0000 7fc845ffb700 10 monclient: dump: 2023-09-30T01:36:38.844 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:36:38.844 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:36:38.844 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:36:38.844 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:36:38.844 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:36:38.844 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:36:38.844 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:36:38.844 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:36:38.844 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:36:38.844 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:36:38.844 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:36:38.844 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:36:38.845 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:36:38.845 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:36:38.845 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:36:38.845 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:36:38.845 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:38.821+0000 7fc845ffb700 10 monclient: _finish_auth 0 2023-09-30T01:36:38.845 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:38.821+0000 7fc845ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:36:38.845 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:38.821+0000 7fc85691e700 5 monclient: authenticate success, global_id 4682 2023-09-30T01:36:38.845 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:38.821+0000 7fc845ffb700 1 -- 172.21.15.88:0/852800833 <== mon.2 v2:172.21.15.88:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc84c00ccd0 con 0x7fc85009ed80 2023-09-30T01:36:38.845 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:38.821+0000 7fc85691e700 10 monclient: _renew_subs 2023-09-30T01:36:38.845 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:38.821+0000 7fc85691e700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.88:3301/0 2023-09-30T01:36:38.845 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:38.821+0000 7fc85691e700 1 -- 172.21.15.88:0/852800833 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc8400093c0 con 0x7fc85009ed80 2023-09-30T01:36:38.845 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:38.821+0000 7fc845ffb700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:36:38.846 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:38.821+0000 7fc845ffb700 1 -- 172.21.15.88:0/852800833 <== mon.2 v2:172.21.15.88:3301/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7fc84c00ce40 con 0x7fc85009ed80 2023-09-30T01:36:38.846 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:38.821+0000 7fc845ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:36:38.846 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:38.821+0000 7fc8477fe700 4 set_mon_vals no callback set 2023-09-30T01:36:38.846 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:38.821+0000 7fc845ffb700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T01:36:38.846 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:38.821+0000 7fc845ffb700 10 monclient: dump: 2023-09-30T01:36:38.846 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:36:38.846 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:36:38.846 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:36:38.846 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:36:38.846 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:36:38.846 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:36:38.846 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:36:38.847 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:36:38.847 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:36:38.847 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:36:38.847 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:36:38.847 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:36:38.847 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:36:38.847 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:36:38.847 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:36:38.847 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:36:38.847 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:38.821+0000 7fc85691e700 10 monclient: _renew_subs 2023-09-30T01:36:38.847 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:38.821+0000 7fc85691e700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.88:3301/0 2023-09-30T01:36:38.847 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:38.821+0000 7fc85691e700 1 -- 172.21.15.88:0/852800833 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc85008f8b0 con 0x7fc85009ed80 2023-09-30T01:36:38.847 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:38.825+0000 7fc85691e700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:36:38.848 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:38.825+0000 7fc85691e700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:36:38.848 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:38.825+0000 7fc85691e700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.88:3301/0 2023-09-30T01:36:38.848 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:38.825+0000 7fc85691e700 1 -- 172.21.15.88:0/852800833 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc850071960 con 0x7fc85009ed80 2023-09-30T01:36:38.848 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:38.825+0000 7fc845ffb700 1 -- 172.21.15.88:0/852800833 <== mon.2 v2:172.21.15.88:3301/0 4 ==== mgrmap(e 5) v1 ==== 40144+0+0 (secure 0 0 0) 0x7fc84c00c850 con 0x7fc85009ed80 2023-09-30T01:36:38.848 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:38.825+0000 7fc845ffb700 1 --2- 172.21.15.88:0/852800833 >> v2:172.21.15.107:6812/29926 conn(0x7fc834033360 0x7fc834035810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:36:38.848 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:38.825+0000 7fc845ffb700 1 -- 172.21.15.88:0/852800833 <== mon.2 v2:172.21.15.88:3301/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 3502+0+0 (secure 0 0 0) 0x7fc84c043800 con 0x7fc85009ed80 2023-09-30T01:36:38.848 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:38.825+0000 7fc85591c700 1 --2- 172.21.15.88:0/852800833 >> v2:172.21.15.107:6812/29926 conn(0x7fc834033360 0x7fc834035810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:36:38.848 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:38.825+0000 7fc85591c700 10 monclient: get_auth_request con 0x7fc834033360 auth_method 0 2023-09-30T01:36:38.848 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:38.825+0000 7fc85591c700 1 --2- 172.21.15.88:0/852800833 >> v2:172.21.15.107:6812/29926 conn(0x7fc834033360 0x7fc834035810 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fc840009900 tx=0x7fc840016000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:36:38.848 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:38.833+0000 7fc845ffb700 1 -- 172.21.15.88:0/852800833 <== mon.2 v2:172.21.15.88:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fc84c0192c0 con 0x7fc85009ed80 2023-09-30T01:36:38.848 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:38.833+0000 7fc845ffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:36:38.848 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:38.833+0000 7fc845ffb700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:36:39.027 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:39.021+0000 7fc85691e700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T01:36:39.027 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:39.021+0000 7fc85691e700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:36:39.027 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:39.021+0000 7fc85691e700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.88:3301/0 2023-09-30T01:36:39.027 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:39.021+0000 7fc85691e700 1 -- 172.21.15.88:0/852800833 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fc850067af0 con 0x7fc85009ed80 2023-09-30T01:36:39.027 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:39.021+0000 7fc845ffb700 1 -- 172.21.15.88:0/852800833 <== mon.2 v2:172.21.15.88:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7fc84c01bba0 con 0x7fc85009ed80 2023-09-30T01:36:39.027 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:39.021+0000 7fc845ffb700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:36:39.027 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:39.021+0000 7fc845ffb700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:36:39.028 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-09-30T01:36:39.028 INFO:teuthology.orchestra.run.smithi088.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:36:39.031 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:39.025+0000 7fc85691e700 1 -- 172.21.15.88:0/852800833 >> v2:172.21.15.107:6812/29926 conn(0x7fc834033360 msgr2=0x7fc834035810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:36:39.031 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:39.025+0000 7fc85691e700 1 --2- 172.21.15.88:0/852800833 >> v2:172.21.15.107:6812/29926 conn(0x7fc834033360 0x7fc834035810 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fc840009900 tx=0x7fc840016000 comp rx=0 tx=0).stop 2023-09-30T01:36:39.031 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:39.025+0000 7fc85691e700 10 monclient: shutdown 2023-09-30T01:36:39.031 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:39.025+0000 7fc85691e700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:36:39.032 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:39.025+0000 7fc85691e700 1 -- 172.21.15.88:0/852800833 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc85009ed80 msgr2=0x7fc85008b580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:36:39.032 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:39.025+0000 7fc85691e700 1 --2- 172.21.15.88:0/852800833 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc85009ed80 0x7fc85008b580 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fc84c00ec50 tx=0x7fc84c009250 comp rx=0 tx=0).stop 2023-09-30T01:36:39.032 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:39.025+0000 7fc85691e700 1 -- 172.21.15.88:0/852800833 shutdown_connections 2023-09-30T01:36:39.032 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:39.025+0000 7fc85691e700 1 --2- 172.21.15.88:0/852800833 >> v2:172.21.15.107:6812/29926 conn(0x7fc834033360 0x7fc834035810 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:36:39.032 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:39.025+0000 7fc85691e700 1 --2- 172.21.15.88:0/852800833 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7fc85009f690 0x7fc85008bac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:36:39.032 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:39.025+0000 7fc85691e700 1 --2- 172.21.15.88:0/852800833 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc85009ed80 0x7fc85008b580 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:36:39.032 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:39.025+0000 7fc85691e700 1 --2- 172.21.15.88:0/852800833 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc8500976f0 0x7fc850096c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:36:39.032 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:39.025+0000 7fc85691e700 1 -- 172.21.15.88:0/852800833 >> 172.21.15.88:0/852800833 conn(0x7fc8500a0910 msgr2=0x7fc8500a2e60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:36:39.032 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:39.025+0000 7fc85691e700 1 -- 172.21.15.88:0/852800833 shutdown_connections 2023-09-30T01:36:39.032 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:39.025+0000 7fc85691e700 1 -- 172.21.15.88:0/852800833 wait complete. 2023-09-30T01:36:42.046 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:36:42.154 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:42.149+0000 7f7855a95700 10 monclient: get_monmap_and_config 2023-09-30T01:36:42.154 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:42.149+0000 7f7855a95700 10 monclient: build_initial_monmap 2023-09-30T01:36:42.154 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:42.149+0000 7f7855a95700 10 monclient: monmap: 2023-09-30T01:36:42.155 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2023-09-30T01:36:42.155 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:36:42.155 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:36:42.154384+0000 2023-09-30T01:36:42.155 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:36:42.154384+0000 2023-09-30T01:36:42.155 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2023-09-30T01:36:42.155 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:36:42.155 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a-legacy 2023-09-30T01:36:42.155 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.107:6789/0 mon.noname-f-legacy 2023-09-30T01:36:42.155 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.noname-b 2023-09-30T01:36:42.155 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.noname-g 2023-09-30T01:36:42.155 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.noname-c 2023-09-30T01:36:42.156 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.noname-h 2023-09-30T01:36:42.156 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.noname-d 2023-09-30T01:36:42.156 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.noname-i 2023-09-30T01:36:42.156 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.noname-e 2023-09-30T01:36:42.156 INFO:teuthology.orchestra.run.smithi088.stderr:9: v2:172.21.15.88:3300/0 mon.noname-a 2023-09-30T01:36:42.156 INFO:teuthology.orchestra.run.smithi088.stderr:10: v2:172.21.15.107:3300/0 mon.noname-f 2023-09-30T01:36:42.156 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:36:42.160 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:42.149+0000 7f7855a95700 1 Processor -- start 2023-09-30T01:36:42.160 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:42.153+0000 7f7855a95700 1 -- start start 2023-09-30T01:36:42.161 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:42.153+0000 7f7855a95700 10 monclient: init 2023-09-30T01:36:42.161 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:42.153+0000 7f7855a95700 10 monclient: _reopen_session rank -1 2023-09-30T01:36:42.161 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:42.153+0000 7f7855a95700 10 monclient: _add_conns ranks=[4,0,9,1,5,2,10,6,7,8,3] 2023-09-30T01:36:42.161 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:42.153+0000 7f7855a95700 1 --2- >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f7850133740 0x7f7850069a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:36:42.161 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:42.153+0000 7f7855a95700 10 monclient(hunting): picked mon.noname-c con 0x7f7850133740 addr [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] 2023-09-30T01:36:42.161 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:42.153+0000 7f7855a95700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f7850133b10 addr v1:172.21.15.88:6789/0 2023-09-30T01:36:42.161 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:42.153+0000 7f7855a95700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f7850133ee0 0x7f785009f580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:36:42.161 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:42.153+0000 7f7855a95700 10 monclient(hunting): picked mon.noname-a con 0x7f7850133ee0 addr v2:172.21.15.88:3300/0 2023-09-30T01:36:42.161 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:42.153+0000 7f7855a95700 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f785004bb20 con 0x7f7850133b10 2023-09-30T01:36:42.161 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:42.153+0000 7f7855a95700 10 monclient(hunting): start opening mon connection 2023-09-30T01:36:42.162 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:42.153+0000 7f7855a95700 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7850068790 con 0x7f7850133ee0 2023-09-30T01:36:42.162 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:42.153+0000 7f7855a95700 10 monclient(hunting): start opening mon connection 2023-09-30T01:36:42.162 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:42.153+0000 7f7855a95700 1 -- --> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] -- mon_getmap magic: 0 v1 -- 0x7f7850133380 con 0x7f7850133740 2023-09-30T01:36:42.162 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:42.153+0000 7f7855a95700 10 monclient(hunting): _renew_subs 2023-09-30T01:36:42.162 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:42.153+0000 7f7855a95700 10 monclient(hunting): authenticate will time out at 1074.192686s 2023-09-30T01:36:42.162 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:42.153+0000 7f7854a93700 1 --2- >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f7850133740 0x7f7850069a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:36:42.162 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:42.153+0000 7f7855294700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f7850133ee0 0x7f785009f580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:36:42.163 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:42.153+0000 7f7854a93700 1 --2- >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f7850133740 0x7f7850069a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3302/0 says I am v2:172.21.15.88:36790/0 (socket says 172.21.15.88:36790) 2023-09-30T01:36:42.163 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:42.153+0000 7f7854a93700 1 -- 172.21.15.88:0/3895208324 learned_addr learned my addr 172.21.15.88:0/3895208324 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-09-30T01:36:42.163 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:42.153+0000 7f7855294700 10 monclient(hunting): get_auth_request con 0x7f7850133ee0 auth_method 0 2023-09-30T01:36:42.163 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:42.153+0000 7f7855294700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:36:42.163 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:42.153+0000 7f7855294700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:36:42.163 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:42.153+0000 7f7855294700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:36:42.163 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:42.153+0000 7f7854a93700 10 monclient(hunting): get_auth_request con 0x7f7850133740 auth_method 0 2023-09-30T01:36:42.163 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:42.153+0000 7f7854a93700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:36:42.163 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:42.153+0000 7f7854a93700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:36:42.163 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:42.153+0000 7f7854a93700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:36:42.164 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:42.153+0000 7f7854a93700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:36:42.164 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:42.153+0000 7f7854a93700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:36:42.164 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:42.153+0000 7f7854a93700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:36:42.164 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:42.153+0000 7f7855294700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:36:42.164 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:42.153+0000 7f7855294700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:36:42.164 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:42.153+0000 7f7855294700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:36:42.164 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:42.153+0000 7f7854a93700 10 monclient(hunting): handle_auth_done global_id 4729 payload 274 2023-09-30T01:36:42.164 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:42.153+0000 7f7854a93700 1 -- 172.21.15.88:0/3895208324 >> v2:172.21.15.88:3300/0 conn(0x7f7850133ee0 msgr2=0x7f785009f580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:36:42.164 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:42.153+0000 7f7854a93700 1 --2- 172.21.15.88:0/3895208324 >> v2:172.21.15.88:3300/0 conn(0x7f7850133ee0 0x7f785009f580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:36:42.164 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:42.153+0000 7f7854a93700 1 -- 172.21.15.88:0/3895208324 >> v1:172.21.15.88:6789/0 conn(0x7f7850133b10 legacy=0x7f785009ee70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:36:42.164 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:42.153+0000 7f7854a93700 10 monclient: _finish_hunting 0 2023-09-30T01:36:42.164 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:42.153+0000 7f7854a93700 1 monclient: found mon.noname-c 2023-09-30T01:36:42.165 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:42.153+0000 7f7854a93700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:36:42.165 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:42.153+0000 7f7854a93700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.88:3302/0 2023-09-30T01:36:42.165 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:42.153+0000 7f7854a93700 1 -- 172.21.15.88:0/3895208324 --> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7850068990 con 0x7f7850133740 2023-09-30T01:36:42.165 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:42.153+0000 7f7854a93700 1 --2- 172.21.15.88:0/3895208324 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f7850133740 0x7f7850069a10 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f78400099c0 tx=0x7f7840009220 comp rx=0 tx=0).ready entity=mon.4 client_cookie=be5fafff7ebb6ed2 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:36:42.165 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:42.153+0000 7f784f7fe700 1 -- 172.21.15.88:0/3895208324 <== mon.4 v2:172.21.15.88:3302/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f7840005c10 con 0x7f7850133740 2023-09-30T01:36:42.165 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:42.153+0000 7f784f7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:36:42.165 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:42.153+0000 7f784f7fe700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T01:36:42.165 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:42.153+0000 7f784f7fe700 10 monclient: dump: 2023-09-30T01:36:42.166 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:36:42.166 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:36:42.166 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:36:42.166 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:36:42.166 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:36:42.166 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:36:42.166 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:36:42.166 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:36:42.166 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:36:42.166 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:36:42.166 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:36:42.167 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:36:42.167 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:36:42.167 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:36:42.167 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:36:42.167 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:36:42.167 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:42.153+0000 7f784f7fe700 10 monclient: _finish_auth 0 2023-09-30T01:36:42.167 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:42.153+0000 7f784f7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:36:42.167 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:42.153+0000 7f784f7fe700 1 -- 172.21.15.88:0/3895208324 <== mon.4 v2:172.21.15.88:3302/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7840005d90 con 0x7f7850133740 2023-09-30T01:36:42.167 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:42.153+0000 7f7855a95700 5 monclient: authenticate success, global_id 4729 2023-09-30T01:36:42.167 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:42.153+0000 7f7855a95700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:36:42.167 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:42.153+0000 7f784f7fe700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:36:42.168 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:42.153+0000 7f7855a95700 10 monclient: get_monmap_and_config success 2023-09-30T01:36:42.168 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:42.153+0000 7f784f7fe700 1 -- 172.21.15.88:0/3895208324 <== mon.4 v2:172.21.15.88:3302/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f7840006030 con 0x7f7850133740 2023-09-30T01:36:42.168 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:42.153+0000 7f784f7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:36:42.168 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:42.153+0000 7f7855a95700 4 set_mon_vals no callback set 2023-09-30T01:36:42.168 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:42.153+0000 7f784f7fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T01:36:42.168 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:42.153+0000 7f784f7fe700 10 monclient: dump: 2023-09-30T01:36:42.168 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:36:42.168 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:36:42.168 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:36:42.168 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:36:42.168 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:36:42.168 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:36:42.169 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:36:42.169 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:36:42.169 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:36:42.169 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:36:42.169 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:36:42.169 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:36:42.169 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:36:42.169 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:36:42.169 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:36:42.169 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:36:42.169 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:42.153+0000 7f7855a95700 10 monclient: shutdown 2023-09-30T01:36:42.169 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:42.153+0000 7f7855a95700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:36:42.170 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:42.153+0000 7f7855a95700 1 -- 172.21.15.88:0/3895208324 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f7850133740 msgr2=0x7f7850069a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:36:42.170 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:42.153+0000 7f7855a95700 1 --2- 172.21.15.88:0/3895208324 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f7850133740 0x7f7850069a10 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f78400099c0 tx=0x7f7840009220 comp rx=0 tx=0).stop 2023-09-30T01:36:42.170 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:42.153+0000 7f7855a95700 1 -- 172.21.15.88:0/3895208324 shutdown_connections 2023-09-30T01:36:42.170 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:42.153+0000 7f7855a95700 1 --2- 172.21.15.88:0/3895208324 >> v2:172.21.15.88:3300/0 conn(0x7f7850133ee0 0x7f785009f580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:36:42.170 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:42.153+0000 7f7855a95700 1 --2- 172.21.15.88:0/3895208324 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f7850133740 0x7f7850069a10 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:36:42.170 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:42.153+0000 7f7855a95700 1 -- 172.21.15.88:0/3895208324 >> 172.21.15.88:0/3895208324 conn(0x7f7850097a40 msgr2=0x7f7850063450 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:36:42.170 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:42.153+0000 7f7855a95700 1 -- 172.21.15.88:0/3895208324 shutdown_connections 2023-09-30T01:36:42.170 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:42.153+0000 7f7855a95700 1 -- 172.21.15.88:0/3895208324 wait complete. 2023-09-30T01:36:42.170 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:42.153+0000 7f7855a95700 10 monclient: build_initial_monmap 2023-09-30T01:36:42.170 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:42.157+0000 7f7855a95700 10 monclient: monmap: 2023-09-30T01:36:42.170 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2023-09-30T01:36:42.171 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:36:42.171 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 0.000000 2023-09-30T01:36:42.171 INFO:teuthology.orchestra.run.smithi088.stderr:created 0.000000 2023-09-30T01:36:42.171 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2023-09-30T01:36:42.171 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 1 2023-09-30T01:36:42.171 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.noname-a 2023-09-30T01:36:42.171 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.noname-f 2023-09-30T01:36:42.171 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.noname-b 2023-09-30T01:36:42.171 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.noname-g 2023-09-30T01:36:42.171 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.noname-c 2023-09-30T01:36:42.171 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.noname-h 2023-09-30T01:36:42.171 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.noname-d 2023-09-30T01:36:42.171 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.noname-i 2023-09-30T01:36:42.172 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.noname-e 2023-09-30T01:36:42.172 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:36:42.172 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:42.157+0000 7f7855a95700 1 Processor -- start 2023-09-30T01:36:42.172 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:42.157+0000 7f7855a95700 1 -- start start 2023-09-30T01:36:42.172 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:42.157+0000 7f7855a95700 10 monclient: init 2023-09-30T01:36:42.177 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:42.157+0000 7f7855a95700 10 monclient: _reopen_session rank -1 2023-09-30T01:36:42.178 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:42.157+0000 7f7855a95700 10 monclient: _add_conns ranks=[3,1,7,5,4,2,0,8,6] 2023-09-30T01:36:42.178 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:42.157+0000 7f7855a95700 1 --2- >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7f7850133740 0x7f7850142970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:36:42.178 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:42.157+0000 7f7855a95700 10 monclient(hunting): picked mon.noname-g con 0x7f7850133740 addr [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] 2023-09-30T01:36:42.178 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:42.157+0000 7f7855a95700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7850133b10 0x7f7850142eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:36:42.178 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:42.157+0000 7f7855a95700 10 monclient(hunting): picked mon.noname-f con 0x7f7850133b10 addr [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] 2023-09-30T01:36:42.178 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:42.157+0000 7f7855a95700 1 --2- >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7f7850133ee0 0x7f785009cc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:36:42.178 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:42.157+0000 7f7855a95700 10 monclient(hunting): picked mon.noname-i con 0x7f7850133ee0 addr [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] 2023-09-30T01:36:42.178 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:42.157+0000 7f7855a95700 10 monclient(hunting): start opening mon connection 2023-09-30T01:36:42.178 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:42.157+0000 7f7855a95700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7850067fc0 con 0x7f7850133b10 2023-09-30T01:36:42.178 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:42.157+0000 7f7855a95700 10 monclient(hunting): start opening mon connection 2023-09-30T01:36:42.178 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:42.157+0000 7f7855a95700 1 -- --> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7850120e90 con 0x7f7850133740 2023-09-30T01:36:42.179 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:42.157+0000 7f7855a95700 10 monclient(hunting): start opening mon connection 2023-09-30T01:36:42.179 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:42.157+0000 7f7855a95700 1 -- --> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] -- mon_getmap magic: 0 v1 -- 0x7f78500a0b90 con 0x7f7850133ee0 2023-09-30T01:36:42.179 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:42.157+0000 7f7855a95700 10 monclient(hunting): _renew_subs 2023-09-30T01:36:42.179 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:42.157+0000 7f7855a95700 10 monclient(hunting): authenticate will time out at 1074.197562s 2023-09-30T01:36:42.179 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:42.157+0000 7f7855294700 1 --2- >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7f7850133ee0 0x7f785009cc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:36:42.179 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:42.157+0000 7f784ffff700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7850133b10 0x7f7850142eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:36:42.179 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:42.157+0000 7f7854a93700 1 --2- >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7f7850133740 0x7f7850142970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:36:42.179 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:42.157+0000 7f7855294700 1 --2- >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7f7850133ee0 0x7f785009cc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3303/0 says I am v2:172.21.15.88:50956/0 (socket says 172.21.15.88:50956) 2023-09-30T01:36:42.179 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:42.157+0000 7f7855294700 1 -- 172.21.15.88:0/2398185520 learned_addr learned my addr 172.21.15.88:0/2398185520 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-09-30T01:36:42.179 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:42.157+0000 7f7854a93700 10 monclient(hunting): get_auth_request con 0x7f7850133740 auth_method 0 2023-09-30T01:36:42.179 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:42.157+0000 7f7854a93700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:36:42.179 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:42.157+0000 7f7854a93700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:36:42.180 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:42.157+0000 7f7854a93700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:36:42.180 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:42.157+0000 7f7855294700 10 monclient(hunting): get_auth_request con 0x7f7850133ee0 auth_method 0 2023-09-30T01:36:42.180 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:42.157+0000 7f7855294700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:36:42.180 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:42.157+0000 7f7855294700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:36:42.180 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:42.157+0000 7f7855294700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:36:42.180 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:42.157+0000 7f784ffff700 10 monclient(hunting): get_auth_request con 0x7f7850133b10 auth_method 0 2023-09-30T01:36:42.180 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:42.157+0000 7f784ffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:36:42.180 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:42.157+0000 7f784ffff700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:36:42.180 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:42.157+0000 7f784ffff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:36:42.180 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:42.157+0000 7f784ffff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:36:42.180 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:42.157+0000 7f784ffff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:36:42.180 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:42.157+0000 7f784ffff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:36:42.181 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:42.157+0000 7f7854a93700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:36:42.181 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:42.157+0000 7f7854a93700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:36:42.181 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:42.157+0000 7f7854a93700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:36:42.181 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:42.157+0000 7f7855294700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:36:42.181 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:42.157+0000 7f7855294700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:36:42.181 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:42.157+0000 7f7855294700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:36:42.181 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:42.157+0000 7f784ffff700 10 monclient(hunting): handle_auth_done global_id 4978 payload 824 2023-09-30T01:36:42.181 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:42.157+0000 7f784ffff700 1 -- 172.21.15.88:0/2398185520 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7f7850133ee0 msgr2=0x7f785009cc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:36:42.181 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:42.157+0000 7f784ffff700 1 --2- 172.21.15.88:0/2398185520 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7f7850133ee0 0x7f785009cc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:36:42.181 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:42.157+0000 7f784ffff700 1 -- 172.21.15.88:0/2398185520 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7f7850133740 msgr2=0x7f7850142970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:36:42.181 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:42.157+0000 7f784ffff700 1 --2- 172.21.15.88:0/2398185520 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7f7850133740 0x7f7850142970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:36:42.182 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:42.157+0000 7f784ffff700 10 monclient: _finish_hunting 0 2023-09-30T01:36:42.182 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:42.157+0000 7f784ffff700 1 monclient: found mon.noname-f 2023-09-30T01:36:42.182 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:42.157+0000 7f784ffff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:36:42.182 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:42.157+0000 7f784ffff700 10 monclient: _send_mon_message to mon.noname-f at v2:172.21.15.107:3300/0 2023-09-30T01:36:42.182 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:42.157+0000 7f784ffff700 1 -- 172.21.15.88:0/2398185520 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7840009480 con 0x7f7850133b10 2023-09-30T01:36:42.182 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:42.157+0000 7f7854a93700 1 --2- 172.21.15.88:0/2398185520 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7f7850133740 0x7f7850142970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:36:42.182 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:42.157+0000 7f784ffff700 1 --2- 172.21.15.88:0/2398185520 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7850133b10 0x7f7850142eb0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f784400c560 tx=0x7f7844007e80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:36:42.182 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:42.157+0000 7f784cff9700 1 -- 172.21.15.88:0/2398185520 <== mon.1 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f7844011a10 con 0x7f7850133b10 2023-09-30T01:36:42.182 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:42.157+0000 7f784cff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:36:42.182 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:42.157+0000 7f784cff9700 10 monclient: got monmap 1 from mon.noname-f (according to old e1) 2023-09-30T01:36:42.182 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:42.157+0000 7f784cff9700 10 monclient: dump: 2023-09-30T01:36:42.183 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:36:42.183 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:36:42.183 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:36:42.183 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:36:42.183 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:36:42.183 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:36:42.183 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:36:42.183 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:36:42.183 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:36:42.183 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:36:42.183 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:36:42.183 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:36:42.183 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:36:42.184 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:36:42.184 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:36:42.184 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:36:42.184 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:42.157+0000 7f784cff9700 10 monclient: _finish_auth 0 2023-09-30T01:36:42.184 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:42.157+0000 7f784cff9700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:36:42.184 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:42.157+0000 7f7855a95700 5 monclient: authenticate success, global_id 4978 2023-09-30T01:36:42.184 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:42.157+0000 7f784cff9700 1 -- 172.21.15.88:0/2398185520 <== mon.1 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7844011bb0 con 0x7f7850133b10 2023-09-30T01:36:42.184 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:42.157+0000 7f784cff9700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:36:42.184 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:42.157+0000 7f7855a95700 10 monclient: _renew_subs 2023-09-30T01:36:42.184 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:42.157+0000 7f784cff9700 1 -- 172.21.15.88:0/2398185520 <== mon.1 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f7844011d60 con 0x7f7850133b10 2023-09-30T01:36:42.184 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:42.157+0000 7f7855a95700 10 monclient: _send_mon_message to mon.f at v2:172.21.15.107:3300/0 2023-09-30T01:36:42.184 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:42.157+0000 7f784effd700 4 set_mon_vals no callback set 2023-09-30T01:36:42.185 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:42.157+0000 7f7855a95700 1 -- 172.21.15.88:0/2398185520 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f785011f2b0 con 0x7f7850133b10 2023-09-30T01:36:42.185 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:42.157+0000 7f784cff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:36:42.185 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:42.157+0000 7f784cff9700 10 monclient: got monmap 1 from mon.f (according to old e1) 2023-09-30T01:36:42.185 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:42.157+0000 7f784cff9700 10 monclient: dump: 2023-09-30T01:36:42.185 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:36:42.185 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:36:42.185 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:36:42.185 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:36:42.185 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:36:42.185 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:36:42.185 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:36:42.185 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:36:42.186 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:36:42.186 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:36:42.186 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:36:42.186 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:36:42.186 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:36:42.186 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:36:42.186 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:36:42.186 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:36:42.186 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:42.161+0000 7f7855a95700 10 monclient: _renew_subs 2023-09-30T01:36:42.186 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:42.161+0000 7f7855a95700 10 monclient: _send_mon_message to mon.f at v2:172.21.15.107:3300/0 2023-09-30T01:36:42.186 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:42.161+0000 7f7855a95700 1 -- 172.21.15.88:0/2398185520 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7850120860 con 0x7f7850133b10 2023-09-30T01:36:42.186 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:42.161+0000 7f7855a95700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:36:42.187 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:42.161+0000 7f7855a95700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:36:42.187 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:42.161+0000 7f7855a95700 10 monclient: _send_mon_message to mon.f at v2:172.21.15.107:3300/0 2023-09-30T01:36:42.187 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:42.161+0000 7f7855a95700 1 -- 172.21.15.88:0/2398185520 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7850071960 con 0x7f7850133b10 2023-09-30T01:36:42.187 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:42.161+0000 7f784cff9700 1 -- 172.21.15.88:0/2398185520 <== mon.1 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 40144+0+0 (secure 0 0 0) 0x7f7844011590 con 0x7f7850133b10 2023-09-30T01:36:42.187 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:42.161+0000 7f784cff9700 1 --2- 172.21.15.88:0/2398185520 >> v2:172.21.15.107:6812/29926 conn(0x7f783c033360 0x7f783c035810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:36:42.187 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:42.161+0000 7f784cff9700 1 -- 172.21.15.88:0/2398185520 <== mon.1 v2:172.21.15.107:3300/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 3502+0+0 (secure 0 0 0) 0x7f78440428f0 con 0x7f7850133b10 2023-09-30T01:36:42.187 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:42.161+0000 7f7854a93700 1 --2- 172.21.15.88:0/2398185520 >> v2:172.21.15.107:6812/29926 conn(0x7f783c033360 0x7f783c035810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:36:42.187 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:42.161+0000 7f7854a93700 10 monclient: get_auth_request con 0x7f783c033360 auth_method 0 2023-09-30T01:36:42.187 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:42.161+0000 7f7854a93700 1 --2- 172.21.15.88:0/2398185520 >> v2:172.21.15.107:6812/29926 conn(0x7f783c033360 0x7f783c035810 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f7840009910 tx=0x7f7840016000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:36:42.187 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:42.173+0000 7f784cff9700 1 -- 172.21.15.88:0/2398185520 <== mon.1 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f7844011840 con 0x7f7850133b10 2023-09-30T01:36:42.187 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:42.173+0000 7f784cff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:36:42.187 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:42.173+0000 7f784cff9700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:36:42.369 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:42.365+0000 7f7855a95700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T01:36:42.370 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:42.365+0000 7f7855a95700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:36:42.370 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:42.365+0000 7f7855a95700 10 monclient: _send_mon_message to mon.f at v2:172.21.15.107:3300/0 2023-09-30T01:36:42.370 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:42.365+0000 7f7855a95700 1 -- 172.21.15.88:0/2398185520 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f7850067950 con 0x7f7850133b10 2023-09-30T01:36:42.370 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:42.365+0000 7f784cff9700 1 -- 172.21.15.88:0/2398185520 <== mon.1 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f7844010a90 con 0x7f7850133b10 2023-09-30T01:36:42.370 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:42.365+0000 7f784cff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:36:42.370 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:42.365+0000 7f784cff9700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:36:42.371 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-09-30T01:36:42.371 INFO:teuthology.orchestra.run.smithi088.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:36:42.375 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:42.369+0000 7f7855a95700 1 -- 172.21.15.88:0/2398185520 >> v2:172.21.15.107:6812/29926 conn(0x7f783c033360 msgr2=0x7f783c035810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:36:42.375 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:42.369+0000 7f7855a95700 1 --2- 172.21.15.88:0/2398185520 >> v2:172.21.15.107:6812/29926 conn(0x7f783c033360 0x7f783c035810 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f7840009910 tx=0x7f7840016000 comp rx=0 tx=0).stop 2023-09-30T01:36:42.375 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:42.369+0000 7f7855a95700 10 monclient: shutdown 2023-09-30T01:36:42.375 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:42.369+0000 7f7855a95700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:36:42.375 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:42.369+0000 7f7855a95700 1 -- 172.21.15.88:0/2398185520 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7850133b10 msgr2=0x7f7850142eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:36:42.375 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:42.369+0000 7f7855a95700 1 --2- 172.21.15.88:0/2398185520 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7850133b10 0x7f7850142eb0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f784400c560 tx=0x7f7844007e80 comp rx=0 tx=0).stop 2023-09-30T01:36:42.375 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:42.369+0000 7f7855a95700 1 -- 172.21.15.88:0/2398185520 shutdown_connections 2023-09-30T01:36:42.375 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:42.369+0000 7f7855a95700 1 --2- 172.21.15.88:0/2398185520 >> v2:172.21.15.107:6812/29926 conn(0x7f783c033360 0x7f783c035810 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:36:42.375 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:42.369+0000 7f7855a95700 1 --2- 172.21.15.88:0/2398185520 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7f7850133ee0 0x7f785009cc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:36:42.376 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:42.369+0000 7f7855a95700 1 --2- 172.21.15.88:0/2398185520 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7850133b10 0x7f7850142eb0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:36:42.376 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:42.369+0000 7f7855a95700 1 --2- 172.21.15.88:0/2398185520 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7f7850133740 0x7f7850142970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:36:42.376 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:42.369+0000 7f7855a95700 1 -- 172.21.15.88:0/2398185520 >> 172.21.15.88:0/2398185520 conn(0x7f7850097a40 msgr2=0x7f7850134400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:36:42.376 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:42.369+0000 7f7855a95700 1 -- 172.21.15.88:0/2398185520 shutdown_connections 2023-09-30T01:36:42.376 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:42.369+0000 7f7855a95700 1 -- 172.21.15.88:0/2398185520 wait complete. 2023-09-30T01:36:45.391 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:36:45.507 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:45.501+0000 7f6bdf59e700 10 monclient: get_monmap_and_config 2023-09-30T01:36:45.507 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:45.501+0000 7f6bdf59e700 10 monclient: build_initial_monmap 2023-09-30T01:36:45.507 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:45.501+0000 7f6bdf59e700 10 monclient: monmap: 2023-09-30T01:36:45.507 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2023-09-30T01:36:45.507 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:36:45.507 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:36:45.507155+0000 2023-09-30T01:36:45.508 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:36:45.507155+0000 2023-09-30T01:36:45.508 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2023-09-30T01:36:45.508 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:36:45.508 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a-legacy 2023-09-30T01:36:45.508 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.107:6789/0 mon.noname-f-legacy 2023-09-30T01:36:45.508 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.noname-b 2023-09-30T01:36:45.508 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.noname-g 2023-09-30T01:36:45.508 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.noname-c 2023-09-30T01:36:45.508 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.noname-h 2023-09-30T01:36:45.508 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.noname-d 2023-09-30T01:36:45.508 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.noname-i 2023-09-30T01:36:45.509 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.noname-e 2023-09-30T01:36:45.509 INFO:teuthology.orchestra.run.smithi088.stderr:9: v2:172.21.15.88:3300/0 mon.noname-a 2023-09-30T01:36:45.509 INFO:teuthology.orchestra.run.smithi088.stderr:10: v2:172.21.15.107:3300/0 mon.noname-f 2023-09-30T01:36:45.509 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:36:45.516 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:45.505+0000 7f6bdf59e700 1 Processor -- start 2023-09-30T01:36:45.516 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:45.505+0000 7f6bdf59e700 1 -- start start 2023-09-30T01:36:45.517 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:45.505+0000 7f6bdf59e700 10 monclient: init 2023-09-30T01:36:45.517 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:45.505+0000 7f6bdf59e700 10 monclient: _reopen_session rank -1 2023-09-30T01:36:45.517 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:45.505+0000 7f6bdf59e700 10 monclient: _add_conns ranks=[10,4,1,7,5,2,6,3,0,9,8] 2023-09-30T01:36:45.517 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:45.505+0000 7f6bdf59e700 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f6be00a3380 0x7f6be00a48b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:36:45.517 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:45.505+0000 7f6bdf59e700 10 monclient(hunting): picked mon.noname-f con 0x7f6be00a3380 addr v2:172.21.15.107:3300/0 2023-09-30T01:36:45.517 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:45.505+0000 7f6bdf59e700 1 --2- >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f6be009ffd0 0x7f6be00a03a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:36:45.517 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:45.505+0000 7f6bdf59e700 10 monclient(hunting): picked mon.noname-c con 0x7f6be009ffd0 addr [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] 2023-09-30T01:36:45.517 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:45.505+0000 7f6bdf59e700 10 monclient(hunting): picked mon.noname-f-legacy con 0x7f6be01333b0 addr v1:172.21.15.107:6789/0 2023-09-30T01:36:45.517 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:45.505+0000 7f6bdf59e700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6be004bb20 con 0x7f6be01333b0 2023-09-30T01:36:45.517 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:45.505+0000 7f6bdf59e700 10 monclient(hunting): start opening mon connection 2023-09-30T01:36:45.518 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:45.505+0000 7f6bdf59e700 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6be0097ab0 con 0x7f6be00a3380 2023-09-30T01:36:45.518 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:45.505+0000 7f6bdf59e700 10 monclient(hunting): start opening mon connection 2023-09-30T01:36:45.518 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:45.505+0000 7f6bdf59e700 1 -- --> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] -- mon_getmap magic: 0 v1 -- 0x7f6be00a08e0 con 0x7f6be009ffd0 2023-09-30T01:36:45.518 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:45.505+0000 7f6bdf59e700 10 monclient(hunting): _renew_subs 2023-09-30T01:36:45.518 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:45.505+0000 7f6bdf59e700 10 monclient(hunting): authenticate will time out at 1077.545137s 2023-09-30T01:36:45.518 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:45.505+0000 7f6bddd9b700 1 --2- >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f6be009ffd0 0x7f6be00a03a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:36:45.518 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:45.505+0000 7f6bddd9b700 1 --2- >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f6be009ffd0 0x7f6be00a03a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3302/0 says I am v2:172.21.15.88:36806/0 (socket says 172.21.15.88:36806) 2023-09-30T01:36:45.518 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:45.505+0000 7f6bddd9b700 1 -- 172.21.15.88:0/1056317381 learned_addr learned my addr 172.21.15.88:0/1056317381 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-09-30T01:36:45.518 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:45.505+0000 7f6bddd9b700 10 monclient(hunting): get_auth_request con 0x7f6be009ffd0 auth_method 0 2023-09-30T01:36:45.518 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:45.505+0000 7f6bddd9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:36:45.518 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:45.505+0000 7f6bddd9b700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:36:45.519 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:45.505+0000 7f6bddd9b700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:36:45.519 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:45.505+0000 7f6bddd9b700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:36:45.519 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:45.505+0000 7f6bddd9b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:36:45.519 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:45.505+0000 7f6bddd9b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:36:45.519 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:45.505+0000 7f6bde59c700 1 --2- 172.21.15.88:0/1056317381 >> v2:172.21.15.107:3300/0 conn(0x7f6be00a3380 0x7f6be00a48b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:36:45.519 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:45.505+0000 7f6bde59c700 10 monclient(hunting): get_auth_request con 0x7f6be00a3380 auth_method 0 2023-09-30T01:36:45.519 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:45.505+0000 7f6bde59c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:36:45.519 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:45.505+0000 7f6bde59c700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:36:45.519 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:45.505+0000 7f6bde59c700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:36:45.519 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:45.505+0000 7f6bddd9b700 10 monclient(hunting): handle_auth_done global_id 4738 payload 274 2023-09-30T01:36:45.520 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:45.505+0000 7f6bddd9b700 1 -- 172.21.15.88:0/1056317381 >> v2:172.21.15.107:3300/0 conn(0x7f6be00a3380 msgr2=0x7f6be00a48b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:36:45.520 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:45.505+0000 7f6bddd9b700 1 --2- 172.21.15.88:0/1056317381 >> v2:172.21.15.107:3300/0 conn(0x7f6be00a3380 0x7f6be00a48b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:36:45.520 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:45.505+0000 7f6bddd9b700 1 -- 172.21.15.88:0/1056317381 >> v1:172.21.15.107:6789/0 conn(0x7f6be01333b0 legacy=0x7f6be0133820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:36:45.520 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:45.505+0000 7f6bddd9b700 10 monclient: _finish_hunting 0 2023-09-30T01:36:45.520 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:45.505+0000 7f6bddd9b700 1 monclient: found mon.noname-c 2023-09-30T01:36:45.520 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:45.505+0000 7f6bddd9b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:36:45.520 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:45.505+0000 7f6bddd9b700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.88:3302/0 2023-09-30T01:36:45.520 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:45.505+0000 7f6bddd9b700 1 -- 172.21.15.88:0/1056317381 --> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6be0098480 con 0x7f6be009ffd0 2023-09-30T01:36:45.520 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:45.505+0000 7f6bddd9b700 1 --2- 172.21.15.88:0/1056317381 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f6be009ffd0 0x7f6be00a03a0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f6bd40099b0 tx=0x7f6bd4009250 comp rx=0 tx=0).ready entity=mon.4 client_cookie=67695889ccc3a690 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:36:45.520 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:45.505+0000 7f6bdd59a700 1 -- 172.21.15.88:0/1056317381 <== mon.4 v2:172.21.15.88:3302/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f6bd400bca0 con 0x7f6be009ffd0 2023-09-30T01:36:45.520 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:45.505+0000 7f6bdd59a700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:36:45.521 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:45.505+0000 7f6bdd59a700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T01:36:45.521 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:45.505+0000 7f6bdd59a700 10 monclient: dump: 2023-09-30T01:36:45.521 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:36:45.521 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:36:45.521 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:36:45.521 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:36:45.521 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:36:45.521 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:36:45.521 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:36:45.521 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:36:45.521 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:36:45.522 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:36:45.522 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:36:45.522 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:36:45.522 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:36:45.522 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:36:45.522 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:36:45.522 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:36:45.522 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:45.505+0000 7f6bdd59a700 10 monclient: _finish_auth 0 2023-09-30T01:36:45.522 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:45.505+0000 7f6bdd59a700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:36:45.522 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:45.505+0000 7f6bdf59e700 5 monclient: authenticate success, global_id 4738 2023-09-30T01:36:45.523 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:45.505+0000 7f6bdf59e700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:36:45.523 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:45.505+0000 7f6bdd59a700 1 -- 172.21.15.88:0/1056317381 <== mon.4 v2:172.21.15.88:3302/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6bd400be20 con 0x7f6be009ffd0 2023-09-30T01:36:45.523 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:45.505+0000 7f6bdd59a700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:36:45.523 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:45.505+0000 7f6bdf59e700 10 monclient: get_monmap_and_config success 2023-09-30T01:36:45.523 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:45.505+0000 7f6bdf59e700 4 set_mon_vals no callback set 2023-09-30T01:36:45.523 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:45.505+0000 7f6bdd59a700 1 -- 172.21.15.88:0/1056317381 <== mon.4 v2:172.21.15.88:3302/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f6bd40118a0 con 0x7f6be009ffd0 2023-09-30T01:36:45.523 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:45.505+0000 7f6bdd59a700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:36:45.523 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:45.505+0000 7f6bdd59a700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T01:36:45.523 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:45.505+0000 7f6bdd59a700 10 monclient: dump: 2023-09-30T01:36:45.523 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:36:45.523 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:36:45.524 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:36:45.524 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:36:45.524 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:36:45.524 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:36:45.524 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:36:45.524 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:36:45.524 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:36:45.524 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:36:45.524 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:36:45.524 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:36:45.524 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:36:45.524 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:36:45.525 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:36:45.525 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:36:45.525 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:45.505+0000 7f6bdf59e700 10 monclient: shutdown 2023-09-30T01:36:45.525 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:45.505+0000 7f6bdf59e700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:36:45.525 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:45.505+0000 7f6bdf59e700 1 -- 172.21.15.88:0/1056317381 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f6be009ffd0 msgr2=0x7f6be00a03a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:36:45.525 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:45.505+0000 7f6bdf59e700 1 --2- 172.21.15.88:0/1056317381 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f6be009ffd0 0x7f6be00a03a0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f6bd40099b0 tx=0x7f6bd4009250 comp rx=0 tx=0).stop 2023-09-30T01:36:45.525 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:45.505+0000 7f6bdf59e700 1 -- 172.21.15.88:0/1056317381 shutdown_connections 2023-09-30T01:36:45.525 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:45.505+0000 7f6bdf59e700 1 --2- 172.21.15.88:0/1056317381 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f6be009ffd0 0x7f6be00a03a0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:36:45.525 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:45.505+0000 7f6bdf59e700 1 --2- 172.21.15.88:0/1056317381 >> v2:172.21.15.107:3300/0 conn(0x7f6be00a3380 0x7f6be00a48b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:36:45.525 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:45.505+0000 7f6bdf59e700 1 -- 172.21.15.88:0/1056317381 >> 172.21.15.88:0/1056317381 conn(0x7f6be006e5e0 msgr2=0x7f6be006e9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:36:45.525 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:45.505+0000 7f6bdf59e700 1 -- 172.21.15.88:0/1056317381 shutdown_connections 2023-09-30T01:36:45.526 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:45.505+0000 7f6bdf59e700 1 -- 172.21.15.88:0/1056317381 wait complete. 2023-09-30T01:36:45.526 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:45.509+0000 7f6bdf59e700 10 monclient: build_initial_monmap 2023-09-30T01:36:45.526 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:45.509+0000 7f6bdf59e700 10 monclient: monmap: 2023-09-30T01:36:45.526 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2023-09-30T01:36:45.526 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:36:45.526 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 0.000000 2023-09-30T01:36:45.526 INFO:teuthology.orchestra.run.smithi088.stderr:created 0.000000 2023-09-30T01:36:45.526 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2023-09-30T01:36:45.526 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 1 2023-09-30T01:36:45.526 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.noname-a 2023-09-30T01:36:45.527 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.noname-f 2023-09-30T01:36:45.527 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.noname-b 2023-09-30T01:36:45.527 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.noname-g 2023-09-30T01:36:45.527 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.noname-c 2023-09-30T01:36:45.527 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.noname-h 2023-09-30T01:36:45.527 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.noname-d 2023-09-30T01:36:45.527 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.noname-i 2023-09-30T01:36:45.527 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.noname-e 2023-09-30T01:36:45.527 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:36:45.527 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:45.509+0000 7f6bdf59e700 1 Processor -- start 2023-09-30T01:36:45.527 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:45.509+0000 7f6bdf59e700 1 -- start start 2023-09-30T01:36:45.527 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:45.509+0000 7f6bdf59e700 10 monclient: init 2023-09-30T01:36:45.528 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:45.509+0000 7f6bdf59e700 10 monclient: _reopen_session rank -1 2023-09-30T01:36:45.528 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:45.509+0000 7f6bdf59e700 10 monclient: _add_conns ranks=[0,5,2,3,7,6,1,4,8] 2023-09-30T01:36:45.528 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:45.509+0000 7f6bdf59e700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6be009ffd0 0x7f6be011e7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:36:45.528 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:45.509+0000 7f6bdf59e700 10 monclient(hunting): picked mon.noname-a con 0x7f6be009ffd0 addr [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] 2023-09-30T01:36:45.528 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:45.509+0000 7f6bdf59e700 1 --2- >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7f6be00a3380 0x7f6be011ed10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:36:45.528 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:45.509+0000 7f6bdf59e700 10 monclient(hunting): picked mon.noname-h con 0x7f6be00a3380 addr [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] 2023-09-30T01:36:45.528 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:45.509+0000 7f6bdf59e700 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6be01333b0 0x7f6be011f250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:36:45.528 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:45.509+0000 7f6bdf59e700 10 monclient(hunting): picked mon.noname-b con 0x7f6be01333b0 addr [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] 2023-09-30T01:36:45.528 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:45.509+0000 7f6bdf59e700 10 monclient(hunting): start opening mon connection 2023-09-30T01:36:45.528 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:45.509+0000 7f6bdf59e700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6be00a0ac0 con 0x7f6be009ffd0 2023-09-30T01:36:45.529 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:45.509+0000 7f6bdf59e700 10 monclient(hunting): start opening mon connection 2023-09-30T01:36:45.529 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:45.509+0000 7f6bdf59e700 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6be0097ce0 con 0x7f6be01333b0 2023-09-30T01:36:45.529 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:45.509+0000 7f6bdf59e700 10 monclient(hunting): start opening mon connection 2023-09-30T01:36:45.529 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:45.509+0000 7f6bdf59e700 1 -- --> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] -- mon_getmap magic: 0 v1 -- 0x7f6be0122210 con 0x7f6be00a3380 2023-09-30T01:36:45.529 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:45.509+0000 7f6bdf59e700 10 monclient(hunting): _renew_subs 2023-09-30T01:36:45.529 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:45.509+0000 7f6bdf59e700 10 monclient(hunting): authenticate will time out at 1077.549006s 2023-09-30T01:36:45.529 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:45.509+0000 7f6bded9d700 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6be01333b0 0x7f6be011f250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:36:45.529 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:45.509+0000 7f6bddd9b700 1 --2- >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7f6be00a3380 0x7f6be011ed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:36:45.530 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:45.509+0000 7f6bded9d700 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6be01333b0 0x7f6be011f250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3301/0 says I am v2:172.21.15.88:60400/0 (socket says 172.21.15.88:60400) 2023-09-30T01:36:45.530 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:45.509+0000 7f6bded9d700 1 -- 172.21.15.88:0/1241579348 learned_addr learned my addr 172.21.15.88:0/1241579348 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-09-30T01:36:45.530 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:45.509+0000 7f6bded9d700 10 monclient(hunting): get_auth_request con 0x7f6be01333b0 auth_method 0 2023-09-30T01:36:45.530 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:45.509+0000 7f6bded9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:36:45.530 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:45.509+0000 7f6bded9d700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:36:45.530 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:45.509+0000 7f6bded9d700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:36:45.530 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:45.509+0000 7f6bddd9b700 10 monclient(hunting): get_auth_request con 0x7f6be00a3380 auth_method 0 2023-09-30T01:36:45.530 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:45.509+0000 7f6bddd9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:36:45.530 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:45.509+0000 7f6bddd9b700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:36:45.530 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:45.509+0000 7f6bddd9b700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:36:45.531 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:45.509+0000 7f6bde59c700 1 --2- 172.21.15.88:0/1241579348 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6be009ffd0 0x7f6be011e7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:36:45.531 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:45.509+0000 7f6bde59c700 10 monclient(hunting): get_auth_request con 0x7f6be009ffd0 auth_method 0 2023-09-30T01:36:45.531 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:45.509+0000 7f6bde59c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:36:45.531 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:45.509+0000 7f6bde59c700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:36:45.531 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:45.509+0000 7f6bde59c700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:36:45.531 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:45.509+0000 7f6bded9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:36:45.531 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:45.509+0000 7f6bded9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:36:45.531 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:45.509+0000 7f6bded9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:36:45.531 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:45.509+0000 7f6bddd9b700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:36:45.531 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:45.509+0000 7f6bddd9b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:36:45.531 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:45.509+0000 7f6bddd9b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:36:45.532 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:45.509+0000 7f6bde59c700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:36:45.532 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:45.509+0000 7f6bde59c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:36:45.532 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:45.509+0000 7f6bde59c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:36:45.532 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:45.509+0000 7f6bddd9b700 10 monclient(hunting): handle_auth_done global_id 4703 payload 824 2023-09-30T01:36:45.532 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:45.509+0000 7f6bddd9b700 1 -- 172.21.15.88:0/1241579348 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6be01333b0 msgr2=0x7f6be011f250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:36:45.532 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:45.509+0000 7f6bddd9b700 1 --2- 172.21.15.88:0/1241579348 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6be01333b0 0x7f6be011f250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:36:45.532 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:45.509+0000 7f6bddd9b700 1 -- 172.21.15.88:0/1241579348 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6be009ffd0 msgr2=0x7f6be011e7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:36:45.532 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:45.509+0000 7f6bddd9b700 1 --2- 172.21.15.88:0/1241579348 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6be009ffd0 0x7f6be011e7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:36:45.532 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:45.509+0000 7f6bddd9b700 10 monclient: _finish_hunting 0 2023-09-30T01:36:45.532 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:45.509+0000 7f6bddd9b700 1 monclient: found mon.noname-h 2023-09-30T01:36:45.532 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:45.509+0000 7f6bddd9b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:36:45.533 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:45.509+0000 7f6bddd9b700 10 monclient: _send_mon_message to mon.noname-h at v2:172.21.15.107:3302/0 2023-09-30T01:36:45.533 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:45.509+0000 7f6bddd9b700 1 -- 172.21.15.88:0/1241579348 --> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6bd40094f0 con 0x7f6be00a3380 2023-09-30T01:36:45.533 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:45.509+0000 7f6bddd9b700 1 --2- 172.21.15.88:0/1241579348 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7f6be00a3380 0x7f6be011ed10 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f6bd4002000 tx=0x7f6bd4016000 comp rx=0 tx=0).ready entity=mon.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:36:45.533 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:45.509+0000 7f6bce7fc700 1 -- 172.21.15.88:0/1241579348 <== mon.5 v2:172.21.15.107:3302/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f6bd4006b30 con 0x7f6be00a3380 2023-09-30T01:36:45.533 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:45.509+0000 7f6bce7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:36:45.533 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:45.509+0000 7f6bce7fc700 10 monclient: got monmap 1 from mon.noname-h (according to old e1) 2023-09-30T01:36:45.533 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:45.509+0000 7f6bce7fc700 10 monclient: dump: 2023-09-30T01:36:45.533 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:36:45.533 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:36:45.533 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:36:45.533 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:36:45.534 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:36:45.534 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:36:45.534 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:36:45.534 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:36:45.534 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:36:45.534 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:36:45.534 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:36:45.534 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:36:45.534 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:36:45.534 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:36:45.534 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:36:45.534 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:36:45.535 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:45.509+0000 7f6bce7fc700 10 monclient: _finish_auth 0 2023-09-30T01:36:45.535 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:45.509+0000 7f6bce7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:36:45.535 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:45.509+0000 7f6bce7fc700 1 -- 172.21.15.88:0/1241579348 <== mon.5 v2:172.21.15.107:3302/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6bd4020070 con 0x7f6be00a3380 2023-09-30T01:36:45.535 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:45.509+0000 7f6bdf59e700 5 monclient: authenticate success, global_id 4703 2023-09-30T01:36:45.535 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:45.509+0000 7f6bdf59e700 10 monclient: _renew_subs 2023-09-30T01:36:45.535 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:45.509+0000 7f6bdf59e700 10 monclient: _send_mon_message to mon.h at v2:172.21.15.107:3302/0 2023-09-30T01:36:45.535 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:45.509+0000 7f6bdf59e700 1 -- 172.21.15.88:0/1241579348 --> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6be01211b0 con 0x7f6be00a3380 2023-09-30T01:36:45.536 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:45.509+0000 7f6bce7fc700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:36:45.536 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:45.509+0000 7f6bce7fc700 1 -- 172.21.15.88:0/1241579348 <== mon.5 v2:172.21.15.107:3302/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f6bd401cd50 con 0x7f6be00a3380 2023-09-30T01:36:45.536 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:45.509+0000 7f6bce7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:36:45.536 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:45.509+0000 7f6bce7fc700 10 monclient: got monmap 1 from mon.h (according to old e1) 2023-09-30T01:36:45.536 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:45.509+0000 7f6bce7fc700 10 monclient: dump: 2023-09-30T01:36:45.536 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:36:45.536 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:36:45.536 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:36:45.536 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:36:45.536 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:36:45.536 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:36:45.536 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:36:45.537 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:36:45.537 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:36:45.537 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:36:45.537 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:36:45.537 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:36:45.537 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:36:45.537 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:36:45.537 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:36:45.537 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:36:45.537 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:45.509+0000 7f6bcffff700 4 set_mon_vals no callback set 2023-09-30T01:36:45.537 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:45.509+0000 7f6bdf59e700 10 monclient: _renew_subs 2023-09-30T01:36:45.538 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:45.509+0000 7f6bdf59e700 10 monclient: _send_mon_message to mon.h at v2:172.21.15.107:3302/0 2023-09-30T01:36:45.538 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:45.509+0000 7f6bdf59e700 1 -- 172.21.15.88:0/1241579348 --> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6be0131fc0 con 0x7f6be00a3380 2023-09-30T01:36:45.539 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:45.513+0000 7f6bdf59e700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:36:45.539 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:45.513+0000 7f6bce7fc700 1 -- 172.21.15.88:0/1241579348 <== mon.5 v2:172.21.15.107:3302/0 4 ==== mgrmap(e 5) v1 ==== 40144+0+0 (secure 0 0 0) 0x7f6bd4022030 con 0x7f6be00a3380 2023-09-30T01:36:45.539 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:45.513+0000 7f6bdf59e700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:36:45.539 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:45.513+0000 7f6bdf59e700 10 monclient: _send_mon_message to mon.h at v2:172.21.15.107:3302/0 2023-09-30T01:36:45.539 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:45.513+0000 7f6bdf59e700 1 -- 172.21.15.88:0/1241579348 --> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6be0122380 con 0x7f6be00a3380 2023-09-30T01:36:45.539 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:45.513+0000 7f6bce7fc700 1 --2- 172.21.15.88:0/1241579348 >> v2:172.21.15.107:6812/29926 conn(0x7f6bc00332c0 0x7f6bc0035770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:36:45.540 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:45.513+0000 7f6bde59c700 1 --2- 172.21.15.88:0/1241579348 >> v2:172.21.15.107:6812/29926 conn(0x7f6bc00332c0 0x7f6bc0035770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:36:45.540 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:45.513+0000 7f6bde59c700 10 monclient: get_auth_request con 0x7f6bc00332c0 auth_method 0 2023-09-30T01:36:45.540 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:45.517+0000 7f6bde59c700 1 --2- 172.21.15.88:0/1241579348 >> v2:172.21.15.107:6812/29926 conn(0x7f6bc00332c0 0x7f6bc0035770 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f6bd0006fd0 tx=0x7f6bd0008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:36:45.540 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:45.521+0000 7f6bce7fc700 1 -- 172.21.15.88:0/1241579348 <== mon.5 v2:172.21.15.107:3302/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 3502+0+0 (secure 0 0 0) 0x7f6bd4011c50 con 0x7f6be00a3380 2023-09-30T01:36:45.540 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:45.525+0000 7f6bce7fc700 1 -- 172.21.15.88:0/1241579348 <== mon.5 v2:172.21.15.107:3302/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f6bd40247e0 con 0x7f6be00a3380 2023-09-30T01:36:45.540 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:45.525+0000 7f6bce7fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:36:45.540 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:45.525+0000 7f6bce7fc700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:36:45.730 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:45.724+0000 7f6bdf59e700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T01:36:45.730 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:45.724+0000 7f6bdf59e700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:36:45.731 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:45.724+0000 7f6bdf59e700 10 monclient: _send_mon_message to mon.h at v2:172.21.15.107:3302/0 2023-09-30T01:36:45.731 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:45.724+0000 7f6bdf59e700 1 -- 172.21.15.88:0/1241579348 --> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f6be009d6a0 con 0x7f6be00a3380 2023-09-30T01:36:45.732 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:45.724+0000 7f6bce7fc700 1 -- 172.21.15.88:0/1241579348 <== mon.5 v2:172.21.15.107:3302/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f6bd40193f0 con 0x7f6be00a3380 2023-09-30T01:36:45.732 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:45.724+0000 7f6bce7fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:36:45.732 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:45.724+0000 7f6bce7fc700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:36:45.732 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-09-30T01:36:45.732 INFO:teuthology.orchestra.run.smithi088.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:36:45.734 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:45.728+0000 7f6bdf59e700 1 -- 172.21.15.88:0/1241579348 >> v2:172.21.15.107:6812/29926 conn(0x7f6bc00332c0 msgr2=0x7f6bc0035770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:36:45.734 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:45.728+0000 7f6bdf59e700 1 --2- 172.21.15.88:0/1241579348 >> v2:172.21.15.107:6812/29926 conn(0x7f6bc00332c0 0x7f6bc0035770 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f6bd0006fd0 tx=0x7f6bd0008040 comp rx=0 tx=0).stop 2023-09-30T01:36:45.734 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:45.728+0000 7f6bdf59e700 10 monclient: shutdown 2023-09-30T01:36:45.734 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:45.728+0000 7f6bdf59e700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:36:45.734 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:45.728+0000 7f6bdf59e700 1 -- 172.21.15.88:0/1241579348 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7f6be00a3380 msgr2=0x7f6be011ed10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:36:45.734 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:45.728+0000 7f6bdf59e700 1 --2- 172.21.15.88:0/1241579348 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7f6be00a3380 0x7f6be011ed10 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f6bd4002000 tx=0x7f6bd4016000 comp rx=0 tx=0).stop 2023-09-30T01:36:45.734 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:45.728+0000 7f6bdf59e700 1 -- 172.21.15.88:0/1241579348 shutdown_connections 2023-09-30T01:36:45.734 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:45.728+0000 7f6bdf59e700 1 --2- 172.21.15.88:0/1241579348 >> v2:172.21.15.107:6812/29926 conn(0x7f6bc00332c0 0x7f6bc0035770 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:36:45.735 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:45.728+0000 7f6bdf59e700 1 --2- 172.21.15.88:0/1241579348 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6be01333b0 0x7f6be011f250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:36:45.735 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:45.728+0000 7f6bdf59e700 1 --2- 172.21.15.88:0/1241579348 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7f6be00a3380 0x7f6be011ed10 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:36:45.735 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:45.728+0000 7f6bdf59e700 1 --2- 172.21.15.88:0/1241579348 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6be009ffd0 0x7f6be011e7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:36:45.735 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:45.728+0000 7f6bdf59e700 1 -- 172.21.15.88:0/1241579348 >> 172.21.15.88:0/1241579348 conn(0x7f6be006e5e0 msgr2=0x7f6be01323c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:36:45.735 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:45.728+0000 7f6bdf59e700 1 -- 172.21.15.88:0/1241579348 shutdown_connections 2023-09-30T01:36:45.735 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:45.728+0000 7f6bdf59e700 1 -- 172.21.15.88:0/1241579348 wait complete. 2023-09-30T01:36:48.745 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:36:48.857 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:48.852+0000 7fda669d1700 10 monclient: get_monmap_and_config 2023-09-30T01:36:48.857 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:48.852+0000 7fda669d1700 10 monclient: build_initial_monmap 2023-09-30T01:36:48.857 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:48.852+0000 7fda669d1700 10 monclient: monmap: 2023-09-30T01:36:48.857 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2023-09-30T01:36:48.858 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:36:48.858 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:36:48.857070+0000 2023-09-30T01:36:48.858 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:36:48.857070+0000 2023-09-30T01:36:48.858 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2023-09-30T01:36:48.858 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:36:48.858 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a-legacy 2023-09-30T01:36:48.858 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.107:6789/0 mon.noname-f-legacy 2023-09-30T01:36:48.858 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.noname-b 2023-09-30T01:36:48.858 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.noname-g 2023-09-30T01:36:48.858 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.noname-c 2023-09-30T01:36:48.859 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.noname-h 2023-09-30T01:36:48.859 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.noname-d 2023-09-30T01:36:48.859 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.noname-i 2023-09-30T01:36:48.859 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.noname-e 2023-09-30T01:36:48.859 INFO:teuthology.orchestra.run.smithi088.stderr:9: v2:172.21.15.88:3300/0 mon.noname-a 2023-09-30T01:36:48.859 INFO:teuthology.orchestra.run.smithi088.stderr:10: v2:172.21.15.107:3300/0 mon.noname-f 2023-09-30T01:36:48.859 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:36:48.862 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:48.852+0000 7fda669d1700 1 Processor -- start 2023-09-30T01:36:48.862 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:48.852+0000 7fda669d1700 1 -- start start 2023-09-30T01:36:48.862 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:48.852+0000 7fda669d1700 10 monclient: init 2023-09-30T01:36:48.863 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:48.852+0000 7fda669d1700 10 monclient: _reopen_session rank -1 2023-09-30T01:36:48.863 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:48.852+0000 7fda669d1700 10 monclient: _add_conns ranks=[1,7,5,10,9,3,4,6,8,0,2] 2023-09-30T01:36:48.863 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:48.852+0000 7fda669d1700 10 monclient(hunting): picked mon.noname-f-legacy con 0x7fda600a3380 addr v1:172.21.15.107:6789/0 2023-09-30T01:36:48.863 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:48.852+0000 7fda669d1700 1 --2- >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7fda6009ffd0 0x7fda600a03a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:36:48.863 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:48.852+0000 7fda669d1700 10 monclient(hunting): picked mon.noname-i con 0x7fda6009ffd0 addr [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] 2023-09-30T01:36:48.863 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:48.852+0000 7fda669d1700 1 --2- >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7fda601386e0 0x7fda6013aac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:36:48.863 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:48.852+0000 7fda669d1700 10 monclient(hunting): picked mon.noname-h con 0x7fda601386e0 addr [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] 2023-09-30T01:36:48.863 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:48.852+0000 7fda669d1700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda6004bb20 con 0x7fda600a3380 2023-09-30T01:36:48.864 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:48.852+0000 7fda669d1700 10 monclient(hunting): start opening mon connection 2023-09-30T01:36:48.864 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:48.852+0000 7fda669d1700 1 -- --> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] -- mon_getmap magic: 0 v1 -- 0x7fda60097ab0 con 0x7fda601386e0 2023-09-30T01:36:48.864 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:48.852+0000 7fda669d1700 10 monclient(hunting): start opening mon connection 2023-09-30T01:36:48.864 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:48.852+0000 7fda669d1700 1 -- --> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] -- mon_getmap magic: 0 v1 -- 0x7fda600a08e0 con 0x7fda6009ffd0 2023-09-30T01:36:48.864 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:48.852+0000 7fda669d1700 10 monclient(hunting): _renew_subs 2023-09-30T01:36:48.864 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:48.852+0000 7fda669d1700 10 monclient(hunting): authenticate will time out at 1080.894718s 2023-09-30T01:36:48.864 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:48.852+0000 7fda661d0700 1 --2- >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7fda601386e0 0x7fda6013aac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:36:48.864 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:48.852+0000 7fda661d0700 1 --2- >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7fda601386e0 0x7fda6013aac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3302/0 says I am v2:172.21.15.88:45750/0 (socket says 172.21.15.88:45750) 2023-09-30T01:36:48.864 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:48.852+0000 7fda661d0700 1 -- 172.21.15.88:0/2827694018 learned_addr learned my addr 172.21.15.88:0/2827694018 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-09-30T01:36:48.865 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:48.852+0000 7fda651ce700 1 --2- >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7fda6009ffd0 0x7fda600a03a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:36:48.865 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:48.852+0000 7fda661d0700 10 monclient(hunting): get_auth_request con 0x7fda601386e0 auth_method 0 2023-09-30T01:36:48.865 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:48.852+0000 7fda661d0700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:36:48.865 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:48.852+0000 7fda661d0700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:36:48.865 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:48.852+0000 7fda661d0700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:36:48.865 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:48.852+0000 7fda659cf700 1 --1- 172.21.15.88:0/2827694018 >> v1:172.21.15.107:6789/0 conn(0x7fda600a3380 0x7fda600a48b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.88:39486/0 (socket says 172.21.15.88:39486) 2023-09-30T01:36:48.865 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:48.852+0000 7fda651ce700 10 monclient(hunting): get_auth_request con 0x7fda6009ffd0 auth_method 0 2023-09-30T01:36:48.865 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:48.852+0000 7fda651ce700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:36:48.865 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:48.852+0000 7fda651ce700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:36:48.865 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:48.852+0000 7fda651ce700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:36:48.866 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:48.852+0000 7fda661d0700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:36:48.866 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:48.852+0000 7fda661d0700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:36:48.866 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:48.852+0000 7fda661d0700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:36:48.866 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:48.852+0000 7fda651ce700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:36:48.866 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:48.852+0000 7fda651ce700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:36:48.866 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:48.852+0000 7fda651ce700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:36:48.866 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:48.852+0000 7fda661d0700 10 monclient(hunting): handle_auth_done global_id 4712 payload 274 2023-09-30T01:36:48.866 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:48.852+0000 7fda661d0700 1 -- 172.21.15.88:0/2827694018 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7fda6009ffd0 msgr2=0x7fda600a03a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:36:48.866 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:48.852+0000 7fda661d0700 1 --2- 172.21.15.88:0/2827694018 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7fda6009ffd0 0x7fda600a03a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:36:48.866 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:48.852+0000 7fda661d0700 1 -- 172.21.15.88:0/2827694018 >> v1:172.21.15.107:6789/0 conn(0x7fda600a3380 legacy=0x7fda600a48b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:36:48.867 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:48.852+0000 7fda661d0700 10 monclient: _finish_hunting 0 2023-09-30T01:36:48.867 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:48.852+0000 7fda661d0700 1 monclient: found mon.noname-h 2023-09-30T01:36:48.867 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:48.852+0000 7fda649cd700 1 -- 172.21.15.88:0/2827694018 <== mon.1 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4192227015 0 0) 0x7fda6004bb20 con 0x7fda600a3380 2023-09-30T01:36:48.867 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:48.852+0000 7fda661d0700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:36:48.867 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:48.852+0000 7fda661d0700 10 monclient: _send_mon_message to mon.noname-h at v2:172.21.15.107:3302/0 2023-09-30T01:36:48.867 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:48.852+0000 7fda661d0700 1 -- 172.21.15.88:0/2827694018 --> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda600984b0 con 0x7fda601386e0 2023-09-30T01:36:48.867 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:48.852+0000 7fda649cd700 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2023-09-30T01:36:48.873 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:48.856+0000 7fda661d0700 1 --2- 172.21.15.88:0/2827694018 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7fda601386e0 0x7fda6013aac0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fda580099e0 tx=0x7fda58009250 comp rx=0 tx=0).ready entity=mon.5 client_cookie=75c8757d2012ae31 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:36:48.873 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:48.856+0000 7fda649cd700 1 -- 172.21.15.88:0/2827694018 <== mon.5 v2:172.21.15.107:3302/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7fda5800bce0 con 0x7fda601386e0 2023-09-30T01:36:48.873 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:48.856+0000 7fda649cd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:36:48.873 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:48.856+0000 7fda649cd700 10 monclient: got monmap 1 from mon.noname-h (according to old e1) 2023-09-30T01:36:48.873 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:48.856+0000 7fda649cd700 10 monclient: dump: 2023-09-30T01:36:48.873 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:36:48.873 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:36:48.874 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:36:48.874 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:36:48.874 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:36:48.874 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:36:48.874 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:36:48.874 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:36:48.874 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:36:48.874 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:36:48.874 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:36:48.874 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:36:48.874 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:36:48.875 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:36:48.875 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:36:48.875 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:36:48.875 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:48.856+0000 7fda649cd700 10 monclient: _finish_auth 0 2023-09-30T01:36:48.875 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:48.856+0000 7fda649cd700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:36:48.875 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:48.856+0000 7fda669d1700 5 monclient: authenticate success, global_id 4712 2023-09-30T01:36:48.875 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:48.856+0000 7fda669d1700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:36:48.875 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:48.856+0000 7fda649cd700 1 -- 172.21.15.88:0/2827694018 <== mon.5 v2:172.21.15.107:3302/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda5800be60 con 0x7fda601386e0 2023-09-30T01:36:48.875 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:48.856+0000 7fda649cd700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:36:48.875 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:48.856+0000 7fda669d1700 10 monclient: get_monmap_and_config success 2023-09-30T01:36:48.875 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:48.856+0000 7fda669d1700 4 set_mon_vals no callback set 2023-09-30T01:36:48.875 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:48.856+0000 7fda649cd700 1 -- 172.21.15.88:0/2827694018 <== mon.5 v2:172.21.15.107:3302/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7fda580118a0 con 0x7fda601386e0 2023-09-30T01:36:48.876 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:48.856+0000 7fda649cd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:36:48.876 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:48.856+0000 7fda649cd700 10 monclient: got monmap 1 from mon.h (according to old e1) 2023-09-30T01:36:48.876 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:48.856+0000 7fda649cd700 10 monclient: dump: 2023-09-30T01:36:48.876 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:36:48.876 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:36:48.876 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:36:48.876 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:36:48.876 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:36:48.876 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:36:48.876 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:36:48.876 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:36:48.876 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:36:48.876 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:36:48.877 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:36:48.877 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:36:48.877 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:36:48.877 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:36:48.877 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:36:48.877 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:36:48.877 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:48.856+0000 7fda669d1700 10 monclient: shutdown 2023-09-30T01:36:48.877 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:48.856+0000 7fda669d1700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:36:48.877 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:48.856+0000 7fda669d1700 1 -- 172.21.15.88:0/2827694018 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7fda601386e0 msgr2=0x7fda6013aac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:36:48.877 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:48.856+0000 7fda669d1700 1 --2- 172.21.15.88:0/2827694018 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7fda601386e0 0x7fda6013aac0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fda580099e0 tx=0x7fda58009250 comp rx=0 tx=0).stop 2023-09-30T01:36:48.877 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:48.856+0000 7fda669d1700 1 -- 172.21.15.88:0/2827694018 shutdown_connections 2023-09-30T01:36:48.877 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:48.856+0000 7fda669d1700 1 --2- 172.21.15.88:0/2827694018 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7fda601386e0 0x7fda6013aac0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:36:48.878 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:48.856+0000 7fda669d1700 1 --2- 172.21.15.88:0/2827694018 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7fda6009ffd0 0x7fda600a03a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:36:48.878 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:48.856+0000 7fda669d1700 1 -- 172.21.15.88:0/2827694018 >> 172.21.15.88:0/2827694018 conn(0x7fda6006e5e0 msgr2=0x7fda6006e9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:36:48.878 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:48.856+0000 7fda669d1700 1 -- 172.21.15.88:0/2827694018 shutdown_connections 2023-09-30T01:36:48.878 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:48.856+0000 7fda669d1700 1 -- 172.21.15.88:0/2827694018 wait complete. 2023-09-30T01:36:48.878 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:48.856+0000 7fda669d1700 10 monclient: build_initial_monmap 2023-09-30T01:36:48.878 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:48.856+0000 7fda669d1700 10 monclient: monmap: 2023-09-30T01:36:48.878 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2023-09-30T01:36:48.878 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:36:48.878 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 0.000000 2023-09-30T01:36:48.878 INFO:teuthology.orchestra.run.smithi088.stderr:created 0.000000 2023-09-30T01:36:48.878 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2023-09-30T01:36:48.878 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 1 2023-09-30T01:36:48.879 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.noname-a 2023-09-30T01:36:48.879 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.noname-f 2023-09-30T01:36:48.879 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.noname-b 2023-09-30T01:36:48.879 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.noname-g 2023-09-30T01:36:48.879 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.noname-c 2023-09-30T01:36:48.879 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.noname-h 2023-09-30T01:36:48.879 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.noname-d 2023-09-30T01:36:48.879 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.noname-i 2023-09-30T01:36:48.879 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.noname-e 2023-09-30T01:36:48.879 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:36:48.879 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:48.860+0000 7fda669d1700 1 Processor -- start 2023-09-30T01:36:48.879 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:48.860+0000 7fda669d1700 1 -- start start 2023-09-30T01:36:48.879 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:48.860+0000 7fda669d1700 10 monclient: init 2023-09-30T01:36:48.880 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:48.860+0000 7fda669d1700 10 monclient: _reopen_session rank -1 2023-09-30T01:36:48.880 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:48.860+0000 7fda669d1700 10 monclient: _add_conns ranks=[2,0,7,3,6,1,5,4,8] 2023-09-30T01:36:48.880 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:48.860+0000 7fda669d1700 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fda6009ffd0 0x7fda60135440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:36:48.880 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:48.860+0000 7fda669d1700 10 monclient(hunting): picked mon.noname-b con 0x7fda6009ffd0 addr [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] 2023-09-30T01:36:48.880 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:48.860+0000 7fda669d1700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fda600a3380 0x7fda60135980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:36:48.880 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:48.860+0000 7fda669d1700 10 monclient(hunting): picked mon.noname-a con 0x7fda600a3380 addr [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] 2023-09-30T01:36:48.880 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:48.860+0000 7fda669d1700 1 --2- >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7fda601386e0 0x7fda60135ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:36:48.880 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:48.860+0000 7fda669d1700 10 monclient(hunting): picked mon.noname-i con 0x7fda601386e0 addr [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] 2023-09-30T01:36:48.880 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:48.860+0000 7fda669d1700 10 monclient(hunting): start opening mon connection 2023-09-30T01:36:48.880 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:48.860+0000 7fda669d1700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda6009a160 con 0x7fda600a3380 2023-09-30T01:36:48.880 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:48.860+0000 7fda669d1700 10 monclient(hunting): start opening mon connection 2023-09-30T01:36:48.880 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:48.860+0000 7fda669d1700 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda6009fb00 con 0x7fda6009ffd0 2023-09-30T01:36:48.881 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:48.860+0000 7fda669d1700 10 monclient(hunting): start opening mon connection 2023-09-30T01:36:48.881 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:48.860+0000 7fda669d1700 1 -- --> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] -- mon_getmap magic: 0 v1 -- 0x7fda600a0ac0 con 0x7fda601386e0 2023-09-30T01:36:48.881 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:48.860+0000 7fda669d1700 10 monclient(hunting): _renew_subs 2023-09-30T01:36:48.881 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:48.860+0000 7fda669d1700 10 monclient(hunting): authenticate will time out at 1080.902045s 2023-09-30T01:36:48.881 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:48.860+0000 7fda659cf700 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fda6009ffd0 0x7fda60135440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:36:48.881 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:48.860+0000 7fda651ce700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fda600a3380 0x7fda60135980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:36:48.881 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:48.860+0000 7fda659cf700 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fda6009ffd0 0x7fda60135440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3301/0 says I am v2:172.21.15.88:60414/0 (socket says 172.21.15.88:60414) 2023-09-30T01:36:48.881 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:48.860+0000 7fda659cf700 1 -- 172.21.15.88:0/490054793 learned_addr learned my addr 172.21.15.88:0/490054793 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-09-30T01:36:48.881 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:48.860+0000 7fda659cf700 10 monclient(hunting): get_auth_request con 0x7fda6009ffd0 auth_method 0 2023-09-30T01:36:48.881 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:48.860+0000 7fda659cf700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:36:48.881 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:48.860+0000 7fda659cf700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:36:48.881 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:48.860+0000 7fda659cf700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:36:48.882 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:48.860+0000 7fda651ce700 10 monclient(hunting): get_auth_request con 0x7fda600a3380 auth_method 0 2023-09-30T01:36:48.882 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:48.860+0000 7fda651ce700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:36:48.882 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:48.860+0000 7fda651ce700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:36:48.882 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:48.860+0000 7fda651ce700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:36:48.882 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:48.860+0000 7fda661d0700 1 --2- 172.21.15.88:0/490054793 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7fda601386e0 0x7fda60135ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:36:48.882 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:48.860+0000 7fda661d0700 10 monclient(hunting): get_auth_request con 0x7fda601386e0 auth_method 0 2023-09-30T01:36:48.882 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:48.860+0000 7fda661d0700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:36:48.882 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:48.860+0000 7fda661d0700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:36:48.882 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:48.860+0000 7fda661d0700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:36:48.882 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:48.860+0000 7fda659cf700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:36:48.882 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:48.860+0000 7fda659cf700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:36:48.882 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:48.860+0000 7fda659cf700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:36:48.883 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:48.860+0000 7fda651ce700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:36:48.883 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:48.860+0000 7fda651ce700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:36:48.883 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:48.860+0000 7fda651ce700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:36:48.883 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:48.860+0000 7fda661d0700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:36:48.883 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:48.860+0000 7fda661d0700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:36:48.883 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:48.860+0000 7fda661d0700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:36:48.883 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:48.860+0000 7fda659cf700 10 monclient(hunting): handle_auth_done global_id 4700 payload 824 2023-09-30T01:36:48.883 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:48.860+0000 7fda659cf700 1 -- 172.21.15.88:0/490054793 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7fda601386e0 msgr2=0x7fda60135ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:36:48.883 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:48.860+0000 7fda659cf700 1 --2- 172.21.15.88:0/490054793 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7fda601386e0 0x7fda60135ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:36:48.883 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:48.860+0000 7fda659cf700 1 -- 172.21.15.88:0/490054793 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fda600a3380 msgr2=0x7fda60135980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:36:48.883 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:48.860+0000 7fda659cf700 1 --2- 172.21.15.88:0/490054793 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fda600a3380 0x7fda60135980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:36:48.883 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:48.860+0000 7fda659cf700 10 monclient: _finish_hunting 0 2023-09-30T01:36:48.883 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:48.860+0000 7fda659cf700 1 monclient: found mon.noname-b 2023-09-30T01:36:48.884 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:48.860+0000 7fda659cf700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:36:48.884 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:48.860+0000 7fda659cf700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.88:3301/0 2023-09-30T01:36:48.884 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:48.860+0000 7fda659cf700 1 -- 172.21.15.88:0/490054793 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda580094f0 con 0x7fda6009ffd0 2023-09-30T01:36:48.884 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:48.860+0000 7fda659cf700 1 --2- 172.21.15.88:0/490054793 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fda6009ffd0 0x7fda60135440 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fda5000c870 tx=0x7fda50007d90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:36:48.884 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:48.860+0000 7fda55ffb700 1 -- 172.21.15.88:0/490054793 <== mon.2 v2:172.21.15.88:3301/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7fda500070d0 con 0x7fda6009ffd0 2023-09-30T01:36:48.884 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:48.860+0000 7fda55ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:36:48.884 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:48.860+0000 7fda55ffb700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T01:36:48.884 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:48.860+0000 7fda55ffb700 10 monclient: dump: 2023-09-30T01:36:48.884 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:36:48.884 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:36:48.884 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:36:48.884 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:36:48.885 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:36:48.885 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:36:48.885 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:36:48.885 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:36:48.885 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:36:48.885 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:36:48.885 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:36:48.885 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:36:48.885 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:36:48.885 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:36:48.885 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:36:48.885 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:36:48.885 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:48.860+0000 7fda55ffb700 10 monclient: _finish_auth 0 2023-09-30T01:36:48.886 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:48.860+0000 7fda55ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:36:48.886 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:48.860+0000 7fda55ffb700 1 -- 172.21.15.88:0/490054793 <== mon.2 v2:172.21.15.88:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda50007250 con 0x7fda6009ffd0 2023-09-30T01:36:48.886 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:48.860+0000 7fda669d1700 5 monclient: authenticate success, global_id 4700 2023-09-30T01:36:48.886 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:48.860+0000 7fda669d1700 10 monclient: _renew_subs 2023-09-30T01:36:48.886 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:48.860+0000 7fda669d1700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.88:3301/0 2023-09-30T01:36:48.886 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:48.860+0000 7fda669d1700 1 -- 172.21.15.88:0/490054793 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda6009a420 con 0x7fda6009ffd0 2023-09-30T01:36:48.886 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:48.860+0000 7fda55ffb700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:36:48.886 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:48.860+0000 7fda55ffb700 1 -- 172.21.15.88:0/490054793 <== mon.2 v2:172.21.15.88:3301/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7fda50011360 con 0x7fda6009ffd0 2023-09-30T01:36:48.886 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:48.860+0000 7fda55ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:36:48.886 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:48.860+0000 7fda55ffb700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T01:36:48.886 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:48.860+0000 7fda577fe700 4 set_mon_vals no callback set 2023-09-30T01:36:48.886 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:48.860+0000 7fda55ffb700 10 monclient: dump: 2023-09-30T01:36:48.887 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:36:48.887 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:36:48.887 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:36:48.887 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:36:48.887 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:36:48.887 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:36:48.887 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:36:48.887 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:36:48.887 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:36:48.887 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:36:48.887 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:36:48.887 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:36:48.887 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:36:48.888 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:36:48.888 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:36:48.888 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:36:48.888 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:48.864+0000 7fda669d1700 10 monclient: _renew_subs 2023-09-30T01:36:48.888 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:48.864+0000 7fda669d1700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.88:3301/0 2023-09-30T01:36:48.888 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:48.864+0000 7fda669d1700 1 -- 172.21.15.88:0/490054793 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda60099b10 con 0x7fda6009ffd0 2023-09-30T01:36:48.888 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:48.864+0000 7fda55ffb700 1 -- 172.21.15.88:0/490054793 <== mon.2 v2:172.21.15.88:3301/0 4 ==== mgrmap(e 5) v1 ==== 40144+0+0 (secure 0 0 0) 0x7fda500073c0 con 0x7fda6009ffd0 2023-09-30T01:36:48.888 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:48.864+0000 7fda669d1700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:36:48.888 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:48.864+0000 7fda669d1700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:36:48.888 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:48.864+0000 7fda669d1700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.88:3301/0 2023-09-30T01:36:48.888 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:48.864+0000 7fda669d1700 1 -- 172.21.15.88:0/490054793 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fda60097d40 con 0x7fda6009ffd0 2023-09-30T01:36:48.888 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:48.864+0000 7fda55ffb700 1 --2- 172.21.15.88:0/490054793 >> v2:172.21.15.107:6812/29926 conn(0x7fda44033360 0x7fda44035810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:36:48.889 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:48.864+0000 7fda55ffb700 1 -- 172.21.15.88:0/490054793 <== mon.2 v2:172.21.15.88:3301/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 3502+0+0 (secure 0 0 0) 0x7fda50042310 con 0x7fda6009ffd0 2023-09-30T01:36:48.889 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:48.864+0000 7fda651ce700 1 --2- 172.21.15.88:0/490054793 >> v2:172.21.15.107:6812/29926 conn(0x7fda44033360 0x7fda44035810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:36:48.889 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:48.864+0000 7fda651ce700 10 monclient: get_auth_request con 0x7fda44033360 auth_method 0 2023-09-30T01:36:48.889 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:48.864+0000 7fda651ce700 1 --2- 172.21.15.88:0/490054793 >> v2:172.21.15.107:6812/29926 conn(0x7fda44033360 0x7fda44035810 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fda5c00f570 tx=0x7fda5c00b040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:36:48.889 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:48.872+0000 7fda55ffb700 1 -- 172.21.15.88:0/490054793 <== mon.2 v2:172.21.15.88:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fda50005a70 con 0x7fda6009ffd0 2023-09-30T01:36:48.889 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:48.872+0000 7fda55ffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:36:48.889 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:48.872+0000 7fda55ffb700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:36:49.072 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:49.068+0000 7fda669d1700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T01:36:49.073 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:49.068+0000 7fda669d1700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:36:49.073 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:49.068+0000 7fda669d1700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.88:3301/0 2023-09-30T01:36:49.073 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:49.068+0000 7fda669d1700 1 -- 172.21.15.88:0/490054793 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fda600a5bc0 con 0x7fda6009ffd0 2023-09-30T01:36:49.073 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:49.068+0000 7fda55ffb700 1 -- 172.21.15.88:0/490054793 <== mon.2 v2:172.21.15.88:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7fda50017030 con 0x7fda6009ffd0 2023-09-30T01:36:49.073 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:49.068+0000 7fda55ffb700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:36:49.074 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:49.068+0000 7fda55ffb700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:36:49.074 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-09-30T01:36:49.074 INFO:teuthology.orchestra.run.smithi088.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:36:49.075 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:49.068+0000 7fda669d1700 1 -- 172.21.15.88:0/490054793 >> v2:172.21.15.107:6812/29926 conn(0x7fda44033360 msgr2=0x7fda44035810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:36:49.076 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:49.068+0000 7fda669d1700 1 --2- 172.21.15.88:0/490054793 >> v2:172.21.15.107:6812/29926 conn(0x7fda44033360 0x7fda44035810 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fda5c00f570 tx=0x7fda5c00b040 comp rx=0 tx=0).stop 2023-09-30T01:36:49.076 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:49.068+0000 7fda669d1700 10 monclient: shutdown 2023-09-30T01:36:49.076 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:49.068+0000 7fda669d1700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:36:49.076 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:49.068+0000 7fda669d1700 1 -- 172.21.15.88:0/490054793 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fda6009ffd0 msgr2=0x7fda60135440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:36:49.076 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:49.068+0000 7fda669d1700 1 --2- 172.21.15.88:0/490054793 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fda6009ffd0 0x7fda60135440 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fda5000c870 tx=0x7fda50007d90 comp rx=0 tx=0).stop 2023-09-30T01:36:49.076 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:49.068+0000 7fda669d1700 1 -- 172.21.15.88:0/490054793 shutdown_connections 2023-09-30T01:36:49.076 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:49.068+0000 7fda669d1700 1 --2- 172.21.15.88:0/490054793 >> v2:172.21.15.107:6812/29926 conn(0x7fda44033360 0x7fda44035810 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:36:49.076 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:49.068+0000 7fda669d1700 1 --2- 172.21.15.88:0/490054793 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7fda601386e0 0x7fda60135ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:36:49.076 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:49.068+0000 7fda669d1700 1 --2- 172.21.15.88:0/490054793 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fda600a3380 0x7fda60135980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:36:49.077 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:49.068+0000 7fda669d1700 1 --2- 172.21.15.88:0/490054793 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fda6009ffd0 0x7fda60135440 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:36:49.077 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:49.068+0000 7fda669d1700 1 -- 172.21.15.88:0/490054793 >> 172.21.15.88:0/490054793 conn(0x7fda6006e5e0 msgr2=0x7fda6012e4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:36:49.077 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:49.068+0000 7fda669d1700 1 -- 172.21.15.88:0/490054793 shutdown_connections 2023-09-30T01:36:49.077 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:49.068+0000 7fda669d1700 1 -- 172.21.15.88:0/490054793 wait complete. 2023-09-30T01:36:52.088 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:36:52.198 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:52.192+0000 7fec66a2e700 10 monclient: get_monmap_and_config 2023-09-30T01:36:52.199 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:52.192+0000 7fec66a2e700 10 monclient: build_initial_monmap 2023-09-30T01:36:52.199 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:52.192+0000 7fec66a2e700 10 monclient: monmap: 2023-09-30T01:36:52.199 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2023-09-30T01:36:52.199 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:36:52.199 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:36:52.198484+0000 2023-09-30T01:36:52.199 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:36:52.198484+0000 2023-09-30T01:36:52.199 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2023-09-30T01:36:52.199 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:36:52.199 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a-legacy 2023-09-30T01:36:52.199 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.107:6789/0 mon.noname-f-legacy 2023-09-30T01:36:52.199 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.noname-b 2023-09-30T01:36:52.200 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.noname-g 2023-09-30T01:36:52.200 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.noname-c 2023-09-30T01:36:52.200 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.noname-h 2023-09-30T01:36:52.200 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.noname-d 2023-09-30T01:36:52.200 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.noname-i 2023-09-30T01:36:52.200 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.noname-e 2023-09-30T01:36:52.200 INFO:teuthology.orchestra.run.smithi088.stderr:9: v2:172.21.15.88:3300/0 mon.noname-a 2023-09-30T01:36:52.200 INFO:teuthology.orchestra.run.smithi088.stderr:10: v2:172.21.15.107:3300/0 mon.noname-f 2023-09-30T01:36:52.200 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:36:52.208 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:52.192+0000 7fec66a2e700 1 Processor -- start 2023-09-30T01:36:52.208 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:52.192+0000 7fec66a2e700 1 -- start start 2023-09-30T01:36:52.208 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:52.192+0000 7fec66a2e700 10 monclient: init 2023-09-30T01:36:52.209 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:52.196+0000 7fec66a2e700 10 monclient: _reopen_session rank -1 2023-09-30T01:36:52.209 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:52.196+0000 7fec66a2e700 10 monclient: _add_conns ranks=[1,6,7,2,3,8,9,0,4,5,10] 2023-09-30T01:36:52.209 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:52.196+0000 7fec66a2e700 10 monclient(hunting): picked mon.noname-f-legacy con 0x7fec600a17d0 addr v1:172.21.15.107:6789/0 2023-09-30T01:36:52.209 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:52.196+0000 7fec66a2e700 1 --2- >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7fec6006ea20 0x7fec6006ee50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:36:52.209 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:52.196+0000 7fec66a2e700 10 monclient(hunting): picked mon.noname-d con 0x7fec6006ea20 addr [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] 2023-09-30T01:36:52.209 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:52.196+0000 7fec66a2e700 1 --2- >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7fec6006f390 0x7fec601384c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:36:52.209 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:52.196+0000 7fec66a2e700 10 monclient(hunting): picked mon.noname-i con 0x7fec6006f390 addr [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] 2023-09-30T01:36:52.209 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:52.196+0000 7fec66a2e700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec6004bb20 con 0x7fec600a17d0 2023-09-30T01:36:52.209 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:52.196+0000 7fec66a2e700 10 monclient(hunting): start opening mon connection 2023-09-30T01:36:52.209 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:52.196+0000 7fec66a2e700 1 -- --> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] -- mon_getmap magic: 0 v1 -- 0x7fec6006c610 con 0x7fec6006ea20 2023-09-30T01:36:52.209 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:52.196+0000 7fec66a2e700 10 monclient(hunting): start opening mon connection 2023-09-30T01:36:52.210 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:52.196+0000 7fec66a2e700 1 -- --> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] -- mon_getmap magic: 0 v1 -- 0x7fec60131cf0 con 0x7fec6006f390 2023-09-30T01:36:52.210 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:52.196+0000 7fec66a2e700 10 monclient(hunting): _renew_subs 2023-09-30T01:36:52.210 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:52.196+0000 7fec66a2e700 10 monclient(hunting): authenticate will time out at 1084.236516s 2023-09-30T01:36:52.210 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:52.196+0000 7fec6522b700 1 --2- >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7fec6006ea20 0x7fec6006ee50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:36:52.210 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:52.196+0000 7fec6522b700 1 --2- >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7fec6006ea20 0x7fec6006ee50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3303/0 says I am v2:172.21.15.88:58658/0 (socket says 172.21.15.88:58658) 2023-09-30T01:36:52.210 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:52.196+0000 7fec6522b700 1 -- 172.21.15.88:0/3708313999 learned_addr learned my addr 172.21.15.88:0/3708313999 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-09-30T01:36:52.210 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:52.196+0000 7fec6522b700 10 monclient(hunting): get_auth_request con 0x7fec6006ea20 auth_method 0 2023-09-30T01:36:52.210 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:52.196+0000 7fec6522b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:36:52.210 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:52.196+0000 7fec6522b700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:36:52.210 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:52.196+0000 7fec6522b700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:36:52.211 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:52.196+0000 7fec6622d700 1 --2- 172.21.15.88:0/3708313999 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7fec6006f390 0x7fec601384c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:36:52.211 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:52.196+0000 7fec6622d700 10 monclient(hunting): get_auth_request con 0x7fec6006f390 auth_method 0 2023-09-30T01:36:52.211 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:52.196+0000 7fec6622d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:36:52.211 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:52.196+0000 7fec6622d700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:36:52.211 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:52.196+0000 7fec6622d700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:36:52.211 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:52.196+0000 7fec6522b700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:36:52.211 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:52.196+0000 7fec6522b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:36:52.211 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:52.196+0000 7fec6522b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:36:52.211 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:52.196+0000 7fec6622d700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:36:52.211 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:52.196+0000 7fec6622d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:36:52.211 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:52.196+0000 7fec6622d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:36:52.212 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:52.196+0000 7fec6522b700 10 monclient(hunting): handle_auth_done global_id 4623 payload 274 2023-09-30T01:36:52.212 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:52.196+0000 7fec6522b700 1 -- 172.21.15.88:0/3708313999 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7fec6006f390 msgr2=0x7fec601384c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:36:52.212 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:52.196+0000 7fec6522b700 1 --2- 172.21.15.88:0/3708313999 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7fec6006f390 0x7fec601384c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:36:52.212 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:52.196+0000 7fec6522b700 1 -- 172.21.15.88:0/3708313999 >> v1:172.21.15.107:6789/0 conn(0x7fec600a17d0 legacy=0x7fec601315e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:36:52.212 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:52.196+0000 7fec64a2a700 1 -- 172.21.15.88:0/3708313999 <== mon.1 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 301876628 0 0) 0x7fec6004bb20 con 0x7fec600a17d0 2023-09-30T01:36:52.212 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:52.196+0000 7fec6522b700 10 monclient: _finish_hunting 0 2023-09-30T01:36:52.212 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:52.196+0000 7fec6522b700 1 monclient: found mon.noname-d 2023-09-30T01:36:52.212 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:52.196+0000 7fec6522b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:36:52.212 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:52.196+0000 7fec6522b700 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.88:3303/0 2023-09-30T01:36:52.212 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:52.196+0000 7fec6522b700 1 -- 172.21.15.88:0/3708313999 --> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec6006cc50 con 0x7fec6006ea20 2023-09-30T01:36:52.212 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:52.196+0000 7fec64a2a700 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2023-09-30T01:36:52.213 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:52.196+0000 7fec6522b700 1 --2- 172.21.15.88:0/3708313999 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7fec6006ea20 0x7fec6006ee50 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fec5c0099e0 tx=0x7fec5c009250 comp rx=0 tx=0).ready entity=mon.6 client_cookie=7b8f6204969cab6f server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:36:52.213 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:52.196+0000 7fec64a2a700 1 -- 172.21.15.88:0/3708313999 <== mon.6 v2:172.21.15.88:3303/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7fec5c00bce0 con 0x7fec6006ea20 2023-09-30T01:36:52.213 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:52.196+0000 7fec64a2a700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:36:52.213 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:52.196+0000 7fec64a2a700 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2023-09-30T01:36:52.213 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:52.196+0000 7fec64a2a700 10 monclient: dump: 2023-09-30T01:36:52.213 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:36:52.213 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:36:52.213 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:36:52.213 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:36:52.213 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:36:52.213 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:36:52.213 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:36:52.213 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:36:52.214 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:36:52.214 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:36:52.214 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:36:52.214 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:36:52.214 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:36:52.214 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:36:52.214 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:36:52.214 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:36:52.214 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:52.196+0000 7fec64a2a700 10 monclient: _finish_auth 0 2023-09-30T01:36:52.214 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:52.196+0000 7fec64a2a700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:36:52.214 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:52.196+0000 7fec66a2e700 5 monclient: authenticate success, global_id 4623 2023-09-30T01:36:52.214 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:52.196+0000 7fec66a2e700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:36:52.215 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:52.196+0000 7fec64a2a700 1 -- 172.21.15.88:0/3708313999 <== mon.6 v2:172.21.15.88:3303/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec5c00be60 con 0x7fec6006ea20 2023-09-30T01:36:52.215 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:52.196+0000 7fec64a2a700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:36:52.215 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:52.196+0000 7fec66a2e700 10 monclient: get_monmap_and_config success 2023-09-30T01:36:52.215 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:52.196+0000 7fec66a2e700 4 set_mon_vals no callback set 2023-09-30T01:36:52.215 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:52.196+0000 7fec64a2a700 1 -- 172.21.15.88:0/3708313999 <== mon.6 v2:172.21.15.88:3303/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7fec5c0118a0 con 0x7fec6006ea20 2023-09-30T01:36:52.215 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:52.196+0000 7fec64a2a700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:36:52.215 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:52.196+0000 7fec64a2a700 10 monclient: got monmap 1 from mon.d (according to old e1) 2023-09-30T01:36:52.215 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:52.196+0000 7fec64a2a700 10 monclient: dump: 2023-09-30T01:36:52.215 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:36:52.215 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:36:52.215 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:36:52.215 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:36:52.216 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:36:52.216 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:36:52.216 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:36:52.216 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:36:52.216 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:36:52.216 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:36:52.216 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:36:52.216 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:36:52.216 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:36:52.216 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:36:52.216 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:36:52.216 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:36:52.217 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:52.196+0000 7fec66a2e700 10 monclient: shutdown 2023-09-30T01:36:52.217 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:52.196+0000 7fec66a2e700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:36:52.217 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:52.196+0000 7fec66a2e700 1 -- 172.21.15.88:0/3708313999 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7fec6006ea20 msgr2=0x7fec6006ee50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:36:52.217 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:52.196+0000 7fec66a2e700 1 --2- 172.21.15.88:0/3708313999 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7fec6006ea20 0x7fec6006ee50 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fec5c0099e0 tx=0x7fec5c009250 comp rx=0 tx=0).stop 2023-09-30T01:36:52.217 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:52.196+0000 7fec66a2e700 1 -- 172.21.15.88:0/3708313999 shutdown_connections 2023-09-30T01:36:52.217 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:52.196+0000 7fec66a2e700 1 --2- 172.21.15.88:0/3708313999 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7fec6006f390 0x7fec601384c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:36:52.217 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:52.196+0000 7fec66a2e700 1 --2- 172.21.15.88:0/3708313999 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7fec6006ea20 0x7fec6006ee50 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:36:52.217 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:52.196+0000 7fec66a2e700 1 -- 172.21.15.88:0/3708313999 >> 172.21.15.88:0/3708313999 conn(0x7fec60097a40 msgr2=0x7fec600a0ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:36:52.217 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:52.196+0000 7fec66a2e700 1 -- 172.21.15.88:0/3708313999 shutdown_connections 2023-09-30T01:36:52.217 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:52.196+0000 7fec66a2e700 1 -- 172.21.15.88:0/3708313999 wait complete. 2023-09-30T01:36:52.217 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:52.196+0000 7fec66a2e700 10 monclient: build_initial_monmap 2023-09-30T01:36:52.217 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:52.196+0000 7fec66a2e700 10 monclient: monmap: 2023-09-30T01:36:52.218 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2023-09-30T01:36:52.218 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:36:52.218 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 0.000000 2023-09-30T01:36:52.218 INFO:teuthology.orchestra.run.smithi088.stderr:created 0.000000 2023-09-30T01:36:52.218 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2023-09-30T01:36:52.218 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 1 2023-09-30T01:36:52.218 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.noname-a 2023-09-30T01:36:52.218 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.noname-f 2023-09-30T01:36:52.218 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.noname-b 2023-09-30T01:36:52.218 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.noname-g 2023-09-30T01:36:52.218 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.noname-c 2023-09-30T01:36:52.219 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.noname-h 2023-09-30T01:36:52.219 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.noname-d 2023-09-30T01:36:52.219 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.noname-i 2023-09-30T01:36:52.219 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.noname-e 2023-09-30T01:36:52.219 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:36:52.219 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:52.200+0000 7fec66a2e700 1 Processor -- start 2023-09-30T01:36:52.219 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:52.200+0000 7fec66a2e700 1 -- start start 2023-09-30T01:36:52.219 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:52.200+0000 7fec66a2e700 10 monclient: init 2023-09-30T01:36:52.219 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:52.200+0000 7fec66a2e700 10 monclient: _reopen_session rank -1 2023-09-30T01:36:52.219 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:52.200+0000 7fec66a2e700 10 monclient: _add_conns ranks=[4,3,1,5,2,6,7,8,0] 2023-09-30T01:36:52.219 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:52.200+0000 7fec66a2e700 1 --2- >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7fec6006ea20 0x7fec601405a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:36:52.220 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:52.200+0000 7fec66a2e700 10 monclient(hunting): picked mon.noname-c con 0x7fec6006ea20 addr [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] 2023-09-30T01:36:52.220 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:52.200+0000 7fec66a2e700 1 --2- >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7fec6006f390 0x7fec60140ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:36:52.220 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:52.200+0000 7fec66a2e700 10 monclient(hunting): picked mon.noname-g con 0x7fec6006f390 addr [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] 2023-09-30T01:36:52.220 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:52.200+0000 7fec66a2e700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fec6012dd60 0x7fec6012c1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:36:52.220 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:52.200+0000 7fec66a2e700 10 monclient(hunting): picked mon.noname-f con 0x7fec6012dd60 addr [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] 2023-09-30T01:36:52.220 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:52.200+0000 7fec66a2e700 10 monclient(hunting): start opening mon connection 2023-09-30T01:36:52.220 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:52.200+0000 7fec66a2e700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec6011aea0 con 0x7fec6012dd60 2023-09-30T01:36:52.220 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:52.200+0000 7fec66a2e700 10 monclient(hunting): start opening mon connection 2023-09-30T01:36:52.220 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:52.200+0000 7fec66a2e700 1 -- --> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec6011be90 con 0x7fec6006f390 2023-09-30T01:36:52.220 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:52.200+0000 7fec66a2e700 10 monclient(hunting): start opening mon connection 2023-09-30T01:36:52.220 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:52.200+0000 7fec66a2e700 1 -- --> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] -- mon_getmap magic: 0 v1 -- 0x7fec6011abc0 con 0x7fec6006ea20 2023-09-30T01:36:52.220 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:52.200+0000 7fec66a2e700 10 monclient(hunting): _renew_subs 2023-09-30T01:36:52.221 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:52.200+0000 7fec66a2e700 10 monclient(hunting): authenticate will time out at 1084.241003s 2023-09-30T01:36:52.221 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:52.200+0000 7fec65a2c700 1 --2- >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7fec6006ea20 0x7fec601405a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:36:52.221 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:52.200+0000 7fec65a2c700 1 --2- >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7fec6006ea20 0x7fec601405a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3302/0 says I am v2:172.21.15.88:53008/0 (socket says 172.21.15.88:53008) 2023-09-30T01:36:52.221 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:52.200+0000 7fec65a2c700 1 -- 172.21.15.88:0/1461721885 learned_addr learned my addr 172.21.15.88:0/1461721885 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-09-30T01:36:52.221 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:52.200+0000 7fec65a2c700 10 monclient(hunting): get_auth_request con 0x7fec6006ea20 auth_method 0 2023-09-30T01:36:52.221 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:52.200+0000 7fec65a2c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:36:52.221 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:52.200+0000 7fec65a2c700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:36:52.221 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:52.200+0000 7fec65a2c700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:36:52.221 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:52.200+0000 7fec65a2c700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:36:52.221 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:52.200+0000 7fec65a2c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:36:52.221 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:52.200+0000 7fec65a2c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:36:52.221 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:52.200+0000 7fec6522b700 1 --2- 172.21.15.88:0/1461721885 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7fec6006f390 0x7fec60140ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:36:52.222 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:52.200+0000 7fec6522b700 10 monclient(hunting): get_auth_request con 0x7fec6006f390 auth_method 0 2023-09-30T01:36:52.222 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:52.200+0000 7fec6522b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:36:52.222 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:52.200+0000 7fec6522b700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:36:52.222 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:52.200+0000 7fec6522b700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:36:52.222 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:52.200+0000 7fec6622d700 1 --2- 172.21.15.88:0/1461721885 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fec6012dd60 0x7fec6012c1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:36:52.222 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:52.200+0000 7fec6622d700 10 monclient(hunting): get_auth_request con 0x7fec6012dd60 auth_method 0 2023-09-30T01:36:52.222 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:52.200+0000 7fec6622d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:36:52.222 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:52.200+0000 7fec6622d700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:36:52.222 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:52.200+0000 7fec6622d700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:36:52.222 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:52.200+0000 7fec6522b700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:36:52.222 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:52.200+0000 7fec6522b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:36:52.223 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:52.200+0000 7fec6522b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:36:52.223 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:52.200+0000 7fec65a2c700 10 monclient(hunting): handle_auth_done global_id 4747 payload 824 2023-09-30T01:36:52.223 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:52.200+0000 7fec65a2c700 1 -- 172.21.15.88:0/1461721885 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7fec6006f390 msgr2=0x7fec60140ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:36:52.223 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:52.200+0000 7fec65a2c700 1 --2- 172.21.15.88:0/1461721885 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7fec6006f390 0x7fec60140ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:36:52.223 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:52.200+0000 7fec65a2c700 1 -- 172.21.15.88:0/1461721885 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fec6012dd60 msgr2=0x7fec6012c1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:36:52.223 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:52.200+0000 7fec65a2c700 1 --2- 172.21.15.88:0/1461721885 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fec6012dd60 0x7fec6012c1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:36:52.223 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:52.200+0000 7fec65a2c700 10 monclient: _finish_hunting 0 2023-09-30T01:36:52.223 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:52.200+0000 7fec65a2c700 1 monclient: found mon.noname-c 2023-09-30T01:36:52.223 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:52.200+0000 7fec65a2c700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:36:52.223 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:52.200+0000 7fec65a2c700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.88:3302/0 2023-09-30T01:36:52.223 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:52.200+0000 7fec65a2c700 1 -- 172.21.15.88:0/1461721885 --> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec5c0094f0 con 0x7fec6006ea20 2023-09-30T01:36:52.223 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:52.200+0000 7fec65a2c700 1 --2- 172.21.15.88:0/1461721885 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7fec6006ea20 0x7fec601405a0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fec5400c590 tx=0x7fec54007d40 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:36:52.224 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:52.200+0000 7fec51ffb700 1 -- 172.21.15.88:0/1461721885 <== mon.4 v2:172.21.15.88:3302/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7fec54011a10 con 0x7fec6006ea20 2023-09-30T01:36:52.224 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:52.200+0000 7fec51ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:36:52.224 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:52.200+0000 7fec51ffb700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T01:36:52.224 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:52.200+0000 7fec51ffb700 10 monclient: dump: 2023-09-30T01:36:52.224 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:36:52.224 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:36:52.224 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:36:52.224 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:36:52.224 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:36:52.224 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:36:52.224 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:36:52.224 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:36:52.225 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:36:52.225 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:36:52.225 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:36:52.225 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:36:52.225 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:36:52.225 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:36:52.225 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:36:52.225 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:36:52.225 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:52.200+0000 7fec51ffb700 10 monclient: _finish_auth 0 2023-09-30T01:36:52.226 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:52.200+0000 7fec51ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:36:52.226 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:52.200+0000 7fec66a2e700 5 monclient: authenticate success, global_id 4747 2023-09-30T01:36:52.226 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:52.200+0000 7fec66a2e700 10 monclient: _renew_subs 2023-09-30T01:36:52.226 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:52.200+0000 7fec51ffb700 1 -- 172.21.15.88:0/1461721885 <== mon.4 v2:172.21.15.88:3302/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec54011bb0 con 0x7fec6006ea20 2023-09-30T01:36:52.226 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:52.200+0000 7fec66a2e700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.88:3302/0 2023-09-30T01:36:52.226 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:52.200+0000 7fec66a2e700 1 -- 172.21.15.88:0/1461721885 --> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec6011b180 con 0x7fec6006ea20 2023-09-30T01:36:52.226 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:52.200+0000 7fec51ffb700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:36:52.226 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:52.200+0000 7fec51ffb700 1 -- 172.21.15.88:0/1461721885 <== mon.4 v2:172.21.15.88:3302/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7fec54011d60 con 0x7fec6006ea20 2023-09-30T01:36:52.226 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:52.200+0000 7fec51ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:36:52.227 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:52.200+0000 7fec51ffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T01:36:52.227 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:52.200+0000 7fec53fff700 4 set_mon_vals no callback set 2023-09-30T01:36:52.227 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:52.200+0000 7fec51ffb700 10 monclient: dump: 2023-09-30T01:36:52.227 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:36:52.227 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:36:52.227 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:36:52.227 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:36:52.227 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:36:52.227 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:36:52.227 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:36:52.227 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:36:52.227 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:36:52.228 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:36:52.228 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:36:52.228 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:36:52.228 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:36:52.228 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:36:52.228 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:36:52.228 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:36:52.228 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:52.200+0000 7fec66a2e700 10 monclient: _renew_subs 2023-09-30T01:36:52.228 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:52.200+0000 7fec66a2e700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.88:3302/0 2023-09-30T01:36:52.228 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:52.200+0000 7fec66a2e700 1 -- 172.21.15.88:0/1461721885 --> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec60141020 con 0x7fec6006ea20 2023-09-30T01:36:52.228 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:52.200+0000 7fec51ffb700 1 -- 172.21.15.88:0/1461721885 <== mon.4 v2:172.21.15.88:3302/0 4 ==== mgrmap(e 5) v1 ==== 40144+0+0 (secure 0 0 0) 0x7fec54011590 con 0x7fec6006ea20 2023-09-30T01:36:52.228 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:52.200+0000 7fec66a2e700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:36:52.229 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:52.200+0000 7fec66a2e700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:36:52.229 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:52.200+0000 7fec66a2e700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.88:3302/0 2023-09-30T01:36:52.229 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:52.200+0000 7fec51ffb700 1 --2- 172.21.15.88:0/1461721885 >> v2:172.21.15.107:6812/29926 conn(0x7fec44033360 0x7fec44035810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:36:52.229 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:52.200+0000 7fec66a2e700 1 -- 172.21.15.88:0/1461721885 --> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fec60071960 con 0x7fec6006ea20 2023-09-30T01:36:52.229 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:52.200+0000 7fec51ffb700 1 -- 172.21.15.88:0/1461721885 <== mon.4 v2:172.21.15.88:3302/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 3502+0+0 (secure 0 0 0) 0x7fec54042560 con 0x7fec6006ea20 2023-09-30T01:36:52.229 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:52.204+0000 7fec6522b700 1 --2- 172.21.15.88:0/1461721885 >> v2:172.21.15.107:6812/29926 conn(0x7fec44033360 0x7fec44035810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:36:52.229 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:52.204+0000 7fec6522b700 10 monclient: get_auth_request con 0x7fec44033360 auth_method 0 2023-09-30T01:36:52.229 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:52.204+0000 7fec6522b700 1 --2- 172.21.15.88:0/1461721885 >> v2:172.21.15.107:6812/29926 conn(0x7fec44033360 0x7fec44035810 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fec5c00bfd0 tx=0x7fec5c017000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:36:52.230 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:52.208+0000 7fec51ffb700 1 -- 172.21.15.88:0/1461721885 <== mon.4 v2:172.21.15.88:3302/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fec54017a90 con 0x7fec6006ea20 2023-09-30T01:36:52.230 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:52.208+0000 7fec51ffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:36:52.230 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:52.208+0000 7fec51ffb700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:36:52.405 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:52.400+0000 7fec66a2e700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T01:36:52.405 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:52.400+0000 7fec66a2e700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:36:52.405 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:52.400+0000 7fec66a2e700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.88:3302/0 2023-09-30T01:36:52.405 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:52.400+0000 7fec66a2e700 1 -- 172.21.15.88:0/1461721885 --> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fec60067b40 con 0x7fec6006ea20 2023-09-30T01:36:52.406 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:52.400+0000 7fec51ffb700 1 -- 172.21.15.88:0/1461721885 <== mon.4 v2:172.21.15.88:3302/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7fec54017030 con 0x7fec6006ea20 2023-09-30T01:36:52.406 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:52.400+0000 7fec51ffb700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:36:52.406 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:52.400+0000 7fec51ffb700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:36:52.406 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-09-30T01:36:52.407 INFO:teuthology.orchestra.run.smithi088.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:36:52.411 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:52.404+0000 7fec66a2e700 1 -- 172.21.15.88:0/1461721885 >> v2:172.21.15.107:6812/29926 conn(0x7fec44033360 msgr2=0x7fec44035810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:36:52.411 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:52.404+0000 7fec66a2e700 1 --2- 172.21.15.88:0/1461721885 >> v2:172.21.15.107:6812/29926 conn(0x7fec44033360 0x7fec44035810 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fec5c00bfd0 tx=0x7fec5c017000 comp rx=0 tx=0).stop 2023-09-30T01:36:52.411 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:52.404+0000 7fec66a2e700 10 monclient: shutdown 2023-09-30T01:36:52.411 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:52.404+0000 7fec66a2e700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:36:52.411 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:52.404+0000 7fec66a2e700 1 -- 172.21.15.88:0/1461721885 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7fec6006ea20 msgr2=0x7fec601405a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:36:52.411 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:52.404+0000 7fec66a2e700 1 --2- 172.21.15.88:0/1461721885 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7fec6006ea20 0x7fec601405a0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fec5400c590 tx=0x7fec54007d40 comp rx=0 tx=0).stop 2023-09-30T01:36:52.411 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:52.404+0000 7fec66a2e700 1 -- 172.21.15.88:0/1461721885 shutdown_connections 2023-09-30T01:36:52.411 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:52.404+0000 7fec66a2e700 1 --2- 172.21.15.88:0/1461721885 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fec6012dd60 0x7fec6012c1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:36:52.411 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:52.404+0000 7fec66a2e700 1 --2- 172.21.15.88:0/1461721885 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7fec6006f390 0x7fec60140ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:36:52.412 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:52.404+0000 7fec66a2e700 1 --2- 172.21.15.88:0/1461721885 >> v2:172.21.15.107:6812/29926 conn(0x7fec44033360 0x7fec44035810 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:36:52.412 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:52.404+0000 7fec66a2e700 1 --2- 172.21.15.88:0/1461721885 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7fec6006ea20 0x7fec601405a0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:36:52.412 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:52.404+0000 7fec66a2e700 1 -- 172.21.15.88:0/1461721885 >> 172.21.15.88:0/1461721885 conn(0x7fec60097a40 msgr2=0x7fec6009ecf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:36:52.412 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:52.404+0000 7fec66a2e700 1 -- 172.21.15.88:0/1461721885 shutdown_connections 2023-09-30T01:36:52.412 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:52.404+0000 7fec66a2e700 1 -- 172.21.15.88:0/1461721885 wait complete. 2023-09-30T01:36:55.424 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:36:55.535 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:55.528+0000 7f1597fff700 10 monclient: get_monmap_and_config 2023-09-30T01:36:55.535 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:55.528+0000 7f1597fff700 10 monclient: build_initial_monmap 2023-09-30T01:36:55.535 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:55.528+0000 7f1597fff700 10 monclient: monmap: 2023-09-30T01:36:55.536 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2023-09-30T01:36:55.536 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:36:55.536 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:36:55.535283+0000 2023-09-30T01:36:55.536 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:36:55.535283+0000 2023-09-30T01:36:55.536 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2023-09-30T01:36:55.536 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:36:55.536 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a-legacy 2023-09-30T01:36:55.536 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.107:6789/0 mon.noname-f-legacy 2023-09-30T01:36:55.536 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.noname-b 2023-09-30T01:36:55.536 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.noname-g 2023-09-30T01:36:55.536 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.noname-c 2023-09-30T01:36:55.536 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.noname-h 2023-09-30T01:36:55.536 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.noname-d 2023-09-30T01:36:55.537 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.noname-i 2023-09-30T01:36:55.537 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.noname-e 2023-09-30T01:36:55.537 INFO:teuthology.orchestra.run.smithi088.stderr:9: v2:172.21.15.88:3300/0 mon.noname-a 2023-09-30T01:36:55.537 INFO:teuthology.orchestra.run.smithi088.stderr:10: v2:172.21.15.107:3300/0 mon.noname-f 2023-09-30T01:36:55.537 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:36:55.540 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:55.532+0000 7f1597fff700 1 Processor -- start 2023-09-30T01:36:55.540 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:55.532+0000 7f1597fff700 1 -- start start 2023-09-30T01:36:55.540 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:55.532+0000 7f1597fff700 10 monclient: init 2023-09-30T01:36:55.541 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:55.532+0000 7f1597fff700 10 monclient: _reopen_session rank -1 2023-09-30T01:36:55.541 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:55.532+0000 7f1597fff700 10 monclient: _add_conns ranks=[5,1,7,0,6,2,10,8,3,4,9] 2023-09-30T01:36:55.541 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:55.532+0000 7f1597fff700 1 --2- >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7f15980a3380 0x7f15980a48b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:36:55.541 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:55.532+0000 7f1597fff700 10 monclient(hunting): picked mon.noname-h con 0x7f15980a3380 addr [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] 2023-09-30T01:36:55.541 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:55.532+0000 7f1597fff700 10 monclient(hunting): picked mon.noname-f-legacy con 0x7f159809ffd0 addr v1:172.21.15.107:6789/0 2023-09-30T01:36:55.541 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:55.532+0000 7f1597fff700 1 --2- >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7f15980a03a0 0x7f1598133ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:36:55.541 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:55.532+0000 7f1597fff700 10 monclient(hunting): picked mon.noname-i con 0x7f15980a03a0 addr [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] 2023-09-30T01:36:55.541 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:55.532+0000 7f1597fff700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f159804bb20 con 0x7f159809ffd0 2023-09-30T01:36:55.541 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:55.532+0000 7f1597fff700 10 monclient(hunting): start opening mon connection 2023-09-30T01:36:55.541 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:55.532+0000 7f1597fff700 1 -- --> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] -- mon_getmap magic: 0 v1 -- 0x7f1598097ab0 con 0x7f15980a3380 2023-09-30T01:36:55.542 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:55.532+0000 7f1597fff700 10 monclient(hunting): start opening mon connection 2023-09-30T01:36:55.542 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:55.532+0000 7f1597fff700 1 -- --> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] -- mon_getmap magic: 0 v1 -- 0x7f1598098480 con 0x7f15980a03a0 2023-09-30T01:36:55.542 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:55.532+0000 7f1597fff700 10 monclient(hunting): _renew_subs 2023-09-30T01:36:55.542 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:55.532+0000 7f1597fff700 10 monclient(hunting): authenticate will time out at 1087.573138s 2023-09-30T01:36:55.542 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:55.532+0000 7f15967fc700 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f159809ffd0 0x7f15981333b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.88:44812/0 (socket says 172.21.15.88:44812) 2023-09-30T01:36:55.542 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:55.532+0000 7f1596ffd700 1 --2- >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7f15980a3380 0x7f15980a48b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:36:55.542 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:55.532+0000 7f15967fc700 1 -- 172.21.15.88:0/1020914340 learned_addr learned my addr 172.21.15.88:0/1020914340 (peer_addr_for_me v1:172.21.15.88:0/0) 2023-09-30T01:36:55.542 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:55.532+0000 7f15977fe700 1 --2- 172.21.15.88:0/1020914340 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7f15980a03a0 0x7f1598133ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:36:55.542 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:55.532+0000 7f1596ffd700 10 monclient(hunting): get_auth_request con 0x7f15980a3380 auth_method 0 2023-09-30T01:36:55.543 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:55.532+0000 7f1596ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:36:55.543 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:55.532+0000 7f1596ffd700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:36:55.543 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:55.532+0000 7f1596ffd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:36:55.543 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:55.532+0000 7f15977fe700 10 monclient(hunting): get_auth_request con 0x7f15980a03a0 auth_method 0 2023-09-30T01:36:55.543 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:55.532+0000 7f15977fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:36:55.543 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:55.532+0000 7f15977fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:36:55.543 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:55.532+0000 7f15977fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:36:55.543 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:55.532+0000 7f15967fc700 1 -- 172.21.15.88:0/1020914340 >> v1:172.21.15.107:6789/0 conn(0x7f159809ffd0 legacy=0x7f15981333b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2023-09-30T01:36:55.543 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:55.532+0000 7f15967fc700 1 -- 172.21.15.88:0/1020914340 >> v1:172.21.15.107:6789/0 conn(0x7f159809ffd0 legacy=0x7f15981333b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-09-30T01:36:55.543 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:55.532+0000 7f15967fc700 1 --1- 172.21.15.88:0/1020914340 >> v1:172.21.15.107:6789/0 conn(0x7f159809ffd0 0x7f15981333b0 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=0).handle_connect_reply_1 read connect reply failed 2023-09-30T01:36:55.543 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:55.532+0000 7f1596ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:36:55.543 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:55.532+0000 7f1596ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:36:55.543 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:55.532+0000 7f1596ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:36:55.544 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:55.532+0000 7f15977fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:36:55.544 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:55.532+0000 7f15977fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:36:55.544 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:55.532+0000 7f15977fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:36:55.544 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:55.532+0000 7f1596ffd700 10 monclient(hunting): handle_auth_done global_id 4721 payload 274 2023-09-30T01:36:55.544 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:55.532+0000 7f1596ffd700 1 -- 172.21.15.88:0/1020914340 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7f15980a03a0 msgr2=0x7f1598133ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:36:55.544 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:55.532+0000 7f1596ffd700 1 --2- 172.21.15.88:0/1020914340 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7f15980a03a0 0x7f1598133ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:36:55.544 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:55.532+0000 7f1596ffd700 1 -- 172.21.15.88:0/1020914340 >> v1:172.21.15.107:6789/0 conn(0x7f159809ffd0 legacy=0x7f15981333b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-09-30T01:36:55.544 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:55.532+0000 7f1596ffd700 10 monclient: _finish_hunting 0 2023-09-30T01:36:55.544 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:55.532+0000 7f1596ffd700 1 monclient: found mon.noname-h 2023-09-30T01:36:55.544 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:55.532+0000 7f1596ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:36:55.544 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:55.532+0000 7f1596ffd700 10 monclient: _send_mon_message to mon.noname-h at v2:172.21.15.107:3302/0 2023-09-30T01:36:55.544 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:55.532+0000 7f1596ffd700 1 -- 172.21.15.88:0/1020914340 --> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15980a3750 con 0x7f15980a3380 2023-09-30T01:36:55.545 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:55.532+0000 7f1596ffd700 1 --2- 172.21.15.88:0/1020914340 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7f15980a3380 0x7f15980a48b0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f1580009910 tx=0x7f15800091a0 comp rx=0 tx=0).ready entity=mon.5 client_cookie=d5a495fcfa573dc4 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:36:55.545 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:55.532+0000 7f1595ffb700 1 -- 172.21.15.88:0/1020914340 <== mon.5 v2:172.21.15.107:3302/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f158000bac0 con 0x7f15980a3380 2023-09-30T01:36:55.545 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:55.532+0000 7f1595ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:36:55.545 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:55.532+0000 7f1595ffb700 10 monclient: got monmap 1 from mon.noname-h (according to old e1) 2023-09-30T01:36:55.545 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:55.532+0000 7f1595ffb700 10 monclient: dump: 2023-09-30T01:36:55.545 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:36:55.545 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:36:55.545 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:36:55.545 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:36:55.545 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:36:55.545 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:36:55.545 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:36:55.546 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:36:55.546 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:36:55.546 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:36:55.546 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:36:55.546 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:36:55.546 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:36:55.546 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:36:55.546 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:36:55.546 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:36:55.546 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:55.532+0000 7f1595ffb700 10 monclient: _finish_auth 0 2023-09-30T01:36:55.546 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:55.532+0000 7f1595ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:36:55.546 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:55.532+0000 7f1597fff700 5 monclient: authenticate success, global_id 4721 2023-09-30T01:36:55.547 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:55.532+0000 7f1597fff700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:36:55.547 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:55.532+0000 7f1595ffb700 1 -- 172.21.15.88:0/1020914340 <== mon.5 v2:172.21.15.107:3302/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f158000bc40 con 0x7f15980a3380 2023-09-30T01:36:55.547 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:55.532+0000 7f1595ffb700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:36:55.547 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:55.532+0000 7f1597fff700 10 monclient: get_monmap_and_config success 2023-09-30T01:36:55.547 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:55.532+0000 7f1597fff700 4 set_mon_vals no callback set 2023-09-30T01:36:55.547 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:55.532+0000 7f1595ffb700 1 -- 172.21.15.88:0/1020914340 <== mon.5 v2:172.21.15.107:3302/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f158000bdb0 con 0x7f15980a3380 2023-09-30T01:36:55.547 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:55.532+0000 7f1595ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:36:55.547 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:55.532+0000 7f1595ffb700 10 monclient: got monmap 1 from mon.h (according to old e1) 2023-09-30T01:36:55.547 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:55.532+0000 7f1595ffb700 10 monclient: dump: 2023-09-30T01:36:55.547 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:36:55.547 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:36:55.547 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:36:55.547 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:36:55.548 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:36:55.548 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:36:55.548 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:36:55.548 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:36:55.548 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:36:55.548 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:36:55.548 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:36:55.548 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:36:55.548 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:36:55.548 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:36:55.548 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:36:55.548 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:36:55.549 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:55.532+0000 7f1597fff700 10 monclient: shutdown 2023-09-30T01:36:55.549 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:55.532+0000 7f1597fff700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:36:55.549 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:55.532+0000 7f1597fff700 1 -- 172.21.15.88:0/1020914340 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7f15980a3380 msgr2=0x7f15980a48b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:36:55.549 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:55.532+0000 7f1597fff700 1 --2- 172.21.15.88:0/1020914340 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7f15980a3380 0x7f15980a48b0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f1580009910 tx=0x7f15800091a0 comp rx=0 tx=0).stop 2023-09-30T01:36:55.549 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:55.532+0000 7f1597fff700 1 -- 172.21.15.88:0/1020914340 shutdown_connections 2023-09-30T01:36:55.549 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:55.532+0000 7f1597fff700 1 --2- 172.21.15.88:0/1020914340 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7f15980a03a0 0x7f1598133ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:36:55.549 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:55.532+0000 7f1597fff700 1 --2- 172.21.15.88:0/1020914340 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7f15980a3380 0x7f15980a48b0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:36:55.549 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:55.532+0000 7f1597fff700 1 -- 172.21.15.88:0/1020914340 >> 172.21.15.88:0/1020914340 conn(0x7f159806e5e0 msgr2=0x7f159806e9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:36:55.549 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:55.532+0000 7f1597fff700 1 -- 172.21.15.88:0/1020914340 shutdown_connections 2023-09-30T01:36:55.549 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:55.532+0000 7f1597fff700 1 -- 172.21.15.88:0/1020914340 wait complete. 2023-09-30T01:36:55.549 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:55.536+0000 7f1597fff700 10 monclient: build_initial_monmap 2023-09-30T01:36:55.549 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:55.536+0000 7f1597fff700 10 monclient: monmap: 2023-09-30T01:36:55.549 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2023-09-30T01:36:55.550 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:36:55.550 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 0.000000 2023-09-30T01:36:55.550 INFO:teuthology.orchestra.run.smithi088.stderr:created 0.000000 2023-09-30T01:36:55.550 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2023-09-30T01:36:55.550 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 1 2023-09-30T01:36:55.550 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.noname-a 2023-09-30T01:36:55.550 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.noname-f 2023-09-30T01:36:55.550 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.noname-b 2023-09-30T01:36:55.550 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.noname-g 2023-09-30T01:36:55.550 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.noname-c 2023-09-30T01:36:55.550 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.noname-h 2023-09-30T01:36:55.551 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.noname-d 2023-09-30T01:36:55.551 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.noname-i 2023-09-30T01:36:55.551 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.noname-e 2023-09-30T01:36:55.551 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:36:55.551 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:55.536+0000 7f1597fff700 1 Processor -- start 2023-09-30T01:36:55.551 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:55.536+0000 7f1597fff700 1 -- start start 2023-09-30T01:36:55.551 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:55.536+0000 7f1597fff700 10 monclient: init 2023-09-30T01:36:55.556 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:55.536+0000 7f1597fff700 10 monclient: _reopen_session rank -1 2023-09-30T01:36:55.556 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:55.536+0000 7f1597fff700 10 monclient: _add_conns ranks=[7,8,0,6,1,4,3,2,5] 2023-09-30T01:36:55.556 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:55.536+0000 7f1597fff700 1 --2- >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7f159809ffd0 0x7f15981425d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:36:55.557 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:55.536+0000 7f1597fff700 10 monclient(hunting): picked mon.noname-i con 0x7f159809ffd0 addr [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] 2023-09-30T01:36:55.557 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:55.536+0000 7f1597fff700 1 --2- >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f15980a03a0 0x7f1598142b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:36:55.557 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:55.536+0000 7f1597fff700 10 monclient(hunting): picked mon.noname-e con 0x7f15980a03a0 addr [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] 2023-09-30T01:36:55.557 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:55.536+0000 7f1597fff700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f15980a3380 0x7f1598143050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:36:55.557 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:55.536+0000 7f1597fff700 10 monclient(hunting): picked mon.noname-a con 0x7f15980a3380 addr [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] 2023-09-30T01:36:55.557 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:55.536+0000 7f1597fff700 10 monclient(hunting): start opening mon connection 2023-09-30T01:36:55.557 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:55.536+0000 7f1597fff700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1598120210 con 0x7f15980a3380 2023-09-30T01:36:55.557 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:55.536+0000 7f1597fff700 10 monclient(hunting): start opening mon connection 2023-09-30T01:36:55.557 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:55.536+0000 7f1597fff700 1 -- --> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] -- mon_getmap magic: 0 v1 -- 0x7f159811ecd0 con 0x7f159809ffd0 2023-09-30T01:36:55.557 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:55.536+0000 7f1597fff700 10 monclient(hunting): start opening mon connection 2023-09-30T01:36:55.557 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:55.536+0000 7f1597fff700 1 -- --> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] -- mon_getmap magic: 0 v1 -- 0x7f15980a0ac0 con 0x7f15980a03a0 2023-09-30T01:36:55.557 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:55.536+0000 7f1597fff700 10 monclient(hunting): _renew_subs 2023-09-30T01:36:55.558 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:55.536+0000 7f1597fff700 10 monclient(hunting): authenticate will time out at 1087.577486s 2023-09-30T01:36:55.558 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:55.536+0000 7f15967fc700 1 --2- >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f15980a03a0 0x7f1598142b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:36:55.558 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:55.536+0000 7f15977fe700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f15980a3380 0x7f1598143050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:36:55.558 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:55.536+0000 7f15967fc700 1 --2- >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f15980a03a0 0x7f1598142b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3304/0 says I am v2:172.21.15.88:60974/0 (socket says 172.21.15.88:60974) 2023-09-30T01:36:55.558 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:55.536+0000 7f15967fc700 1 -- 172.21.15.88:0/2643026641 learned_addr learned my addr 172.21.15.88:0/2643026641 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-09-30T01:36:55.558 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:55.536+0000 7f15967fc700 10 monclient(hunting): get_auth_request con 0x7f15980a03a0 auth_method 0 2023-09-30T01:36:55.558 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:55.536+0000 7f15967fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:36:55.558 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:55.536+0000 7f15967fc700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:36:55.558 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:55.536+0000 7f15967fc700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:36:55.558 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:55.536+0000 7f15977fe700 10 monclient(hunting): get_auth_request con 0x7f15980a3380 auth_method 0 2023-09-30T01:36:55.559 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:55.536+0000 7f15977fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:36:55.559 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:55.536+0000 7f15977fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:36:55.559 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:55.536+0000 7f15977fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:36:55.559 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:55.536+0000 7f1596ffd700 1 --2- 172.21.15.88:0/2643026641 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7f159809ffd0 0x7f15981425d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:36:55.559 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:55.536+0000 7f1596ffd700 10 monclient(hunting): get_auth_request con 0x7f159809ffd0 auth_method 0 2023-09-30T01:36:55.559 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:55.536+0000 7f1596ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:36:55.559 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:55.536+0000 7f1596ffd700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:36:55.559 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:55.536+0000 7f1596ffd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:36:55.559 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:55.536+0000 7f15967fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:36:55.559 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:55.536+0000 7f15967fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:36:55.559 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:55.536+0000 7f15967fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:36:55.559 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:55.536+0000 7f15977fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:36:55.560 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:55.536+0000 7f15977fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:36:55.560 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:55.536+0000 7f15977fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:36:55.560 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:55.536+0000 7f1596ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:36:55.560 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:55.536+0000 7f1596ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:36:55.560 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:55.536+0000 7f1596ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:36:55.560 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:55.536+0000 7f1596ffd700 10 monclient(hunting): handle_auth_done global_id 4768 payload 824 2023-09-30T01:36:55.560 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:55.536+0000 7f1596ffd700 1 -- 172.21.15.88:0/2643026641 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f15980a03a0 msgr2=0x7f1598142b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:36:55.560 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:55.536+0000 7f1596ffd700 1 --2- 172.21.15.88:0/2643026641 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f15980a03a0 0x7f1598142b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:36:55.560 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:55.536+0000 7f1596ffd700 1 -- 172.21.15.88:0/2643026641 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f15980a3380 msgr2=0x7f1598143050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:36:55.560 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:55.536+0000 7f1596ffd700 1 --2- 172.21.15.88:0/2643026641 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f15980a3380 0x7f1598143050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:36:55.560 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:55.536+0000 7f1596ffd700 10 monclient: _finish_hunting 0 2023-09-30T01:36:55.560 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:55.536+0000 7f1596ffd700 1 monclient: found mon.noname-i 2023-09-30T01:36:55.561 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:55.536+0000 7f1596ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:36:55.561 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:55.536+0000 7f1596ffd700 10 monclient: _send_mon_message to mon.noname-i at v2:172.21.15.107:3303/0 2023-09-30T01:36:55.561 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:55.536+0000 7f1596ffd700 1 -- 172.21.15.88:0/2643026641 --> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1580009420 con 0x7f159809ffd0 2023-09-30T01:36:55.561 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:55.536+0000 7f1596ffd700 1 --2- 172.21.15.88:0/2643026641 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7f159809ffd0 0x7f15981425d0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f1580015000 tx=0x7f1580016000 comp rx=0 tx=0).ready entity=mon.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:36:55.561 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:55.536+0000 7f158effd700 1 -- 172.21.15.88:0/2643026641 <== mon.7 v2:172.21.15.107:3303/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f1580011dd0 con 0x7f159809ffd0 2023-09-30T01:36:55.561 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:55.536+0000 7f158effd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:36:55.561 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:55.536+0000 7f158effd700 10 monclient: got monmap 1 from mon.noname-i (according to old e1) 2023-09-30T01:36:55.561 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:55.536+0000 7f158effd700 10 monclient: dump: 2023-09-30T01:36:55.561 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:36:55.561 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:36:55.561 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:36:55.562 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:36:55.562 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:36:55.562 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:36:55.562 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:36:55.562 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:36:55.562 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:36:55.562 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:36:55.562 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:36:55.562 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:36:55.562 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:36:55.562 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:36:55.563 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:36:55.563 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:36:55.563 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:55.536+0000 7f158effd700 10 monclient: _finish_auth 0 2023-09-30T01:36:55.563 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:55.536+0000 7f158effd700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:36:55.563 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:55.536+0000 7f1597fff700 5 monclient: authenticate success, global_id 4768 2023-09-30T01:36:55.563 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:55.536+0000 7f158effd700 1 -- 172.21.15.88:0/2643026641 <== mon.7 v2:172.21.15.107:3303/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1580020070 con 0x7f159809ffd0 2023-09-30T01:36:55.563 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:55.536+0000 7f158effd700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:36:55.563 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:55.536+0000 7f158effd700 1 -- 172.21.15.88:0/2643026641 <== mon.7 v2:172.21.15.107:3303/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f1580025900 con 0x7f159809ffd0 2023-09-30T01:36:55.563 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:55.536+0000 7f1597fff700 10 monclient: _renew_subs 2023-09-30T01:36:55.563 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:55.536+0000 7f1597fff700 10 monclient: _send_mon_message to mon.i at v2:172.21.15.107:3303/0 2023-09-30T01:36:55.563 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:55.536+0000 7f1597fff700 1 -- 172.21.15.88:0/2643026641 --> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1598097ce0 con 0x7f159809ffd0 2023-09-30T01:36:55.563 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:55.536+0000 7f15957fa700 4 set_mon_vals no callback set 2023-09-30T01:36:55.564 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:55.536+0000 7f158effd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:36:55.564 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:55.536+0000 7f158effd700 10 monclient: got monmap 1 from mon.i (according to old e1) 2023-09-30T01:36:55.564 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:55.536+0000 7f158effd700 10 monclient: dump: 2023-09-30T01:36:55.564 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:36:55.564 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:36:55.564 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:36:55.564 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:36:55.564 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:36:55.564 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:36:55.564 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:36:55.564 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:36:55.564 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:36:55.565 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:36:55.565 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:36:55.565 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:36:55.565 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:36:55.565 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:36:55.565 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:36:55.565 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:36:55.565 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:55.536+0000 7f1597fff700 10 monclient: _renew_subs 2023-09-30T01:36:55.565 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:55.536+0000 7f1597fff700 10 monclient: _send_mon_message to mon.i at v2:172.21.15.107:3303/0 2023-09-30T01:36:55.565 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:55.536+0000 7f1597fff700 1 -- 172.21.15.88:0/2643026641 --> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f159811e740 con 0x7f159809ffd0 2023-09-30T01:36:55.565 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:55.540+0000 7f1597fff700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:36:55.566 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:55.540+0000 7f1597fff700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:36:55.566 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:55.540+0000 7f1597fff700 10 monclient: _send_mon_message to mon.i at v2:172.21.15.107:3303/0 2023-09-30T01:36:55.566 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:55.540+0000 7f1597fff700 1 -- 172.21.15.88:0/2643026641 --> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1598120680 con 0x7f159809ffd0 2023-09-30T01:36:55.566 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:55.540+0000 7f158effd700 1 -- 172.21.15.88:0/2643026641 <== mon.7 v2:172.21.15.107:3303/0 4 ==== mgrmap(e 5) v1 ==== 40144+0+0 (secure 0 0 0) 0x7f1580022030 con 0x7f159809ffd0 2023-09-30T01:36:55.566 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:55.540+0000 7f158effd700 1 --2- 172.21.15.88:0/2643026641 >> v2:172.21.15.107:6812/29926 conn(0x7f1578033290 0x7f1578035740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:36:55.566 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:55.540+0000 7f158effd700 1 -- 172.21.15.88:0/2643026641 <== mon.7 v2:172.21.15.107:3303/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 3502+0+0 (secure 0 0 0) 0x7f1580044d60 con 0x7f159809ffd0 2023-09-30T01:36:55.566 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:55.540+0000 7f15967fc700 1 --2- 172.21.15.88:0/2643026641 >> v2:172.21.15.107:6812/29926 conn(0x7f1578033290 0x7f1578035740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:36:55.566 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:55.540+0000 7f15967fc700 10 monclient: get_auth_request con 0x7f1578033290 auth_method 0 2023-09-30T01:36:55.566 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:55.540+0000 7f15967fc700 1 --2- 172.21.15.88:0/2643026641 >> v2:172.21.15.107:6812/29926 conn(0x7f1578033290 0x7f1578035740 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f1588007fd0 tx=0x7f1588004ab0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:36:55.567 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:55.552+0000 7f158effd700 1 -- 172.21.15.88:0/2643026641 <== mon.7 v2:172.21.15.107:3303/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f15800199e0 con 0x7f159809ffd0 2023-09-30T01:36:55.567 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:55.552+0000 7f158effd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:36:55.567 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:55.552+0000 7f158effd700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:36:55.753 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:55.748+0000 7f1597fff700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T01:36:55.753 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:55.748+0000 7f1597fff700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:36:55.753 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:55.748+0000 7f1597fff700 10 monclient: _send_mon_message to mon.i at v2:172.21.15.107:3303/0 2023-09-30T01:36:55.754 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:55.748+0000 7f1597fff700 1 -- 172.21.15.88:0/2643026641 --> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f15980710d0 con 0x7f159809ffd0 2023-09-30T01:36:55.754 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:55.748+0000 7f158effd700 1 -- 172.21.15.88:0/2643026641 <== mon.7 v2:172.21.15.107:3303/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f158002a030 con 0x7f159809ffd0 2023-09-30T01:36:55.754 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:55.748+0000 7f158effd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:36:55.754 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:55.748+0000 7f158effd700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:36:55.754 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-09-30T01:36:55.755 INFO:teuthology.orchestra.run.smithi088.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:36:55.756 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:55.752+0000 7f1597fff700 1 -- 172.21.15.88:0/2643026641 >> v2:172.21.15.107:6812/29926 conn(0x7f1578033290 msgr2=0x7f1578035740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:36:55.756 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:55.752+0000 7f1597fff700 1 --2- 172.21.15.88:0/2643026641 >> v2:172.21.15.107:6812/29926 conn(0x7f1578033290 0x7f1578035740 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f1588007fd0 tx=0x7f1588004ab0 comp rx=0 tx=0).stop 2023-09-30T01:36:55.757 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:55.752+0000 7f1597fff700 10 monclient: shutdown 2023-09-30T01:36:55.757 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:55.752+0000 7f1597fff700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:36:55.757 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:55.752+0000 7f1597fff700 1 -- 172.21.15.88:0/2643026641 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7f159809ffd0 msgr2=0x7f15981425d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:36:55.757 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:55.752+0000 7f1597fff700 1 --2- 172.21.15.88:0/2643026641 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7f159809ffd0 0x7f15981425d0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f1580015000 tx=0x7f1580016000 comp rx=0 tx=0).stop 2023-09-30T01:36:55.757 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:55.752+0000 7f1597fff700 1 -- 172.21.15.88:0/2643026641 shutdown_connections 2023-09-30T01:36:55.757 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:55.752+0000 7f1597fff700 1 --2- 172.21.15.88:0/2643026641 >> v2:172.21.15.107:6812/29926 conn(0x7f1578033290 0x7f1578035740 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:36:55.757 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:55.752+0000 7f1597fff700 1 --2- 172.21.15.88:0/2643026641 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f15980a03a0 0x7f1598142b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:36:55.757 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:55.752+0000 7f1597fff700 1 --2- 172.21.15.88:0/2643026641 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f15980a3380 0x7f1598143050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:36:55.757 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:55.752+0000 7f1597fff700 1 --2- 172.21.15.88:0/2643026641 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7f159809ffd0 0x7f15981425d0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:36:55.757 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:55.752+0000 7f1597fff700 1 -- 172.21.15.88:0/2643026641 >> 172.21.15.88:0/2643026641 conn(0x7f159806e5e0 msgr2=0x7f15980a1bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:36:55.757 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:55.752+0000 7f1597fff700 1 -- 172.21.15.88:0/2643026641 shutdown_connections 2023-09-30T01:36:55.758 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:55.752+0000 7f1597fff700 1 -- 172.21.15.88:0/2643026641 wait complete. 2023-09-30T01:36:58.769 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:36:58.883 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:58.876+0000 7fb9c24c0700 10 monclient: get_monmap_and_config 2023-09-30T01:36:58.884 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:58.876+0000 7fb9c24c0700 10 monclient: build_initial_monmap 2023-09-30T01:36:58.884 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:58.876+0000 7fb9c24c0700 10 monclient: monmap: 2023-09-30T01:36:58.884 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2023-09-30T01:36:58.884 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:36:58.884 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:36:58.883319+0000 2023-09-30T01:36:58.884 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:36:58.883319+0000 2023-09-30T01:36:58.885 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2023-09-30T01:36:58.885 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:36:58.885 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a-legacy 2023-09-30T01:36:58.885 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.107:6789/0 mon.noname-f-legacy 2023-09-30T01:36:58.885 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.noname-b 2023-09-30T01:36:58.885 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.noname-g 2023-09-30T01:36:58.885 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.noname-c 2023-09-30T01:36:58.885 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.noname-h 2023-09-30T01:36:58.885 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.noname-d 2023-09-30T01:36:58.885 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.noname-i 2023-09-30T01:36:58.885 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.noname-e 2023-09-30T01:36:58.886 INFO:teuthology.orchestra.run.smithi088.stderr:9: v2:172.21.15.88:3300/0 mon.noname-a 2023-09-30T01:36:58.886 INFO:teuthology.orchestra.run.smithi088.stderr:10: v2:172.21.15.107:3300/0 mon.noname-f 2023-09-30T01:36:58.886 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:36:58.894 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:58.880+0000 7fb9c24c0700 1 Processor -- start 2023-09-30T01:36:58.894 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:58.880+0000 7fb9c24c0700 1 -- start start 2023-09-30T01:36:58.894 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:58.880+0000 7fb9c24c0700 10 monclient: init 2023-09-30T01:36:58.894 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:58.880+0000 7fb9c24c0700 10 monclient: _reopen_session rank -1 2023-09-30T01:36:58.894 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:58.880+0000 7fb9c24c0700 10 monclient: _add_conns ranks=[8,4,5,3,2,9,7,1,6,10,0] 2023-09-30T01:36:58.895 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:58.880+0000 7fb9c24c0700 1 --2- >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7fb9bc0a3380 0x7fb9bc0a48b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:36:58.895 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:58.880+0000 7fb9c24c0700 10 monclient(hunting): picked mon.noname-e con 0x7fb9bc0a3380 addr [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] 2023-09-30T01:36:58.895 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:58.880+0000 7fb9c24c0700 1 --2- >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7fb9bc09ffd0 0x7fb9bc0a03a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:36:58.895 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:58.880+0000 7fb9c24c0700 10 monclient(hunting): picked mon.noname-c con 0x7fb9bc09ffd0 addr [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] 2023-09-30T01:36:58.895 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:58.880+0000 7fb9c24c0700 1 --2- >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7fb9bc1333b0 0x7fb9bc133800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:36:58.895 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:58.880+0000 7fb9c24c0700 10 monclient(hunting): picked mon.noname-h con 0x7fb9bc1333b0 addr [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] 2023-09-30T01:36:58.895 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:58.880+0000 7fb9c24c0700 10 monclient(hunting): start opening mon connection 2023-09-30T01:36:58.895 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:58.880+0000 7fb9c24c0700 1 -- --> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] -- mon_getmap magic: 0 v1 -- 0x7fb9bc0a08e0 con 0x7fb9bc09ffd0 2023-09-30T01:36:58.895 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:58.880+0000 7fb9c24c0700 10 monclient(hunting): start opening mon connection 2023-09-30T01:36:58.895 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:58.880+0000 7fb9c24c0700 1 -- --> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] -- mon_getmap magic: 0 v1 -- 0x7fb9bc098480 con 0x7fb9bc1333b0 2023-09-30T01:36:58.896 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:58.880+0000 7fb9c24c0700 10 monclient(hunting): start opening mon connection 2023-09-30T01:36:58.896 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:58.880+0000 7fb9c24c0700 1 -- --> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] -- mon_getmap magic: 0 v1 -- 0x7fb9bc0a3750 con 0x7fb9bc0a3380 2023-09-30T01:36:58.896 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:58.880+0000 7fb9c24c0700 10 monclient(hunting): _renew_subs 2023-09-30T01:36:58.896 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:58.880+0000 7fb9c24c0700 10 monclient(hunting): authenticate will time out at 1090.921094s 2023-09-30T01:36:58.896 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:58.880+0000 7fb9c0cbd700 1 --2- >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7fb9bc09ffd0 0x7fb9bc0a03a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:36:58.896 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:58.880+0000 7fb9c0cbd700 1 --2- >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7fb9bc09ffd0 0x7fb9bc0a03a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3302/0 says I am v2:172.21.15.88:53014/0 (socket says 172.21.15.88:53014) 2023-09-30T01:36:58.896 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:58.880+0000 7fb9c0cbd700 1 -- 172.21.15.88:0/1523113525 learned_addr learned my addr 172.21.15.88:0/1523113525 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-09-30T01:36:58.896 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:58.880+0000 7fb9c0cbd700 10 monclient(hunting): get_auth_request con 0x7fb9bc09ffd0 auth_method 0 2023-09-30T01:36:58.896 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:58.880+0000 7fb9c0cbd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:36:58.896 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:58.880+0000 7fb9c0cbd700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:36:58.896 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:58.880+0000 7fb9c0cbd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:36:58.897 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:58.880+0000 7fb9c14be700 1 --2- 172.21.15.88:0/1523113525 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7fb9bc0a3380 0x7fb9bc0a48b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:36:58.897 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:58.880+0000 7fb9c0cbd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:36:58.897 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:58.880+0000 7fb9c0cbd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:36:58.897 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:58.880+0000 7fb9c0cbd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:36:58.897 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:58.880+0000 7fb9c14be700 10 monclient(hunting): get_auth_request con 0x7fb9bc0a3380 auth_method 0 2023-09-30T01:36:58.897 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:58.880+0000 7fb9c14be700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:36:58.897 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:58.880+0000 7fb9c14be700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:36:58.897 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:58.880+0000 7fb9c14be700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:36:58.897 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:58.880+0000 7fb9c1cbf700 1 --2- 172.21.15.88:0/1523113525 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7fb9bc1333b0 0x7fb9bc133800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:36:58.897 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:58.880+0000 7fb9c0cbd700 10 monclient(hunting): handle_auth_done global_id 4756 payload 274 2023-09-30T01:36:58.897 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:58.880+0000 7fb9c0cbd700 1 -- 172.21.15.88:0/1523113525 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7fb9bc0a3380 msgr2=0x7fb9bc0a48b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:36:58.898 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:58.880+0000 7fb9c0cbd700 1 --2- 172.21.15.88:0/1523113525 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7fb9bc0a3380 0x7fb9bc0a48b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:36:58.898 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:58.880+0000 7fb9c0cbd700 1 -- 172.21.15.88:0/1523113525 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7fb9bc1333b0 msgr2=0x7fb9bc133800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:36:58.898 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:58.880+0000 7fb9c0cbd700 1 --2- 172.21.15.88:0/1523113525 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7fb9bc1333b0 0x7fb9bc133800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:36:58.898 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:58.880+0000 7fb9c0cbd700 10 monclient: _finish_hunting 0 2023-09-30T01:36:58.898 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:58.880+0000 7fb9c0cbd700 1 monclient: found mon.noname-c 2023-09-30T01:36:58.898 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:58.880+0000 7fb9c0cbd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:36:58.898 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:58.880+0000 7fb9c0cbd700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.88:3302/0 2023-09-30T01:36:58.898 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:58.880+0000 7fb9c0cbd700 1 -- 172.21.15.88:0/1523113525 --> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9bc04bb20 con 0x7fb9bc09ffd0 2023-09-30T01:36:58.898 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:58.880+0000 7fb9c1cbf700 10 monclient: get_auth_request con 0x7fb9bc1333b0 auth_method 0 2023-09-30T01:36:58.898 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:58.880+0000 7fb9c14be700 1 --2- 172.21.15.88:0/1523113525 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7fb9bc0a3380 0x7fb9bc0a48b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-30T01:36:58.899 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:58.880+0000 7fb9c1cbf700 1 --2- 172.21.15.88:0/1523113525 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7fb9bc1333b0 0x7fb9bc133800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-09-30T01:36:58.899 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:58.880+0000 7fb9c0cbd700 1 --2- 172.21.15.88:0/1523113525 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7fb9bc09ffd0 0x7fb9bc0a03a0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fb9b4009f50 tx=0x7fb9b4009840 comp rx=0 tx=0).ready entity=mon.4 client_cookie=b3ad1b6f05168094 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:36:58.899 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:58.880+0000 7fb9bbfff700 1 -- 172.21.15.88:0/1523113525 <== mon.4 v2:172.21.15.88:3302/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7fb9b400e070 con 0x7fb9bc09ffd0 2023-09-30T01:36:58.899 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:58.880+0000 7fb9bbfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:36:58.899 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:58.880+0000 7fb9bbfff700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T01:36:58.899 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:58.880+0000 7fb9bbfff700 10 monclient: dump: 2023-09-30T01:36:58.899 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:36:58.899 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:36:58.900 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:36:58.900 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:36:58.900 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:36:58.900 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:36:58.900 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:36:58.900 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:36:58.900 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:36:58.900 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:36:58.900 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:36:58.900 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:36:58.901 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:36:58.901 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:36:58.901 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:36:58.901 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:36:58.901 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:58.880+0000 7fb9bbfff700 10 monclient: _finish_auth 0 2023-09-30T01:36:58.901 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:58.880+0000 7fb9bbfff700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:36:58.901 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:58.880+0000 7fb9bbfff700 1 -- 172.21.15.88:0/1523113525 <== mon.4 v2:172.21.15.88:3302/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb9b4007400 con 0x7fb9bc09ffd0 2023-09-30T01:36:58.901 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:58.880+0000 7fb9c24c0700 5 monclient: authenticate success, global_id 4756 2023-09-30T01:36:58.901 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:58.880+0000 7fb9c24c0700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:36:58.901 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:58.880+0000 7fb9bbfff700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:36:58.901 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:58.880+0000 7fb9bbfff700 1 -- 172.21.15.88:0/1523113525 <== mon.4 v2:172.21.15.88:3302/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7fb9b40092e0 con 0x7fb9bc09ffd0 2023-09-30T01:36:58.902 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:58.880+0000 7fb9bbfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:36:58.902 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:58.880+0000 7fb9bbfff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T01:36:58.902 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:58.880+0000 7fb9bbfff700 10 monclient: dump: 2023-09-30T01:36:58.902 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:36:58.902 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:36:58.902 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:36:58.902 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:36:58.902 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:36:58.902 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:36:58.903 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:36:58.903 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:36:58.903 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:36:58.903 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:36:58.903 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:36:58.903 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:36:58.903 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:36:58.903 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:36:58.903 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:36:58.903 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:36:58.904 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:58.880+0000 7fb9c24c0700 10 monclient: get_monmap_and_config success 2023-09-30T01:36:58.904 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:58.880+0000 7fb9c24c0700 4 set_mon_vals no callback set 2023-09-30T01:36:58.904 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:58.880+0000 7fb9c24c0700 10 monclient: shutdown 2023-09-30T01:36:58.904 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:58.880+0000 7fb9c24c0700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:36:58.904 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:58.880+0000 7fb9c24c0700 1 -- 172.21.15.88:0/1523113525 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7fb9bc09ffd0 msgr2=0x7fb9bc0a03a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:36:58.904 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:58.880+0000 7fb9c24c0700 1 --2- 172.21.15.88:0/1523113525 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7fb9bc09ffd0 0x7fb9bc0a03a0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fb9b4009f50 tx=0x7fb9b4009840 comp rx=0 tx=0).stop 2023-09-30T01:36:58.904 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:58.884+0000 7fb9c24c0700 1 -- 172.21.15.88:0/1523113525 shutdown_connections 2023-09-30T01:36:58.904 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:58.884+0000 7fb9c24c0700 1 --2- 172.21.15.88:0/1523113525 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7fb9bc1333b0 0x7fb9bc133800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:36:58.904 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:58.884+0000 7fb9c24c0700 1 --2- 172.21.15.88:0/1523113525 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7fb9bc09ffd0 0x7fb9bc0a03a0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:36:58.904 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:58.884+0000 7fb9c24c0700 1 --2- 172.21.15.88:0/1523113525 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7fb9bc0a3380 0x7fb9bc0a48b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:36:58.905 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:58.884+0000 7fb9c24c0700 1 -- 172.21.15.88:0/1523113525 >> 172.21.15.88:0/1523113525 conn(0x7fb9bc06e5e0 msgr2=0x7fb9bc06e9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:36:58.905 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:58.884+0000 7fb9c24c0700 1 -- 172.21.15.88:0/1523113525 shutdown_connections 2023-09-30T01:36:58.905 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:58.884+0000 7fb9c24c0700 1 -- 172.21.15.88:0/1523113525 wait complete. 2023-09-30T01:36:58.905 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:58.884+0000 7fb9c24c0700 10 monclient: build_initial_monmap 2023-09-30T01:36:58.905 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:58.884+0000 7fb9c24c0700 10 monclient: monmap: 2023-09-30T01:36:58.905 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2023-09-30T01:36:58.905 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:36:58.905 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 0.000000 2023-09-30T01:36:58.905 INFO:teuthology.orchestra.run.smithi088.stderr:created 0.000000 2023-09-30T01:36:58.905 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2023-09-30T01:36:58.905 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 1 2023-09-30T01:36:58.906 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.noname-a 2023-09-30T01:36:58.906 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.noname-f 2023-09-30T01:36:58.906 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.noname-b 2023-09-30T01:36:58.906 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.noname-g 2023-09-30T01:36:58.906 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.noname-c 2023-09-30T01:36:58.906 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.noname-h 2023-09-30T01:36:58.906 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.noname-d 2023-09-30T01:36:58.906 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.noname-i 2023-09-30T01:36:58.906 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.noname-e 2023-09-30T01:36:58.906 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:36:58.907 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:58.884+0000 7fb9c24c0700 1 Processor -- start 2023-09-30T01:36:58.907 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:58.884+0000 7fb9c24c0700 1 -- start start 2023-09-30T01:36:58.907 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:58.884+0000 7fb9c24c0700 10 monclient: init 2023-09-30T01:36:58.907 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:58.884+0000 7fb9c24c0700 10 monclient: _reopen_session rank -1 2023-09-30T01:36:58.907 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:58.884+0000 7fb9c24c0700 10 monclient: _add_conns ranks=[1,7,8,3,5,6,0,2,4] 2023-09-30T01:36:58.907 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:58.884+0000 7fb9c24c0700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb9bc09ffd0 0x7fb9bc1422c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:36:58.907 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:58.884+0000 7fb9c24c0700 10 monclient(hunting): picked mon.noname-f con 0x7fb9bc09ffd0 addr [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] 2023-09-30T01:36:58.907 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:58.884+0000 7fb9c24c0700 1 --2- >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7fb9bc0a3380 0x7fb9bc130650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:36:58.907 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:58.884+0000 7fb9c24c0700 10 monclient(hunting): picked mon.noname-i con 0x7fb9bc0a3380 addr [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] 2023-09-30T01:36:58.907 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:58.884+0000 7fb9c24c0700 1 --2- >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7fb9bc1333b0 0x7fb9bc09d080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:36:58.907 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:58.884+0000 7fb9c24c0700 10 monclient(hunting): picked mon.noname-e con 0x7fb9bc1333b0 addr [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] 2023-09-30T01:36:58.908 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:58.884+0000 7fb9c24c0700 10 monclient(hunting): start opening mon connection 2023-09-30T01:36:58.908 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:58.884+0000 7fb9c24c0700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9bc097ce0 con 0x7fb9bc09ffd0 2023-09-30T01:36:58.908 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:58.884+0000 7fb9c24c0700 10 monclient(hunting): start opening mon connection 2023-09-30T01:36:58.908 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:58.884+0000 7fb9c24c0700 1 -- --> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] -- mon_getmap magic: 0 v1 -- 0x7fb9bc09f8f0 con 0x7fb9bc0a3380 2023-09-30T01:36:58.908 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:58.884+0000 7fb9c24c0700 10 monclient(hunting): start opening mon connection 2023-09-30T01:36:58.908 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:58.884+0000 7fb9c24c0700 1 -- --> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] -- mon_getmap magic: 0 v1 -- 0x7fb9bc09f230 con 0x7fb9bc1333b0 2023-09-30T01:36:58.908 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:58.884+0000 7fb9c24c0700 10 monclient(hunting): _renew_subs 2023-09-30T01:36:58.908 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:58.884+0000 7fb9c24c0700 10 monclient(hunting): authenticate will time out at 1090.927841s 2023-09-30T01:36:58.908 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:58.888+0000 7fb9c1cbf700 1 --2- >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7fb9bc1333b0 0x7fb9bc09d080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:36:58.908 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:58.888+0000 7fb9c1cbf700 1 --2- >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7fb9bc1333b0 0x7fb9bc09d080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3304/0 says I am v2:172.21.15.88:60998/0 (socket says 172.21.15.88:60998) 2023-09-30T01:36:58.909 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:58.888+0000 7fb9c1cbf700 1 -- 172.21.15.88:0/1139072218 learned_addr learned my addr 172.21.15.88:0/1139072218 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-09-30T01:36:58.909 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:58.888+0000 7fb9c1cbf700 10 monclient(hunting): get_auth_request con 0x7fb9bc1333b0 auth_method 0 2023-09-30T01:36:58.909 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:58.888+0000 7fb9c1cbf700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:36:58.909 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:58.888+0000 7fb9c1cbf700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:36:58.909 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:58.888+0000 7fb9c1cbf700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:36:58.909 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:58.888+0000 7fb9c14be700 1 --2- 172.21.15.88:0/1139072218 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb9bc09ffd0 0x7fb9bc1422c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:36:58.909 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:58.888+0000 7fb9c14be700 10 monclient(hunting): get_auth_request con 0x7fb9bc09ffd0 auth_method 0 2023-09-30T01:36:58.909 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:58.888+0000 7fb9c0cbd700 1 --2- 172.21.15.88:0/1139072218 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7fb9bc0a3380 0x7fb9bc130650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:36:58.909 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:58.888+0000 7fb9c14be700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:36:58.910 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:58.888+0000 7fb9c14be700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:36:58.910 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:58.888+0000 7fb9c14be700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:36:58.910 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:58.888+0000 7fb9c0cbd700 10 monclient(hunting): get_auth_request con 0x7fb9bc0a3380 auth_method 0 2023-09-30T01:36:58.910 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:58.888+0000 7fb9c0cbd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:36:58.910 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:58.888+0000 7fb9c0cbd700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:36:58.910 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:58.888+0000 7fb9c0cbd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:36:58.910 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:58.888+0000 7fb9c1cbf700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:36:58.910 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:58.888+0000 7fb9c1cbf700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:36:58.910 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:58.888+0000 7fb9c1cbf700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:36:58.911 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:58.888+0000 7fb9c14be700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:36:58.911 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:58.888+0000 7fb9c14be700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:36:58.911 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:58.888+0000 7fb9c14be700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:36:58.911 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:58.888+0000 7fb9c0cbd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:36:58.911 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:58.888+0000 7fb9c0cbd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:36:58.911 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:58.888+0000 7fb9c0cbd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:36:58.911 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:58.888+0000 7fb9c1cbf700 10 monclient(hunting): handle_auth_done global_id 4760 payload 824 2023-09-30T01:36:58.911 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:58.888+0000 7fb9c1cbf700 1 -- 172.21.15.88:0/1139072218 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7fb9bc0a3380 msgr2=0x7fb9bc130650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:36:58.911 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:58.888+0000 7fb9c1cbf700 1 --2- 172.21.15.88:0/1139072218 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7fb9bc0a3380 0x7fb9bc130650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:36:58.911 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:58.888+0000 7fb9c1cbf700 1 -- 172.21.15.88:0/1139072218 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb9bc09ffd0 msgr2=0x7fb9bc1422c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:36:58.911 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:58.888+0000 7fb9c1cbf700 1 --2- 172.21.15.88:0/1139072218 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb9bc09ffd0 0x7fb9bc1422c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:36:58.912 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:58.888+0000 7fb9c1cbf700 10 monclient: _finish_hunting 0 2023-09-30T01:36:58.912 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:58.888+0000 7fb9c1cbf700 1 monclient: found mon.noname-e 2023-09-30T01:36:58.912 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:58.888+0000 7fb9c1cbf700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:36:58.912 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:58.888+0000 7fb9c1cbf700 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.88:3304/0 2023-09-30T01:36:58.912 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:58.888+0000 7fb9c1cbf700 1 -- 172.21.15.88:0/1139072218 --> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9b4009ae0 con 0x7fb9bc1333b0 2023-09-30T01:36:58.912 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:58.888+0000 7fb9c1cbf700 1 --2- 172.21.15.88:0/1139072218 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7fb9bc1333b0 0x7fb9bc09d080 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fb9ac00c880 tx=0x7fb9ac008040 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:36:58.912 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:58.888+0000 7fb9b97fa700 1 -- 172.21.15.88:0/1139072218 <== mon.8 v2:172.21.15.88:3304/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7fb9ac011a80 con 0x7fb9bc1333b0 2023-09-30T01:36:58.912 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:58.888+0000 7fb9b97fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:36:58.912 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:58.888+0000 7fb9b97fa700 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2023-09-30T01:36:58.912 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:58.888+0000 7fb9b97fa700 10 monclient: dump: 2023-09-30T01:36:58.912 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:36:58.912 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:36:58.913 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:36:58.913 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:36:58.913 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:36:58.913 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:36:58.913 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:36:58.913 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:36:58.913 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:36:58.913 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:36:58.913 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:36:58.913 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:36:58.913 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:36:58.914 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:36:58.914 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:36:58.914 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:36:58.914 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:58.888+0000 7fb9b97fa700 10 monclient: _finish_auth 0 2023-09-30T01:36:58.914 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:58.888+0000 7fb9b97fa700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:36:58.914 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:58.888+0000 7fb9b97fa700 1 -- 172.21.15.88:0/1139072218 <== mon.8 v2:172.21.15.88:3304/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb9ac011c20 con 0x7fb9bc1333b0 2023-09-30T01:36:58.914 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:58.888+0000 7fb9c24c0700 5 monclient: authenticate success, global_id 4760 2023-09-30T01:36:58.914 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:58.888+0000 7fb9c24c0700 10 monclient: _renew_subs 2023-09-30T01:36:58.914 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:58.888+0000 7fb9c24c0700 10 monclient: _send_mon_message to mon.e at v2:172.21.15.88:3304/0 2023-09-30T01:36:58.914 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:58.888+0000 7fb9c24c0700 1 -- 172.21.15.88:0/1139072218 --> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb9bc0a0ac0 con 0x7fb9bc1333b0 2023-09-30T01:36:58.914 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:58.888+0000 7fb9b97fa700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:36:58.915 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:58.888+0000 7fb9b97fa700 1 -- 172.21.15.88:0/1139072218 <== mon.8 v2:172.21.15.88:3304/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7fb9ac011d90 con 0x7fb9bc1333b0 2023-09-30T01:36:58.915 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:58.888+0000 7fb9b97fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:36:58.915 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:58.888+0000 7fb9b97fa700 10 monclient: got monmap 1 from mon.e (according to old e1) 2023-09-30T01:36:58.915 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:58.888+0000 7fb9b97fa700 10 monclient: dump: 2023-09-30T01:36:58.915 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:36:58.915 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:36:58.915 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:36:58.915 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:36:58.915 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:36:58.915 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:36:58.916 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:36:58.916 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:36:58.916 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:36:58.916 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:36:58.916 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:36:58.916 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:36:58.916 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:36:58.916 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:36:58.916 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:36:58.916 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:36:58.916 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:58.888+0000 7fb9c24c0700 10 monclient: _renew_subs 2023-09-30T01:36:58.917 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:58.888+0000 7fb9bb7fe700 4 set_mon_vals no callback set 2023-09-30T01:36:58.917 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:58.888+0000 7fb9c24c0700 10 monclient: _send_mon_message to mon.e at v2:172.21.15.88:3304/0 2023-09-30T01:36:58.917 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:58.888+0000 7fb9c24c0700 1 -- 172.21.15.88:0/1139072218 --> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9bc09ecb0 con 0x7fb9bc1333b0 2023-09-30T01:36:58.918 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:58.888+0000 7fb9b97fa700 1 -- 172.21.15.88:0/1139072218 <== mon.8 v2:172.21.15.88:3304/0 4 ==== mgrmap(e 5) v1 ==== 40144+0+0 (secure 0 0 0) 0x7fb9ac011600 con 0x7fb9bc1333b0 2023-09-30T01:36:58.918 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:58.888+0000 7fb9c24c0700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:36:58.918 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:58.888+0000 7fb9c24c0700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:36:58.918 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:58.888+0000 7fb9c24c0700 10 monclient: _send_mon_message to mon.e at v2:172.21.15.88:3304/0 2023-09-30T01:36:58.918 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:58.888+0000 7fb9c24c0700 1 -- 172.21.15.88:0/1139072218 --> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb9bc0710d0 con 0x7fb9bc1333b0 2023-09-30T01:36:58.919 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:58.888+0000 7fb9b97fa700 1 --2- 172.21.15.88:0/1139072218 >> v2:172.21.15.107:6812/29926 conn(0x7fb9a8033360 0x7fb9a8035810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:36:58.919 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:58.888+0000 7fb9b97fa700 1 -- 172.21.15.88:0/1139072218 <== mon.8 v2:172.21.15.88:3304/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 3502+0+0 (secure 0 0 0) 0x7fb9ac042f20 con 0x7fb9bc1333b0 2023-09-30T01:36:58.919 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:58.888+0000 7fb9c14be700 1 --2- 172.21.15.88:0/1139072218 >> v2:172.21.15.107:6812/29926 conn(0x7fb9a8033360 0x7fb9a8035810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:36:58.919 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:58.888+0000 7fb9c14be700 10 monclient: get_auth_request con 0x7fb9a8033360 auth_method 0 2023-09-30T01:36:58.919 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:58.892+0000 7fb9c14be700 1 --2- 172.21.15.88:0/1139072218 >> v2:172.21.15.107:6812/29926 conn(0x7fb9a8033360 0x7fb9a8035810 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fb9b0006f00 tx=0x7fb9b0008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:36:58.919 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:58.896+0000 7fb9b97fa700 1 -- 172.21.15.88:0/1139072218 <== mon.8 v2:172.21.15.88:3304/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fb9ac072be0 con 0x7fb9bc1333b0 2023-09-30T01:36:58.919 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:58.896+0000 7fb9b97fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:36:58.919 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:58.896+0000 7fb9b97fa700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:36:59.095 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:59.088+0000 7fb9c24c0700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T01:36:59.095 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:59.088+0000 7fb9c24c0700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:36:59.095 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:59.088+0000 7fb9c24c0700 10 monclient: _send_mon_message to mon.e at v2:172.21.15.88:3304/0 2023-09-30T01:36:59.095 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:59.088+0000 7fb9c24c0700 1 -- 172.21.15.88:0/1139072218 --> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fb9bc071550 con 0x7fb9bc1333b0 2023-09-30T01:36:59.096 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:59.088+0000 7fb9b97fa700 1 -- 172.21.15.88:0/1139072218 <== mon.8 v2:172.21.15.88:3304/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7fb9ac0118b0 con 0x7fb9bc1333b0 2023-09-30T01:36:59.096 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:59.088+0000 7fb9b97fa700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:36:59.096 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:59.088+0000 7fb9b97fa700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:36:59.096 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-09-30T01:36:59.096 INFO:teuthology.orchestra.run.smithi088.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:36:59.097 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:59.092+0000 7fb9c24c0700 1 -- 172.21.15.88:0/1139072218 >> v2:172.21.15.107:6812/29926 conn(0x7fb9a8033360 msgr2=0x7fb9a8035810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:36:59.098 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:59.092+0000 7fb9c24c0700 1 --2- 172.21.15.88:0/1139072218 >> v2:172.21.15.107:6812/29926 conn(0x7fb9a8033360 0x7fb9a8035810 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fb9b0006f00 tx=0x7fb9b0008040 comp rx=0 tx=0).stop 2023-09-30T01:36:59.098 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:59.092+0000 7fb9c24c0700 10 monclient: shutdown 2023-09-30T01:36:59.098 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:59.092+0000 7fb9c24c0700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:36:59.098 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:59.092+0000 7fb9c24c0700 1 -- 172.21.15.88:0/1139072218 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7fb9bc1333b0 msgr2=0x7fb9bc09d080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:36:59.098 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:59.092+0000 7fb9c24c0700 1 --2- 172.21.15.88:0/1139072218 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7fb9bc1333b0 0x7fb9bc09d080 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fb9ac00c880 tx=0x7fb9ac008040 comp rx=0 tx=0).stop 2023-09-30T01:36:59.098 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:59.092+0000 7fb9c24c0700 1 -- 172.21.15.88:0/1139072218 shutdown_connections 2023-09-30T01:36:59.098 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:59.092+0000 7fb9c24c0700 1 --2- 172.21.15.88:0/1139072218 >> v2:172.21.15.107:6812/29926 conn(0x7fb9a8033360 0x7fb9a8035810 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:36:59.098 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:59.092+0000 7fb9c24c0700 1 --2- 172.21.15.88:0/1139072218 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7fb9bc1333b0 0x7fb9bc09d080 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:36:59.098 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:59.092+0000 7fb9c24c0700 1 --2- 172.21.15.88:0/1139072218 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7fb9bc0a3380 0x7fb9bc130650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:36:59.098 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:59.092+0000 7fb9c24c0700 1 --2- 172.21.15.88:0/1139072218 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb9bc09ffd0 0x7fb9bc1422c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:36:59.099 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:59.092+0000 7fb9c24c0700 1 -- 172.21.15.88:0/1139072218 >> 172.21.15.88:0/1139072218 conn(0x7fb9bc06e5e0 msgr2=0x7fb9bc1320c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:36:59.099 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:59.092+0000 7fb9c24c0700 1 -- 172.21.15.88:0/1139072218 shutdown_connections 2023-09-30T01:36:59.099 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:36:59.092+0000 7fb9c24c0700 1 -- 172.21.15.88:0/1139072218 wait complete. 2023-09-30T01:37:02.110 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:37:02.227 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:02.220+0000 7faca7fff700 10 monclient: get_monmap_and_config 2023-09-30T01:37:02.227 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:02.220+0000 7faca7fff700 10 monclient: build_initial_monmap 2023-09-30T01:37:02.227 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:02.220+0000 7faca7fff700 10 monclient: monmap: 2023-09-30T01:37:02.227 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2023-09-30T01:37:02.227 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:37:02.227 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:37:02.226778+0000 2023-09-30T01:37:02.227 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:37:02.226778+0000 2023-09-30T01:37:02.228 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2023-09-30T01:37:02.228 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:37:02.228 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a-legacy 2023-09-30T01:37:02.228 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.107:6789/0 mon.noname-f-legacy 2023-09-30T01:37:02.228 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.noname-b 2023-09-30T01:37:02.228 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.noname-g 2023-09-30T01:37:02.228 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.noname-c 2023-09-30T01:37:02.228 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.noname-h 2023-09-30T01:37:02.229 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.noname-d 2023-09-30T01:37:02.229 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.noname-i 2023-09-30T01:37:02.229 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.noname-e 2023-09-30T01:37:02.229 INFO:teuthology.orchestra.run.smithi088.stderr:9: v2:172.21.15.88:3300/0 mon.noname-a 2023-09-30T01:37:02.229 INFO:teuthology.orchestra.run.smithi088.stderr:10: v2:172.21.15.107:3300/0 mon.noname-f 2023-09-30T01:37:02.229 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:37:02.239 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:02.224+0000 7faca7fff700 1 Processor -- start 2023-09-30T01:37:02.239 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:02.224+0000 7faca7fff700 1 -- start start 2023-09-30T01:37:02.240 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:02.224+0000 7faca7fff700 10 monclient: init 2023-09-30T01:37:02.240 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:02.224+0000 7faca7fff700 10 monclient: _reopen_session rank -1 2023-09-30T01:37:02.240 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:02.224+0000 7faca7fff700 10 monclient: _add_conns ranks=[0,6,1,10,9,4,8,7,5,2,3] 2023-09-30T01:37:02.240 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:02.224+0000 7faca7fff700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7faca80976f0 addr v1:172.21.15.88:6789/0 2023-09-30T01:37:02.240 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:02.224+0000 7faca7fff700 1 --2- >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7faca8098de0 0x7faca809f490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:37:02.240 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:02.224+0000 7faca7fff700 10 monclient(hunting): picked mon.noname-d con 0x7faca8098de0 addr [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] 2023-09-30T01:37:02.240 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:02.224+0000 7faca7fff700 10 monclient(hunting): picked mon.noname-f-legacy con 0x7faca81357b0 addr v1:172.21.15.107:6789/0 2023-09-30T01:37:02.240 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:02.224+0000 7faca7fff700 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faca804bb20 con 0x7faca80976f0 2023-09-30T01:37:02.240 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:02.224+0000 7faca7fff700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faca8098420 con 0x7faca81357b0 2023-09-30T01:37:02.241 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:02.224+0000 7faca7fff700 10 monclient(hunting): start opening mon connection 2023-09-30T01:37:02.241 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:02.224+0000 7faca7fff700 1 -- --> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] -- mon_getmap magic: 0 v1 -- 0x7faca8098a50 con 0x7faca8098de0 2023-09-30T01:37:02.241 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:02.224+0000 7faca7fff700 10 monclient(hunting): _renew_subs 2023-09-30T01:37:02.242 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:02.224+0000 7faca6ffd700 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7faca80976f0 0x7faca809ed80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:59804/0 (socket says 172.21.15.88:59804) 2023-09-30T01:37:02.242 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:02.224+0000 7faca6ffd700 1 -- 172.21.15.88:0/3827557340 learned_addr learned my addr 172.21.15.88:0/3827557340 (peer_addr_for_me v1:172.21.15.88:0/0) 2023-09-30T01:37:02.242 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:02.224+0000 7faca7fff700 10 monclient(hunting): authenticate will time out at 1094.266103s 2023-09-30T01:37:02.242 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:02.224+0000 7fac9ffff700 1 --2- 172.21.15.88:0/3827557340 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7faca8098de0 0x7faca809f490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:37:02.242 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:02.224+0000 7fac9ffff700 10 monclient(hunting): get_auth_request con 0x7faca8098de0 auth_method 0 2023-09-30T01:37:02.242 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:02.224+0000 7fac9ffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:37:02.242 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:02.224+0000 7fac9ffff700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:37:02.242 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:02.224+0000 7fac9ffff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:37:02.243 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:02.224+0000 7fac9ffff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:37:02.243 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:02.224+0000 7fac9ffff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:37:02.243 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:02.224+0000 7fac9ffff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:37:02.243 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:02.224+0000 7faca67fc700 1 -- 172.21.15.88:0/3827557340 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2210073515 0 0) 0x7faca804bb20 con 0x7faca80976f0 2023-09-30T01:37:02.243 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:02.224+0000 7faca67fc700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:37:02.243 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:02.224+0000 7faca67fc700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:37:02.243 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:02.224+0000 7faca67fc700 10 monclient(hunting): my global_id is 5049 2023-09-30T01:37:02.243 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:02.224+0000 7faca67fc700 1 -- 172.21.15.88:0/3827557340 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fac94003570 con 0x7faca80976f0 2023-09-30T01:37:02.243 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:02.224+0000 7fac9ffff700 10 monclient(hunting): handle_auth_done global_id 4632 payload 274 2023-09-30T01:37:02.243 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:02.224+0000 7fac9ffff700 1 -- 172.21.15.88:0/3827557340 >> v1:172.21.15.107:6789/0 conn(0x7faca81357b0 legacy=0x7faca8137b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:37:02.244 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:02.224+0000 7fac9ffff700 1 -- 172.21.15.88:0/3827557340 >> v1:172.21.15.88:6789/0 conn(0x7faca80976f0 legacy=0x7faca809ed80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:37:02.244 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:02.224+0000 7fac9ffff700 10 monclient: _finish_hunting 0 2023-09-30T01:37:02.244 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:02.224+0000 7fac9ffff700 1 monclient: found mon.noname-d 2023-09-30T01:37:02.244 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:02.224+0000 7fac9ffff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:37:02.244 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:02.224+0000 7fac9ffff700 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.88:3303/0 2023-09-30T01:37:02.244 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:02.224+0000 7fac9ffff700 1 -- 172.21.15.88:0/3827557340 --> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faca8097ac0 con 0x7faca8098de0 2023-09-30T01:37:02.244 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:02.224+0000 7fac9ffff700 1 --2- 172.21.15.88:0/3827557340 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7faca8098de0 0x7faca809f490 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fac90009940 tx=0x7fac900091a0 comp rx=0 tx=0).ready entity=mon.6 client_cookie=42209e014e4979b7 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:37:02.244 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:02.224+0000 7faca67fc700 1 -- 172.21.15.88:0/3827557340 <== mon.6 v2:172.21.15.88:3303/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7fac9000baf0 con 0x7faca8098de0 2023-09-30T01:37:02.244 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:02.224+0000 7faca67fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:37:02.244 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:02.224+0000 7faca67fc700 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2023-09-30T01:37:02.244 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:02.224+0000 7faca67fc700 10 monclient: dump: 2023-09-30T01:37:02.244 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:37:02.245 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:37:02.245 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:37:02.245 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:37:02.245 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:37:02.245 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:37:02.245 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:37:02.245 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:37:02.245 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:37:02.245 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:37:02.245 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:37:02.245 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:37:02.245 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:37:02.246 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:37:02.246 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:37:02.246 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:37:02.246 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:02.224+0000 7faca67fc700 10 monclient: _finish_auth 0 2023-09-30T01:37:02.246 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:02.224+0000 7faca67fc700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:37:02.246 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:02.224+0000 7faca7fff700 5 monclient: authenticate success, global_id 4632 2023-09-30T01:37:02.246 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:02.224+0000 7faca67fc700 1 -- 172.21.15.88:0/3827557340 <== mon.6 v2:172.21.15.88:3303/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac9000bc70 con 0x7faca8098de0 2023-09-30T01:37:02.246 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:02.224+0000 7faca67fc700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:37:02.246 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:02.224+0000 7faca7fff700 10 monclient: get_monmap_and_config success 2023-09-30T01:37:02.246 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:02.224+0000 7faca67fc700 1 -- 172.21.15.88:0/3827557340 <== mon.6 v2:172.21.15.88:3303/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7fac9000bde0 con 0x7faca8098de0 2023-09-30T01:37:02.246 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:02.224+0000 7faca67fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:37:02.247 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:02.224+0000 7faca7fff700 4 set_mon_vals no callback set 2023-09-30T01:37:02.247 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:02.224+0000 7faca67fc700 10 monclient: got monmap 1 from mon.d (according to old e1) 2023-09-30T01:37:02.247 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:02.224+0000 7faca67fc700 10 monclient: dump: 2023-09-30T01:37:02.247 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:37:02.247 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:37:02.247 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:37:02.247 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:37:02.247 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:37:02.247 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:37:02.247 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:37:02.247 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:37:02.248 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:37:02.248 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:37:02.248 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:37:02.248 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:37:02.248 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:37:02.248 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:37:02.248 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:37:02.248 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:37:02.248 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:02.228+0000 7faca7fff700 10 monclient: shutdown 2023-09-30T01:37:02.248 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:02.228+0000 7faca7fff700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:37:02.249 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:02.228+0000 7faca7fff700 1 -- 172.21.15.88:0/3827557340 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7faca8098de0 msgr2=0x7faca809f490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:37:02.249 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:02.228+0000 7faca7fff700 1 --2- 172.21.15.88:0/3827557340 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7faca8098de0 0x7faca809f490 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fac90009940 tx=0x7fac900091a0 comp rx=0 tx=0).stop 2023-09-30T01:37:02.249 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:02.228+0000 7faca7fff700 1 -- 172.21.15.88:0/3827557340 shutdown_connections 2023-09-30T01:37:02.249 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:02.228+0000 7faca7fff700 1 --2- 172.21.15.88:0/3827557340 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7faca8098de0 0x7faca809f490 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:37:02.249 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:02.228+0000 7faca7fff700 1 -- 172.21.15.88:0/3827557340 >> 172.21.15.88:0/3827557340 conn(0x7faca80a0910 msgr2=0x7faca80a33c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:37:02.249 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:02.228+0000 7faca7fff700 1 -- 172.21.15.88:0/3827557340 shutdown_connections 2023-09-30T01:37:02.249 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:02.228+0000 7faca7fff700 1 -- 172.21.15.88:0/3827557340 wait complete. 2023-09-30T01:37:02.249 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:02.228+0000 7faca7fff700 10 monclient: build_initial_monmap 2023-09-30T01:37:02.249 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:02.228+0000 7faca7fff700 10 monclient: monmap: 2023-09-30T01:37:02.249 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2023-09-30T01:37:02.249 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:37:02.249 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 0.000000 2023-09-30T01:37:02.250 INFO:teuthology.orchestra.run.smithi088.stderr:created 0.000000 2023-09-30T01:37:02.250 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2023-09-30T01:37:02.250 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 1 2023-09-30T01:37:02.250 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.noname-a 2023-09-30T01:37:02.250 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.noname-f 2023-09-30T01:37:02.250 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.noname-b 2023-09-30T01:37:02.250 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.noname-g 2023-09-30T01:37:02.250 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.noname-c 2023-09-30T01:37:02.250 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.noname-h 2023-09-30T01:37:02.250 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.noname-d 2023-09-30T01:37:02.251 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.noname-i 2023-09-30T01:37:02.251 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.noname-e 2023-09-30T01:37:02.251 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:37:02.251 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:02.228+0000 7faca7fff700 1 Processor -- start 2023-09-30T01:37:02.251 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:02.228+0000 7faca7fff700 1 -- start start 2023-09-30T01:37:02.251 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:02.228+0000 7faca7fff700 10 monclient: init 2023-09-30T01:37:02.251 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:02.228+0000 7faca7fff700 10 monclient: _reopen_session rank -1 2023-09-30T01:37:02.251 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:02.228+0000 7faca7fff700 10 monclient: _add_conns ranks=[4,1,6,7,8,0,3,5,2] 2023-09-30T01:37:02.251 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:02.228+0000 7faca7fff700 1 --2- >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7faca80976f0 0x7faca8134680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:37:02.251 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:02.228+0000 7faca7fff700 10 monclient(hunting): picked mon.noname-c con 0x7faca80976f0 addr [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] 2023-09-30T01:37:02.251 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:02.228+0000 7faca7fff700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faca8098de0 0x7faca806d290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:37:02.252 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:02.228+0000 7faca7fff700 10 monclient(hunting): picked mon.noname-f con 0x7faca8098de0 addr [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] 2023-09-30T01:37:02.252 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:02.228+0000 7faca7fff700 1 --2- >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7faca81357b0 0x7faca806d7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:37:02.252 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:02.228+0000 7faca7fff700 10 monclient(hunting): picked mon.noname-d con 0x7faca81357b0 addr [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] 2023-09-30T01:37:02.252 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:02.228+0000 7faca7fff700 10 monclient(hunting): start opening mon connection 2023-09-30T01:37:02.252 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:02.228+0000 7faca7fff700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faca812a150 con 0x7faca8098de0 2023-09-30T01:37:02.252 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:02.228+0000 7faca7fff700 10 monclient(hunting): start opening mon connection 2023-09-30T01:37:02.252 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:02.228+0000 7faca7fff700 1 -- --> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] -- mon_getmap magic: 0 v1 -- 0x7faca8128760 con 0x7faca80976f0 2023-09-30T01:37:02.252 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:02.228+0000 7faca7fff700 10 monclient(hunting): start opening mon connection 2023-09-30T01:37:02.252 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:02.228+0000 7faca7fff700 1 -- --> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] -- mon_getmap magic: 0 v1 -- 0x7faca8129b00 con 0x7faca81357b0 2023-09-30T01:37:02.252 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:02.228+0000 7faca7fff700 10 monclient(hunting): _renew_subs 2023-09-30T01:37:02.253 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:02.228+0000 7faca7fff700 10 monclient(hunting): authenticate will time out at 1094.271325s 2023-09-30T01:37:02.253 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:02.228+0000 7faca6ffd700 1 --2- >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7faca80976f0 0x7faca8134680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:37:02.253 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:02.228+0000 7faca77fe700 1 --2- >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7faca81357b0 0x7faca806d7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:37:02.253 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:02.228+0000 7fac9ffff700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faca8098de0 0x7faca806d290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:37:02.253 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:02.228+0000 7faca6ffd700 1 --2- >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7faca80976f0 0x7faca8134680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3302/0 says I am v2:172.21.15.88:60974/0 (socket says 172.21.15.88:60974) 2023-09-30T01:37:02.253 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:02.228+0000 7faca6ffd700 1 -- 172.21.15.88:0/3959634233 learned_addr learned my addr 172.21.15.88:0/3959634233 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-09-30T01:37:02.253 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:02.228+0000 7faca6ffd700 10 monclient(hunting): get_auth_request con 0x7faca80976f0 auth_method 0 2023-09-30T01:37:02.253 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:02.228+0000 7faca6ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:37:02.253 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:02.228+0000 7faca6ffd700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:37:02.253 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:02.228+0000 7faca6ffd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:37:02.254 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:02.228+0000 7fac9ffff700 10 monclient(hunting): get_auth_request con 0x7faca8098de0 auth_method 0 2023-09-30T01:37:02.254 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:02.228+0000 7fac9ffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:37:02.254 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:02.228+0000 7fac9ffff700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:37:02.254 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:02.228+0000 7fac9ffff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:37:02.254 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:02.228+0000 7faca77fe700 10 monclient(hunting): get_auth_request con 0x7faca81357b0 auth_method 0 2023-09-30T01:37:02.254 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:02.228+0000 7faca77fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:37:02.254 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:02.228+0000 7faca77fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:37:02.254 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:02.228+0000 7faca77fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:37:02.254 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:02.228+0000 7faca6ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:37:02.255 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:02.228+0000 7faca6ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:37:02.255 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:02.228+0000 7faca6ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:37:02.255 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:02.228+0000 7faca77fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:37:02.255 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:02.228+0000 7faca77fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:37:02.255 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:02.232+0000 7faca77fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:37:02.255 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:02.232+0000 7fac9ffff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:37:02.255 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:02.232+0000 7fac9ffff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:37:02.256 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:02.232+0000 7fac9ffff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:37:02.256 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:02.232+0000 7faca6ffd700 10 monclient(hunting): handle_auth_done global_id 4765 payload 824 2023-09-30T01:37:02.256 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:02.232+0000 7faca6ffd700 1 -- 172.21.15.88:0/3959634233 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7faca81357b0 msgr2=0x7faca806d7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:37:02.256 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:02.232+0000 7faca6ffd700 1 --2- 172.21.15.88:0/3959634233 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7faca81357b0 0x7faca806d7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:37:02.256 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:02.232+0000 7faca6ffd700 1 -- 172.21.15.88:0/3959634233 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faca8098de0 msgr2=0x7faca806d290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:37:02.256 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:02.232+0000 7faca6ffd700 1 --2- 172.21.15.88:0/3959634233 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faca8098de0 0x7faca806d290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:37:02.256 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:02.232+0000 7faca6ffd700 10 monclient: _finish_hunting 0 2023-09-30T01:37:02.256 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:02.232+0000 7faca6ffd700 1 monclient: found mon.noname-c 2023-09-30T01:37:02.256 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:02.232+0000 7faca6ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:37:02.256 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:02.232+0000 7faca6ffd700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.88:3302/0 2023-09-30T01:37:02.257 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:02.232+0000 7faca6ffd700 1 -- 172.21.15.88:0/3959634233 --> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac90009420 con 0x7faca80976f0 2023-09-30T01:37:02.257 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:02.232+0000 7faca77fe700 1 --2- 172.21.15.88:0/3959634233 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7faca81357b0 0x7faca806d7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:37:02.257 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:02.232+0000 7faca6ffd700 1 --2- 172.21.15.88:0/3959634233 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7faca80976f0 0x7faca8134680 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fac9800c790 tx=0x7fac98007c60 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:37:02.257 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:02.232+0000 7fac9effd700 1 -- 172.21.15.88:0/3959634233 <== mon.4 v2:172.21.15.88:3302/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7fac98007dc0 con 0x7faca80976f0 2023-09-30T01:37:02.257 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:02.232+0000 7fac9effd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:37:02.257 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:02.232+0000 7fac9effd700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T01:37:02.257 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:02.232+0000 7fac9effd700 10 monclient: dump: 2023-09-30T01:37:02.257 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:37:02.257 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:37:02.257 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:37:02.258 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:37:02.258 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:37:02.258 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:37:02.258 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:37:02.258 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:37:02.258 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:37:02.258 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:37:02.258 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:37:02.258 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:37:02.259 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:37:02.259 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:37:02.259 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:37:02.259 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:37:02.259 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:02.232+0000 7fac9effd700 10 monclient: _finish_auth 0 2023-09-30T01:37:02.259 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:02.232+0000 7fac9effd700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:37:02.259 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:02.232+0000 7faca7fff700 5 monclient: authenticate success, global_id 4765 2023-09-30T01:37:02.259 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:02.232+0000 7fac9effd700 1 -- 172.21.15.88:0/3959634233 <== mon.4 v2:172.21.15.88:3302/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac980057f0 con 0x7faca80976f0 2023-09-30T01:37:02.259 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:02.232+0000 7faca7fff700 10 monclient: _renew_subs 2023-09-30T01:37:02.259 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:02.232+0000 7faca7fff700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.88:3302/0 2023-09-30T01:37:02.260 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:02.232+0000 7faca7fff700 1 -- 172.21.15.88:0/3959634233 --> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faca8128930 con 0x7faca80976f0 2023-09-30T01:37:02.260 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:02.232+0000 7fac9effd700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:37:02.260 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:02.232+0000 7fac9effd700 1 -- 172.21.15.88:0/3959634233 <== mon.4 v2:172.21.15.88:3302/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7fac98005de0 con 0x7faca80976f0 2023-09-30T01:37:02.260 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:02.232+0000 7fac9effd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:37:02.260 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:02.232+0000 7faca57fa700 4 set_mon_vals no callback set 2023-09-30T01:37:02.260 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:02.232+0000 7fac9effd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T01:37:02.260 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:02.232+0000 7fac9effd700 10 monclient: dump: 2023-09-30T01:37:02.260 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:37:02.260 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:37:02.260 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:37:02.260 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:37:02.261 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:37:02.261 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:37:02.261 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:37:02.261 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:37:02.261 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:37:02.261 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:37:02.261 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:37:02.261 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:37:02.261 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:37:02.261 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:37:02.262 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:37:02.262 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:37:02.262 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:02.232+0000 7faca7fff700 10 monclient: _renew_subs 2023-09-30T01:37:02.262 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:02.232+0000 7faca7fff700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.88:3302/0 2023-09-30T01:37:02.262 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:02.232+0000 7faca7fff700 1 -- 172.21.15.88:0/3959634233 --> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faca8129820 con 0x7faca80976f0 2023-09-30T01:37:02.262 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:02.232+0000 7faca7fff700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:37:02.262 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:02.232+0000 7faca7fff700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:37:02.262 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:02.232+0000 7faca7fff700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.88:3302/0 2023-09-30T01:37:02.262 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:02.232+0000 7faca7fff700 1 -- 172.21.15.88:0/3959634233 --> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faca8071960 con 0x7faca80976f0 2023-09-30T01:37:02.262 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:02.232+0000 7fac9effd700 1 -- 172.21.15.88:0/3959634233 <== mon.4 v2:172.21.15.88:3302/0 4 ==== mgrmap(e 5) v1 ==== 40144+0+0 (secure 0 0 0) 0x7fac98005960 con 0x7faca80976f0 2023-09-30T01:37:02.263 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:02.232+0000 7fac9effd700 1 --2- 172.21.15.88:0/3959634233 >> v2:172.21.15.107:6812/29926 conn(0x7fac88033360 0x7fac88035810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:37:02.263 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:02.232+0000 7fac9effd700 1 -- 172.21.15.88:0/3959634233 <== mon.4 v2:172.21.15.88:3302/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 3502+0+0 (secure 0 0 0) 0x7fac98042440 con 0x7faca80976f0 2023-09-30T01:37:02.263 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:02.232+0000 7fac9ffff700 1 --2- 172.21.15.88:0/3959634233 >> v2:172.21.15.107:6812/29926 conn(0x7fac88033360 0x7fac88035810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:37:02.263 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:02.236+0000 7fac9ffff700 10 monclient: get_auth_request con 0x7fac88033360 auth_method 0 2023-09-30T01:37:02.263 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:02.236+0000 7fac9ffff700 1 --2- 172.21.15.88:0/3959634233 >> v2:172.21.15.107:6812/29926 conn(0x7fac88033360 0x7fac88035810 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fac900091e0 tx=0x7fac90017000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:37:02.264 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:02.244+0000 7fac9effd700 1 -- 172.21.15.88:0/3959634233 <== mon.4 v2:172.21.15.88:3302/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fac98018860 con 0x7faca80976f0 2023-09-30T01:37:02.264 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:02.244+0000 7fac9effd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:37:02.264 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:02.244+0000 7fac9effd700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:37:02.445 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:02.440+0000 7faca7fff700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T01:37:02.446 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:02.440+0000 7faca7fff700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:37:02.446 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:02.440+0000 7faca7fff700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.88:3302/0 2023-09-30T01:37:02.446 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:02.440+0000 7faca7fff700 1 -- 172.21.15.88:0/3959634233 --> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7faca80a0ce0 con 0x7faca80976f0 2023-09-30T01:37:02.447 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:02.440+0000 7fac9effd700 1 -- 172.21.15.88:0/3959634233 <== mon.4 v2:172.21.15.88:3302/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7fac98017020 con 0x7faca80976f0 2023-09-30T01:37:02.447 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:02.440+0000 7fac9effd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:37:02.447 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:02.440+0000 7fac9effd700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:37:02.447 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-09-30T01:37:02.447 INFO:teuthology.orchestra.run.smithi088.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:37:02.450 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:02.444+0000 7faca7fff700 1 -- 172.21.15.88:0/3959634233 >> v2:172.21.15.107:6812/29926 conn(0x7fac88033360 msgr2=0x7fac88035810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:37:02.451 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:02.444+0000 7faca7fff700 1 --2- 172.21.15.88:0/3959634233 >> v2:172.21.15.107:6812/29926 conn(0x7fac88033360 0x7fac88035810 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fac900091e0 tx=0x7fac90017000 comp rx=0 tx=0).stop 2023-09-30T01:37:02.451 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:02.444+0000 7faca7fff700 10 monclient: shutdown 2023-09-30T01:37:02.451 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:02.444+0000 7faca7fff700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:37:02.451 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:02.444+0000 7faca7fff700 1 -- 172.21.15.88:0/3959634233 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7faca80976f0 msgr2=0x7faca8134680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:37:02.451 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:02.444+0000 7faca7fff700 1 --2- 172.21.15.88:0/3959634233 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7faca80976f0 0x7faca8134680 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fac9800c790 tx=0x7fac98007c60 comp rx=0 tx=0).stop 2023-09-30T01:37:02.451 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:02.444+0000 7faca7fff700 1 -- 172.21.15.88:0/3959634233 shutdown_connections 2023-09-30T01:37:02.451 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:02.444+0000 7faca7fff700 1 --2- 172.21.15.88:0/3959634233 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7faca81357b0 0x7faca806d7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:37:02.451 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:02.444+0000 7faca7fff700 1 --2- 172.21.15.88:0/3959634233 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faca8098de0 0x7faca806d290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:37:02.451 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:02.444+0000 7faca7fff700 1 --2- 172.21.15.88:0/3959634233 >> v2:172.21.15.107:6812/29926 conn(0x7fac88033360 0x7fac88035810 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:37:02.451 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:02.444+0000 7faca7fff700 1 --2- 172.21.15.88:0/3959634233 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7faca80976f0 0x7faca8134680 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:37:02.452 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:02.444+0000 7faca7fff700 1 -- 172.21.15.88:0/3959634233 >> 172.21.15.88:0/3959634233 conn(0x7faca80a0910 msgr2=0x7faca809ec00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:37:02.452 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:02.444+0000 7faca7fff700 1 -- 172.21.15.88:0/3959634233 shutdown_connections 2023-09-30T01:37:02.452 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:02.444+0000 7faca7fff700 1 -- 172.21.15.88:0/3959634233 wait complete. 2023-09-30T01:37:05.466 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:37:05.575 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:05.568+0000 7f0223fff700 10 monclient: get_monmap_and_config 2023-09-30T01:37:05.575 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:05.568+0000 7f0223fff700 10 monclient: build_initial_monmap 2023-09-30T01:37:05.575 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:05.568+0000 7f0223fff700 10 monclient: monmap: 2023-09-30T01:37:05.575 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2023-09-30T01:37:05.575 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:37:05.576 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:37:05.575312+0000 2023-09-30T01:37:05.576 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:37:05.575312+0000 2023-09-30T01:37:05.576 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2023-09-30T01:37:05.576 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:37:05.576 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a-legacy 2023-09-30T01:37:05.576 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.107:6789/0 mon.noname-f-legacy 2023-09-30T01:37:05.576 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.noname-b 2023-09-30T01:37:05.576 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.noname-g 2023-09-30T01:37:05.576 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.noname-c 2023-09-30T01:37:05.576 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.noname-h 2023-09-30T01:37:05.576 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.noname-d 2023-09-30T01:37:05.576 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.noname-i 2023-09-30T01:37:05.577 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.noname-e 2023-09-30T01:37:05.577 INFO:teuthology.orchestra.run.smithi088.stderr:9: v2:172.21.15.88:3300/0 mon.noname-a 2023-09-30T01:37:05.577 INFO:teuthology.orchestra.run.smithi088.stderr:10: v2:172.21.15.107:3300/0 mon.noname-f 2023-09-30T01:37:05.577 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:37:05.584 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:05.572+0000 7f0223fff700 1 Processor -- start 2023-09-30T01:37:05.584 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:05.572+0000 7f0223fff700 1 -- start start 2023-09-30T01:37:05.585 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:05.572+0000 7f0223fff700 10 monclient: init 2023-09-30T01:37:05.585 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:05.572+0000 7f0223fff700 10 monclient: _reopen_session rank -1 2023-09-30T01:37:05.585 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:05.572+0000 7f0223fff700 10 monclient: _add_conns ranks=[10,4,1,7,9,6,0,8,5,3,2] 2023-09-30T01:37:05.585 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:05.572+0000 7f0223fff700 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f0224066f90 0x7f022406a480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:37:05.585 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:05.572+0000 7f0223fff700 10 monclient(hunting): picked mon.noname-f con 0x7f0224066f90 addr v2:172.21.15.107:3300/0 2023-09-30T01:37:05.585 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:05.572+0000 7f0223fff700 1 --2- >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f022406a9c0 0x7f022406adb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:37:05.585 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:05.572+0000 7f0223fff700 10 monclient(hunting): picked mon.noname-c con 0x7f022406a9c0 addr [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] 2023-09-30T01:37:05.585 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:05.572+0000 7f0223fff700 10 monclient(hunting): picked mon.noname-f-legacy con 0x7f022406d390 addr v1:172.21.15.107:6789/0 2023-09-30T01:37:05.585 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:05.572+0000 7f0223fff700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f022404bb20 con 0x7f022406d390 2023-09-30T01:37:05.585 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:05.572+0000 7f0223fff700 10 monclient(hunting): start opening mon connection 2023-09-30T01:37:05.585 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:05.572+0000 7f0223fff700 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f022406c100 con 0x7f0224066f90 2023-09-30T01:37:05.585 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:05.572+0000 7f0223fff700 10 monclient(hunting): start opening mon connection 2023-09-30T01:37:05.586 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:05.572+0000 7f0223fff700 1 -- --> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] -- mon_getmap magic: 0 v1 -- 0x7f022409ecd0 con 0x7f022406a9c0 2023-09-30T01:37:05.586 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:05.572+0000 7f0223fff700 10 monclient(hunting): _renew_subs 2023-09-30T01:37:05.586 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:05.572+0000 7f0223fff700 10 monclient(hunting): authenticate will time out at 1097.613354s 2023-09-30T01:37:05.586 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:05.572+0000 7f02227fc700 1 --2- >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f022406a9c0 0x7f022406adb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:37:05.586 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:05.572+0000 7f02227fc700 1 --2- >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f022406a9c0 0x7f022406adb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3302/0 says I am v2:172.21.15.88:60978/0 (socket says 172.21.15.88:60978) 2023-09-30T01:37:05.586 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:05.572+0000 7f02227fc700 1 -- 172.21.15.88:0/3498813528 learned_addr learned my addr 172.21.15.88:0/3498813528 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-09-30T01:37:05.586 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:05.572+0000 7f02227fc700 10 monclient(hunting): get_auth_request con 0x7f022406a9c0 auth_method 0 2023-09-30T01:37:05.586 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:05.572+0000 7f02227fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:37:05.586 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:05.572+0000 7f02227fc700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:37:05.586 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:05.572+0000 7f02227fc700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:37:05.586 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:05.572+0000 7f02227fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:37:05.587 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:05.572+0000 7f02227fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:37:05.587 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:05.572+0000 7f02227fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:37:05.587 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:05.572+0000 7f0222ffd700 1 --2- 172.21.15.88:0/3498813528 >> v2:172.21.15.107:3300/0 conn(0x7f0224066f90 0x7f022406a480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:37:05.587 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:05.572+0000 7f0222ffd700 10 monclient(hunting): get_auth_request con 0x7f0224066f90 auth_method 0 2023-09-30T01:37:05.587 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:05.572+0000 7f0222ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:37:05.587 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:05.572+0000 7f0222ffd700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:37:05.587 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:05.572+0000 7f0222ffd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:37:05.587 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:05.572+0000 7f02227fc700 10 monclient(hunting): handle_auth_done global_id 4774 payload 274 2023-09-30T01:37:05.587 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:05.572+0000 7f02227fc700 1 -- 172.21.15.88:0/3498813528 >> v2:172.21.15.107:3300/0 conn(0x7f0224066f90 msgr2=0x7f022406a480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:37:05.587 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:05.572+0000 7f02227fc700 1 --2- 172.21.15.88:0/3498813528 >> v2:172.21.15.107:3300/0 conn(0x7f0224066f90 0x7f022406a480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:37:05.587 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:05.572+0000 7f02227fc700 1 -- 172.21.15.88:0/3498813528 >> v1:172.21.15.107:6789/0 conn(0x7f022406d390 legacy=0x7f022406d7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:37:05.587 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:05.572+0000 7f02227fc700 10 monclient: _finish_hunting 0 2023-09-30T01:37:05.588 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:05.572+0000 7f02227fc700 1 monclient: found mon.noname-c 2023-09-30T01:37:05.588 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:05.572+0000 7f02227fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:37:05.588 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:05.572+0000 7f02227fc700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.88:3302/0 2023-09-30T01:37:05.588 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:05.572+0000 7f02227fc700 1 -- 172.21.15.88:0/3498813528 --> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f022406b2f0 con 0x7f022406a9c0 2023-09-30T01:37:05.588 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:05.572+0000 7f0222ffd700 1 --2- 172.21.15.88:0/3498813528 >> v2:172.21.15.107:3300/0 conn(0x7f0224066f90 0x7f022406a480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-30T01:37:05.588 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:05.572+0000 7f02227fc700 1 --2- 172.21.15.88:0/3498813528 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f022406a9c0 0x7f022406adb0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f02140099b0 tx=0x7f0214009250 comp rx=0 tx=0).ready entity=mon.4 client_cookie=732d771c0f19807b server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:37:05.588 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:05.572+0000 7f0221ffb700 1 -- 172.21.15.88:0/3498813528 <== mon.4 v2:172.21.15.88:3302/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f021400bce0 con 0x7f022406a9c0 2023-09-30T01:37:05.588 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:05.572+0000 7f0221ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:37:05.588 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:05.572+0000 7f0221ffb700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T01:37:05.588 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:05.572+0000 7f0221ffb700 10 monclient: dump: 2023-09-30T01:37:05.588 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:37:05.588 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:37:05.589 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:37:05.589 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:37:05.589 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:37:05.589 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:37:05.589 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:37:05.589 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:37:05.589 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:37:05.589 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:37:05.589 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:37:05.589 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:37:05.589 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:37:05.589 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:37:05.589 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:37:05.590 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:37:05.590 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:05.572+0000 7f0221ffb700 10 monclient: _finish_auth 0 2023-09-30T01:37:05.590 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:05.572+0000 7f0221ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:37:05.590 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:05.572+0000 7f0221ffb700 1 -- 172.21.15.88:0/3498813528 <== mon.4 v2:172.21.15.88:3302/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f021400be60 con 0x7f022406a9c0 2023-09-30T01:37:05.590 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:05.572+0000 7f0223fff700 5 monclient: authenticate success, global_id 4774 2023-09-30T01:37:05.590 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:05.572+0000 7f0223fff700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:37:05.590 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:05.572+0000 7f0221ffb700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:37:05.590 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:05.572+0000 7f0223fff700 10 monclient: get_monmap_and_config success 2023-09-30T01:37:05.590 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:05.572+0000 7f0221ffb700 1 -- 172.21.15.88:0/3498813528 <== mon.4 v2:172.21.15.88:3302/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f02140118a0 con 0x7f022406a9c0 2023-09-30T01:37:05.590 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:05.572+0000 7f0221ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:37:05.590 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:05.572+0000 7f0223fff700 4 set_mon_vals no callback set 2023-09-30T01:37:05.590 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:05.572+0000 7f0221ffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T01:37:05.591 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:05.572+0000 7f0221ffb700 10 monclient: dump: 2023-09-30T01:37:05.591 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:37:05.591 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:37:05.591 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:37:05.591 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:37:05.591 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:37:05.591 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:37:05.591 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:37:05.591 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:37:05.591 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:37:05.591 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:37:05.591 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:37:05.592 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:37:05.592 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:37:05.592 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:37:05.592 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:37:05.592 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:37:05.592 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:05.572+0000 7f0223fff700 10 monclient: shutdown 2023-09-30T01:37:05.592 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:05.572+0000 7f0223fff700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:37:05.592 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:05.572+0000 7f0223fff700 1 -- 172.21.15.88:0/3498813528 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f022406a9c0 msgr2=0x7f022406adb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:37:05.592 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:05.572+0000 7f0223fff700 1 --2- 172.21.15.88:0/3498813528 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f022406a9c0 0x7f022406adb0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f02140099b0 tx=0x7f0214009250 comp rx=0 tx=0).stop 2023-09-30T01:37:05.592 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:05.572+0000 7f0223fff700 1 -- 172.21.15.88:0/3498813528 shutdown_connections 2023-09-30T01:37:05.592 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:05.572+0000 7f0223fff700 1 --2- 172.21.15.88:0/3498813528 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f022406a9c0 0x7f022406adb0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:37:05.592 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:05.572+0000 7f0223fff700 1 --2- 172.21.15.88:0/3498813528 >> v2:172.21.15.107:3300/0 conn(0x7f0224066f90 0x7f022406a480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:37:05.593 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:05.572+0000 7f0223fff700 1 -- 172.21.15.88:0/3498813528 >> 172.21.15.88:0/3498813528 conn(0x7f0224063610 msgr2=0x7f02240a2b90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:37:05.593 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:05.572+0000 7f0223fff700 1 -- 172.21.15.88:0/3498813528 shutdown_connections 2023-09-30T01:37:05.593 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:05.572+0000 7f0223fff700 1 -- 172.21.15.88:0/3498813528 wait complete. 2023-09-30T01:37:05.593 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:05.572+0000 7f0223fff700 10 monclient: build_initial_monmap 2023-09-30T01:37:05.593 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:05.576+0000 7f0223fff700 10 monclient: monmap: 2023-09-30T01:37:05.593 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2023-09-30T01:37:05.593 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:37:05.593 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 0.000000 2023-09-30T01:37:05.593 INFO:teuthology.orchestra.run.smithi088.stderr:created 0.000000 2023-09-30T01:37:05.593 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2023-09-30T01:37:05.593 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 1 2023-09-30T01:37:05.593 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.noname-a 2023-09-30T01:37:05.593 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.noname-f 2023-09-30T01:37:05.594 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.noname-b 2023-09-30T01:37:05.594 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.noname-g 2023-09-30T01:37:05.594 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.noname-c 2023-09-30T01:37:05.594 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.noname-h 2023-09-30T01:37:05.594 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.noname-d 2023-09-30T01:37:05.594 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.noname-i 2023-09-30T01:37:05.594 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.noname-e 2023-09-30T01:37:05.594 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:37:05.594 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:05.576+0000 7f0223fff700 1 Processor -- start 2023-09-30T01:37:05.594 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:05.576+0000 7f0223fff700 1 -- start start 2023-09-30T01:37:05.594 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:05.576+0000 7f0223fff700 10 monclient: init 2023-09-30T01:37:05.594 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:05.576+0000 7f0223fff700 10 monclient: _reopen_session rank -1 2023-09-30T01:37:05.595 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:05.576+0000 7f0223fff700 10 monclient: _add_conns ranks=[1,6,0,8,5,7,2,4,3] 2023-09-30T01:37:05.595 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:05.576+0000 7f0223fff700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0224066f90 0x7f022413a920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:37:05.595 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:05.576+0000 7f0223fff700 10 monclient(hunting): picked mon.noname-f con 0x7f0224066f90 addr [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] 2023-09-30T01:37:05.595 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:05.576+0000 7f0223fff700 1 --2- >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7f022406a9c0 0x7f0224139fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:37:05.595 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:05.576+0000 7f0223fff700 10 monclient(hunting): picked mon.noname-d con 0x7f022406a9c0 addr [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] 2023-09-30T01:37:05.595 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:05.576+0000 7f0223fff700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f022406d390 0x7f022413de90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:37:05.595 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:05.576+0000 7f0223fff700 10 monclient(hunting): picked mon.noname-a con 0x7f022406d390 addr [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] 2023-09-30T01:37:05.595 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:05.576+0000 7f0223fff700 10 monclient(hunting): start opening mon connection 2023-09-30T01:37:05.595 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:05.576+0000 7f0223fff700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f022411f9a0 con 0x7f022406d390 2023-09-30T01:37:05.595 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:05.576+0000 7f0223fff700 10 monclient(hunting): start opening mon connection 2023-09-30T01:37:05.595 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:05.576+0000 7f0223fff700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f022411f350 con 0x7f0224066f90 2023-09-30T01:37:05.595 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:05.576+0000 7f0223fff700 10 monclient(hunting): start opening mon connection 2023-09-30T01:37:05.596 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:05.576+0000 7f0223fff700 1 -- --> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] -- mon_getmap magic: 0 v1 -- 0x7f022411cde0 con 0x7f022406a9c0 2023-09-30T01:37:05.596 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:05.576+0000 7f0223fff700 10 monclient(hunting): _renew_subs 2023-09-30T01:37:05.596 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:05.576+0000 7f0223fff700 10 monclient(hunting): authenticate will time out at 1097.617412s 2023-09-30T01:37:05.596 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:05.576+0000 7f02227fc700 1 --2- >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7f022406a9c0 0x7f0224139fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:37:05.596 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:05.576+0000 7f02237fe700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f022406d390 0x7f022413de90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:37:05.596 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:05.576+0000 7f02227fc700 1 --2- >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7f022406a9c0 0x7f0224139fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3303/0 says I am v2:172.21.15.88:42704/0 (socket says 172.21.15.88:42704) 2023-09-30T01:37:05.596 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:05.576+0000 7f02237fe700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f022406d390 0x7f022413de90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:45392/0 (socket says 172.21.15.88:45392) 2023-09-30T01:37:05.596 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:05.576+0000 7f02227fc700 1 -- 172.21.15.88:0/440691586 learned_addr learned my addr 172.21.15.88:0/440691586 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-09-30T01:37:05.596 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:05.576+0000 7f0222ffd700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0224066f90 0x7f022413a920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:37:05.596 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:05.576+0000 7f02227fc700 10 monclient(hunting): get_auth_request con 0x7f022406a9c0 auth_method 0 2023-09-30T01:37:05.596 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:05.576+0000 7f02227fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:37:05.596 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:05.576+0000 7f02227fc700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:37:05.597 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:05.576+0000 7f02227fc700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:37:05.597 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:05.576+0000 7f02237fe700 10 monclient(hunting): get_auth_request con 0x7f022406d390 auth_method 0 2023-09-30T01:37:05.597 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:05.576+0000 7f02237fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:37:05.597 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:05.576+0000 7f02237fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:37:05.597 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:05.576+0000 7f02237fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:37:05.597 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:05.576+0000 7f0222ffd700 10 monclient(hunting): get_auth_request con 0x7f0224066f90 auth_method 0 2023-09-30T01:37:05.597 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:05.576+0000 7f0222ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:37:05.597 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:05.576+0000 7f0222ffd700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:37:05.597 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:05.576+0000 7f0222ffd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:37:05.597 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:05.576+0000 7f02227fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:37:05.597 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:05.576+0000 7f02227fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:37:05.597 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:05.576+0000 7f02227fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:37:05.597 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:05.576+0000 7f02237fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:37:05.598 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:05.576+0000 7f02237fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:37:05.598 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:05.576+0000 7f02237fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:37:05.598 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:05.576+0000 7f0222ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:37:05.598 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:05.576+0000 7f0222ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:37:05.598 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:05.576+0000 7f0222ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:37:05.598 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:05.576+0000 7f02227fc700 10 monclient(hunting): handle_auth_done global_id 4650 payload 824 2023-09-30T01:37:05.598 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:05.576+0000 7f02227fc700 1 -- 172.21.15.88:0/440691586 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0224066f90 msgr2=0x7f022413a920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:37:05.598 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:05.576+0000 7f02227fc700 1 --2- 172.21.15.88:0/440691586 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0224066f90 0x7f022413a920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:37:05.598 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:05.576+0000 7f02227fc700 1 -- 172.21.15.88:0/440691586 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f022406d390 msgr2=0x7f022413de90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:37:05.598 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:05.576+0000 7f02227fc700 1 --2- 172.21.15.88:0/440691586 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f022406d390 0x7f022413de90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:37:05.598 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:05.576+0000 7f02227fc700 10 monclient: _finish_hunting 0 2023-09-30T01:37:05.599 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:05.576+0000 7f02227fc700 1 monclient: found mon.noname-d 2023-09-30T01:37:05.599 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:05.576+0000 7f02227fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:37:05.599 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:05.576+0000 7f02227fc700 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.88:3303/0 2023-09-30T01:37:05.599 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:05.576+0000 7f02227fc700 1 -- 172.21.15.88:0/440691586 --> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02140094f0 con 0x7f022406a9c0 2023-09-30T01:37:05.599 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:05.576+0000 7f02237fe700 1 --2- 172.21.15.88:0/440691586 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f022406d390 0x7f022413de90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:37:05.599 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:05.576+0000 7f02227fc700 1 --2- 172.21.15.88:0/440691586 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7f022406a9c0 0x7f0224139fd0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f02140090b0 tx=0x7f0214017000 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:37:05.599 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:05.576+0000 7f020affd700 1 -- 172.21.15.88:0/440691586 <== mon.6 v2:172.21.15.88:3303/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f0214011c50 con 0x7f022406a9c0 2023-09-30T01:37:05.599 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:05.576+0000 7f020affd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:37:05.599 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:05.576+0000 7f020affd700 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2023-09-30T01:37:05.599 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:05.576+0000 7f020affd700 10 monclient: dump: 2023-09-30T01:37:05.599 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:37:05.599 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:37:05.600 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:37:05.600 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:37:05.600 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:37:05.601 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:37:05.602 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:37:05.602 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:37:05.602 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:37:05.602 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:37:05.602 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:37:05.602 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:37:05.602 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:37:05.602 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:37:05.602 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:37:05.602 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:37:05.602 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:05.576+0000 7f020affd700 10 monclient: _finish_auth 0 2023-09-30T01:37:05.603 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:05.576+0000 7f020affd700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:37:05.603 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:05.576+0000 7f020affd700 1 -- 172.21.15.88:0/440691586 <== mon.6 v2:172.21.15.88:3303/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f021401f070 con 0x7f022406a9c0 2023-09-30T01:37:05.603 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:05.576+0000 7f0223fff700 5 monclient: authenticate success, global_id 4650 2023-09-30T01:37:05.603 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:05.576+0000 7f0223fff700 10 monclient: _renew_subs 2023-09-30T01:37:05.603 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:05.576+0000 7f0223fff700 10 monclient: _send_mon_message to mon.d at v2:172.21.15.88:3303/0 2023-09-30T01:37:05.603 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:05.576+0000 7f0223fff700 1 -- 172.21.15.88:0/440691586 --> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f022411f070 con 0x7f022406a9c0 2023-09-30T01:37:05.603 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:05.576+0000 7f020affd700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:37:05.603 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:05.576+0000 7f020affd700 1 -- 172.21.15.88:0/440691586 <== mon.6 v2:172.21.15.88:3303/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f0214011e30 con 0x7f022406a9c0 2023-09-30T01:37:05.603 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:05.576+0000 7f020affd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:37:05.603 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:05.576+0000 7f020affd700 10 monclient: got monmap 1 from mon.d (according to old e1) 2023-09-30T01:37:05.603 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:05.576+0000 7f02217fa700 4 set_mon_vals no callback set 2023-09-30T01:37:05.603 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:05.576+0000 7f020affd700 10 monclient: dump: 2023-09-30T01:37:05.603 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:37:05.604 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:37:05.604 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:37:05.604 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:37:05.604 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:37:05.604 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:37:05.604 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:37:05.604 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:37:05.604 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:37:05.604 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:37:05.604 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:37:05.604 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:37:05.604 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:37:05.604 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:37:05.605 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:37:05.605 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:37:05.605 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:05.576+0000 7f0223fff700 10 monclient: _renew_subs 2023-09-30T01:37:05.605 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:05.576+0000 7f0223fff700 10 monclient: _send_mon_message to mon.d at v2:172.21.15.88:3303/0 2023-09-30T01:37:05.605 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:05.576+0000 7f0223fff700 1 -- 172.21.15.88:0/440691586 --> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f022413a510 con 0x7f022406a9c0 2023-09-30T01:37:05.605 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:05.576+0000 7f020affd700 1 -- 172.21.15.88:0/440691586 <== mon.6 v2:172.21.15.88:3303/0 4 ==== mgrmap(e 5) v1 ==== 40144+0+0 (secure 0 0 0) 0x7f021401a430 con 0x7f022406a9c0 2023-09-30T01:37:05.605 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:05.576+0000 7f0208ff9700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:37:05.605 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:05.576+0000 7f0208ff9700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:37:05.605 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:05.576+0000 7f0208ff9700 10 monclient: _send_mon_message to mon.d at v2:172.21.15.88:3303/0 2023-09-30T01:37:05.605 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:05.576+0000 7f0208ff9700 1 -- 172.21.15.88:0/440691586 --> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f01fc0050b0 con 0x7f022406a9c0 2023-09-30T01:37:05.605 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:05.580+0000 7f020affd700 1 --2- 172.21.15.88:0/440691586 >> v2:172.21.15.107:6812/29926 conn(0x7f0210033360 0x7f0210035810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:37:05.605 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:05.580+0000 7f020affd700 1 -- 172.21.15.88:0/440691586 <== mon.6 v2:172.21.15.88:3303/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 3502+0+0 (secure 0 0 0) 0x7f0214021070 con 0x7f022406a9c0 2023-09-30T01:37:05.606 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:05.580+0000 7f0222ffd700 1 --2- 172.21.15.88:0/440691586 >> v2:172.21.15.107:6812/29926 conn(0x7f0210033360 0x7f0210035810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:37:05.606 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:05.580+0000 7f0222ffd700 10 monclient: get_auth_request con 0x7f0210033360 auth_method 0 2023-09-30T01:37:05.606 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:05.580+0000 7f0222ffd700 1 --2- 172.21.15.88:0/440691586 >> v2:172.21.15.107:6812/29926 conn(0x7f0210033360 0x7f0210035810 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f0218006f00 tx=0x7f0218008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:37:05.606 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:05.584+0000 7f020affd700 1 -- 172.21.15.88:0/440691586 <== mon.6 v2:172.21.15.88:3303/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f02140255c0 con 0x7f022406a9c0 2023-09-30T01:37:05.606 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:05.584+0000 7f020affd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:37:05.606 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:05.584+0000 7f020affd700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:37:05.784 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:05.780+0000 7f0208ff9700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T01:37:05.784 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:05.780+0000 7f0208ff9700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:37:05.784 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:05.780+0000 7f0208ff9700 10 monclient: _send_mon_message to mon.d at v2:172.21.15.88:3303/0 2023-09-30T01:37:05.784 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:05.780+0000 7f0208ff9700 1 -- 172.21.15.88:0/440691586 --> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f01fc005360 con 0x7f022406a9c0 2023-09-30T01:37:05.786 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:05.780+0000 7f020affd700 1 -- 172.21.15.88:0/440691586 <== mon.6 v2:172.21.15.88:3303/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f0214015030 con 0x7f022406a9c0 2023-09-30T01:37:05.786 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:05.780+0000 7f020affd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:37:05.786 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:05.780+0000 7f020affd700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:37:05.787 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-09-30T01:37:05.787 INFO:teuthology.orchestra.run.smithi088.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:37:05.789 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:05.784+0000 7f0208ff9700 1 -- 172.21.15.88:0/440691586 >> v2:172.21.15.107:6812/29926 conn(0x7f0210033360 msgr2=0x7f0210035810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:37:05.789 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:05.784+0000 7f0208ff9700 1 --2- 172.21.15.88:0/440691586 >> v2:172.21.15.107:6812/29926 conn(0x7f0210033360 0x7f0210035810 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f0218006f00 tx=0x7f0218008040 comp rx=0 tx=0).stop 2023-09-30T01:37:05.789 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:05.784+0000 7f0208ff9700 10 monclient: shutdown 2023-09-30T01:37:05.789 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:05.784+0000 7f0208ff9700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:37:05.789 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:05.784+0000 7f0208ff9700 1 -- 172.21.15.88:0/440691586 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7f022406a9c0 msgr2=0x7f0224139fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:37:05.789 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:05.784+0000 7f0208ff9700 1 --2- 172.21.15.88:0/440691586 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7f022406a9c0 0x7f0224139fd0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f02140090b0 tx=0x7f0214017000 comp rx=0 tx=0).stop 2023-09-30T01:37:05.789 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:05.784+0000 7f0208ff9700 1 -- 172.21.15.88:0/440691586 shutdown_connections 2023-09-30T01:37:05.789 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:05.784+0000 7f0208ff9700 1 --2- 172.21.15.88:0/440691586 >> v2:172.21.15.107:6812/29926 conn(0x7f0210033360 0x7f0210035810 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:37:05.789 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:05.784+0000 7f0208ff9700 1 --2- 172.21.15.88:0/440691586 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f022406d390 0x7f022413de90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:37:05.789 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:05.784+0000 7f0208ff9700 1 --2- 172.21.15.88:0/440691586 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7f022406a9c0 0x7f0224139fd0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:37:05.790 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:05.784+0000 7f0208ff9700 1 --2- 172.21.15.88:0/440691586 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0224066f90 0x7f022413a920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:37:05.790 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:05.784+0000 7f0208ff9700 1 -- 172.21.15.88:0/440691586 >> 172.21.15.88:0/440691586 conn(0x7f0224063610 msgr2=0x7f022406b8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:37:05.790 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:05.784+0000 7f0208ff9700 1 -- 172.21.15.88:0/440691586 shutdown_connections 2023-09-30T01:37:05.790 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:05.784+0000 7f0208ff9700 1 -- 172.21.15.88:0/440691586 wait complete. 2023-09-30T01:37:08.800 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:37:08.899 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:08.892+0000 7f9286d44700 10 monclient: get_monmap_and_config 2023-09-30T01:37:08.899 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:08.892+0000 7f9286d44700 10 monclient: build_initial_monmap 2023-09-30T01:37:08.899 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:08.892+0000 7f9286d44700 10 monclient: monmap: 2023-09-30T01:37:08.899 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2023-09-30T01:37:08.899 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:37:08.899 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:37:08.898838+0000 2023-09-30T01:37:08.899 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:37:08.898838+0000 2023-09-30T01:37:08.899 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2023-09-30T01:37:08.899 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:37:08.899 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a-legacy 2023-09-30T01:37:08.900 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.107:6789/0 mon.noname-f-legacy 2023-09-30T01:37:08.900 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.noname-b 2023-09-30T01:37:08.900 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.noname-g 2023-09-30T01:37:08.900 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.noname-c 2023-09-30T01:37:08.900 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.noname-h 2023-09-30T01:37:08.900 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.noname-d 2023-09-30T01:37:08.900 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.noname-i 2023-09-30T01:37:08.900 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.noname-e 2023-09-30T01:37:08.900 INFO:teuthology.orchestra.run.smithi088.stderr:9: v2:172.21.15.88:3300/0 mon.noname-a 2023-09-30T01:37:08.900 INFO:teuthology.orchestra.run.smithi088.stderr:10: v2:172.21.15.107:3300/0 mon.noname-f 2023-09-30T01:37:08.900 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:37:08.911 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:08.892+0000 7f9286d44700 1 Processor -- start 2023-09-30T01:37:08.911 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:08.892+0000 7f9286d44700 1 -- start start 2023-09-30T01:37:08.911 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:08.892+0000 7f9286d44700 10 monclient: init 2023-09-30T01:37:08.911 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:08.896+0000 7f9286d44700 10 monclient: _reopen_session rank -1 2023-09-30T01:37:08.912 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:08.896+0000 7f9286d44700 10 monclient: _add_conns ranks=[6,1,10,8,5,2,9,4,7,3,0] 2023-09-30T01:37:08.912 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:08.896+0000 7f9286d44700 1 --2- >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7f92800a3380 0x7f92800a48b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:37:08.912 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:08.896+0000 7f9286d44700 10 monclient(hunting): picked mon.noname-d con 0x7f92800a3380 addr [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] 2023-09-30T01:37:08.912 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:08.896+0000 7f9286d44700 10 monclient(hunting): picked mon.noname-f-legacy con 0x7f928009ffd0 addr v1:172.21.15.107:6789/0 2023-09-30T01:37:08.912 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:08.896+0000 7f9286d44700 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f92800a03a0 0x7f9280133ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:37:08.912 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:08.896+0000 7f9286d44700 10 monclient(hunting): picked mon.noname-f con 0x7f92800a03a0 addr v2:172.21.15.107:3300/0 2023-09-30T01:37:08.912 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:08.896+0000 7f9286d44700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f928004bb20 con 0x7f928009ffd0 2023-09-30T01:37:08.912 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:08.896+0000 7f9286d44700 10 monclient(hunting): start opening mon connection 2023-09-30T01:37:08.912 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:08.896+0000 7f9286d44700 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9280097ab0 con 0x7f92800a03a0 2023-09-30T01:37:08.912 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:08.896+0000 7f9286d44700 10 monclient(hunting): start opening mon connection 2023-09-30T01:37:08.912 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:08.896+0000 7f9286d44700 1 -- --> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] -- mon_getmap magic: 0 v1 -- 0x7f9280098480 con 0x7f92800a3380 2023-09-30T01:37:08.912 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:08.896+0000 7f9286d44700 10 monclient(hunting): _renew_subs 2023-09-30T01:37:08.913 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:08.896+0000 7f9286d44700 10 monclient(hunting): authenticate will time out at 1100.936317s 2023-09-30T01:37:08.913 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:08.896+0000 7f9285d42700 1 --2- >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7f92800a3380 0x7f92800a48b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:37:08.913 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:08.896+0000 7f9286543700 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f92800a03a0 0x7f9280133ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:37:08.913 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:08.896+0000 7f9285d42700 1 --2- >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7f92800a3380 0x7f92800a48b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3303/0 says I am v2:172.21.15.88:42720/0 (socket says 172.21.15.88:42720) 2023-09-30T01:37:08.913 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:08.896+0000 7f9285d42700 1 -- 172.21.15.88:0/3504250234 learned_addr learned my addr 172.21.15.88:0/3504250234 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-09-30T01:37:08.913 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:08.896+0000 7f9285d42700 10 monclient(hunting): get_auth_request con 0x7f92800a3380 auth_method 0 2023-09-30T01:37:08.913 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:08.896+0000 7f9285d42700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:37:08.913 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:08.896+0000 7f9285d42700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:37:08.913 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:08.896+0000 7f9285d42700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:37:08.913 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:08.896+0000 7f9286543700 10 monclient(hunting): get_auth_request con 0x7f92800a03a0 auth_method 0 2023-09-30T01:37:08.913 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:08.896+0000 7f9286543700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:37:08.913 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:08.896+0000 7f9286543700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:37:08.914 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:08.896+0000 7f9286543700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:37:08.914 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:08.896+0000 7f9285d42700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:37:08.914 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:08.896+0000 7f9285d42700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:37:08.914 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:08.896+0000 7f9285d42700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:37:08.914 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:08.896+0000 7f9286543700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:37:08.914 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:08.896+0000 7f9286543700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:37:08.914 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:08.896+0000 7f9286543700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:37:08.914 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:08.896+0000 7f9285d42700 10 monclient(hunting): handle_auth_done global_id 4659 payload 274 2023-09-30T01:37:08.914 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:08.896+0000 7f9284d40700 1 -- 172.21.15.88:0/3504250234 <== mon.1 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1040412832 0 0) 0x7f928004bb20 con 0x7f928009ffd0 2023-09-30T01:37:08.914 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:08.896+0000 7f9285d42700 1 -- 172.21.15.88:0/3504250234 >> v2:172.21.15.107:3300/0 conn(0x7f92800a03a0 msgr2=0x7f9280133ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:37:08.915 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:08.896+0000 7f9285d42700 1 --2- 172.21.15.88:0/3504250234 >> v2:172.21.15.107:3300/0 conn(0x7f92800a03a0 0x7f9280133ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:37:08.915 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:08.896+0000 7f9285d42700 1 -- 172.21.15.88:0/3504250234 >> v1:172.21.15.107:6789/0 conn(0x7f928009ffd0 legacy=0x7f92801333b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:37:08.915 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:08.896+0000 7f9285d42700 10 monclient: _finish_hunting 0 2023-09-30T01:37:08.915 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:08.896+0000 7f9285d42700 1 monclient: found mon.noname-d 2023-09-30T01:37:08.915 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:08.896+0000 7f9285d42700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:37:08.915 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:08.896+0000 7f9285d42700 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.88:3303/0 2023-09-30T01:37:08.915 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:08.896+0000 7f9285d42700 1 -- 172.21.15.88:0/3504250234 --> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92800a3750 con 0x7f92800a3380 2023-09-30T01:37:08.915 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:08.896+0000 7f9284d40700 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2023-09-30T01:37:08.915 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:08.896+0000 7f9285d42700 1 --2- 172.21.15.88:0/3504250234 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7f92800a3380 0x7f92800a48b0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f927c0099b0 tx=0x7f927c009250 comp rx=0 tx=0).ready entity=mon.6 client_cookie=c327658012572fab server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:37:08.915 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:08.896+0000 7f9284d40700 1 -- 172.21.15.88:0/3504250234 <== mon.6 v2:172.21.15.88:3303/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f927c009ea0 con 0x7f92800a3380 2023-09-30T01:37:08.915 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:08.896+0000 7f9284d40700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:37:08.915 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:08.896+0000 7f9284d40700 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2023-09-30T01:37:08.916 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:08.896+0000 7f9284d40700 10 monclient: dump: 2023-09-30T01:37:08.916 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:37:08.916 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:37:08.916 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:37:08.916 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:37:08.916 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:37:08.916 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:37:08.916 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:37:08.916 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:37:08.916 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:37:08.916 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:37:08.916 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:37:08.917 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:37:08.917 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:37:08.917 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:37:08.917 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:37:08.917 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:37:08.917 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:08.896+0000 7f9284d40700 10 monclient: _finish_auth 0 2023-09-30T01:37:08.917 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:08.896+0000 7f9284d40700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:37:08.917 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:08.896+0000 7f9284d40700 1 -- 172.21.15.88:0/3504250234 <== mon.6 v2:172.21.15.88:3303/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f927c00bce0 con 0x7f92800a3380 2023-09-30T01:37:08.917 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:08.896+0000 7f9286d44700 5 monclient: authenticate success, global_id 4659 2023-09-30T01:37:08.917 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:08.896+0000 7f9284d40700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:37:08.917 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:08.896+0000 7f9286d44700 10 monclient: get_monmap_and_config success 2023-09-30T01:37:08.917 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:08.896+0000 7f9286d44700 4 set_mon_vals no callback set 2023-09-30T01:37:08.918 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:08.896+0000 7f9284d40700 1 -- 172.21.15.88:0/3504250234 <== mon.6 v2:172.21.15.88:3303/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f927c00be50 con 0x7f92800a3380 2023-09-30T01:37:08.918 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:08.896+0000 7f9284d40700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:37:08.918 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:08.896+0000 7f9284d40700 10 monclient: got monmap 1 from mon.d (according to old e1) 2023-09-30T01:37:08.918 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:08.896+0000 7f9284d40700 10 monclient: dump: 2023-09-30T01:37:08.918 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:37:08.918 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:37:08.918 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:37:08.918 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:37:08.918 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:37:08.918 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:37:08.918 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:37:08.919 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:37:08.919 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:37:08.919 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:37:08.919 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:37:08.919 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:37:08.919 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:37:08.919 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:37:08.919 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:37:08.919 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:37:08.919 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:08.896+0000 7f9286d44700 10 monclient: shutdown 2023-09-30T01:37:08.919 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:08.896+0000 7f9286d44700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:37:08.919 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:08.896+0000 7f9286d44700 1 -- 172.21.15.88:0/3504250234 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7f92800a3380 msgr2=0x7f92800a48b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:37:08.920 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:08.896+0000 7f9286d44700 1 --2- 172.21.15.88:0/3504250234 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7f92800a3380 0x7f92800a48b0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f927c0099b0 tx=0x7f927c009250 comp rx=0 tx=0).stop 2023-09-30T01:37:08.920 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:08.896+0000 7f9286d44700 1 -- 172.21.15.88:0/3504250234 shutdown_connections 2023-09-30T01:37:08.920 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:08.896+0000 7f9286d44700 1 --2- 172.21.15.88:0/3504250234 >> v2:172.21.15.107:3300/0 conn(0x7f92800a03a0 0x7f9280133ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:37:08.920 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:08.896+0000 7f9286d44700 1 --2- 172.21.15.88:0/3504250234 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7f92800a3380 0x7f92800a48b0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:37:08.920 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:08.896+0000 7f9286d44700 1 -- 172.21.15.88:0/3504250234 >> 172.21.15.88:0/3504250234 conn(0x7f928006e5e0 msgr2=0x7f928006e9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:37:08.920 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:08.896+0000 7f9286d44700 1 -- 172.21.15.88:0/3504250234 shutdown_connections 2023-09-30T01:37:08.920 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:08.896+0000 7f9286d44700 1 -- 172.21.15.88:0/3504250234 wait complete. 2023-09-30T01:37:08.921 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:08.896+0000 7f9286d44700 10 monclient: build_initial_monmap 2023-09-30T01:37:08.921 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:08.900+0000 7f9286d44700 10 monclient: monmap: 2023-09-30T01:37:08.921 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2023-09-30T01:37:08.921 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:37:08.921 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 0.000000 2023-09-30T01:37:08.921 INFO:teuthology.orchestra.run.smithi088.stderr:created 0.000000 2023-09-30T01:37:08.921 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2023-09-30T01:37:08.921 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 1 2023-09-30T01:37:08.921 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.noname-a 2023-09-30T01:37:08.921 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.noname-f 2023-09-30T01:37:08.921 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.noname-b 2023-09-30T01:37:08.921 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.noname-g 2023-09-30T01:37:08.921 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.noname-c 2023-09-30T01:37:08.922 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.noname-h 2023-09-30T01:37:08.922 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.noname-d 2023-09-30T01:37:08.922 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.noname-i 2023-09-30T01:37:08.922 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.noname-e 2023-09-30T01:37:08.922 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:37:08.922 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:08.900+0000 7f9286d44700 1 Processor -- start 2023-09-30T01:37:08.922 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:08.900+0000 7f9286d44700 1 -- start start 2023-09-30T01:37:08.922 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:08.900+0000 7f9286d44700 10 monclient: init 2023-09-30T01:37:08.922 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:08.900+0000 7f9286d44700 10 monclient: _reopen_session rank -1 2023-09-30T01:37:08.922 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:08.900+0000 7f9286d44700 10 monclient: _add_conns ranks=[5,6,4,1,7,3,2,8,0] 2023-09-30T01:37:08.922 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:08.900+0000 7f9286d44700 1 --2- >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7f928009ffd0 0x7f9280141f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:37:08.922 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:08.900+0000 7f9286d44700 10 monclient(hunting): picked mon.noname-h con 0x7f928009ffd0 addr [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] 2023-09-30T01:37:08.923 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:08.900+0000 7f9286d44700 1 --2- >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7f92800a03a0 0x7f928013e070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:37:08.923 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:08.900+0000 7f9286d44700 10 monclient(hunting): picked mon.noname-d con 0x7f92800a03a0 addr [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] 2023-09-30T01:37:08.923 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:08.900+0000 7f9286d44700 1 --2- >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f92800a3380 0x7f928013e9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:37:08.923 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:08.900+0000 7f9286d44700 10 monclient(hunting): picked mon.noname-c con 0x7f92800a3380 addr [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] 2023-09-30T01:37:08.923 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:08.900+0000 7f9286d44700 10 monclient(hunting): start opening mon connection 2023-09-30T01:37:08.923 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:08.900+0000 7f9286d44700 1 -- --> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] -- mon_getmap magic: 0 v1 -- 0x7f9280123a60 con 0x7f92800a3380 2023-09-30T01:37:08.923 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:08.900+0000 7f9286d44700 10 monclient(hunting): start opening mon connection 2023-09-30T01:37:08.923 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:08.900+0000 7f9286d44700 1 -- --> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] -- mon_getmap magic: 0 v1 -- 0x7f9280123130 con 0x7f928009ffd0 2023-09-30T01:37:08.923 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:08.900+0000 7f9286d44700 10 monclient(hunting): start opening mon connection 2023-09-30T01:37:08.923 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:08.900+0000 7f9286d44700 1 -- --> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] -- mon_getmap magic: 0 v1 -- 0x7f92801236f0 con 0x7f92800a03a0 2023-09-30T01:37:08.923 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:08.900+0000 7f9286d44700 10 monclient(hunting): _renew_subs 2023-09-30T01:37:08.923 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:08.900+0000 7f9286d44700 10 monclient(hunting): authenticate will time out at 1100.942331s 2023-09-30T01:37:08.923 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:08.900+0000 7f9285541700 1 --2- >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7f92800a03a0 0x7f928013e070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:37:08.924 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:08.900+0000 7f9285d42700 1 --2- >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7f928009ffd0 0x7f9280141f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:37:08.924 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:08.900+0000 7f9286543700 1 --2- >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f92800a3380 0x7f928013e9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:37:08.924 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:08.900+0000 7f9285541700 1 --2- >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7f92800a03a0 0x7f928013e070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3303/0 says I am v2:172.21.15.88:42724/0 (socket says 172.21.15.88:42724) 2023-09-30T01:37:08.924 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:08.900+0000 7f9285541700 1 -- 172.21.15.88:0/3821470241 learned_addr learned my addr 172.21.15.88:0/3821470241 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-09-30T01:37:08.924 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:08.900+0000 7f9285541700 10 monclient(hunting): get_auth_request con 0x7f92800a03a0 auth_method 0 2023-09-30T01:37:08.924 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:08.900+0000 7f9285541700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:37:08.924 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:08.900+0000 7f9285541700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:37:08.924 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:08.900+0000 7f9285541700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:37:08.924 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:08.900+0000 7f9285d42700 10 monclient(hunting): get_auth_request con 0x7f928009ffd0 auth_method 0 2023-09-30T01:37:08.924 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:08.900+0000 7f9285d42700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:37:08.924 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:08.900+0000 7f9285d42700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:37:08.924 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:08.900+0000 7f9285d42700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:37:08.925 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:08.900+0000 7f9286543700 10 monclient(hunting): get_auth_request con 0x7f92800a3380 auth_method 0 2023-09-30T01:37:08.925 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:08.900+0000 7f9286543700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:37:08.925 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:08.900+0000 7f9286543700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:37:08.925 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:08.900+0000 7f9286543700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:37:08.925 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:08.900+0000 7f9285541700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:37:08.925 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:08.900+0000 7f9285541700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:37:08.925 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:08.900+0000 7f9285541700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:37:08.925 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:08.900+0000 7f9285d42700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:37:08.925 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:08.900+0000 7f9285d42700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:37:08.925 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:08.900+0000 7f9285d42700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:37:08.925 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:08.900+0000 7f9286543700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:37:08.926 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:08.900+0000 7f9286543700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:37:08.926 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:08.900+0000 7f9286543700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:37:08.926 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:08.900+0000 7f9285541700 10 monclient(hunting): handle_auth_done global_id 4668 payload 824 2023-09-30T01:37:08.926 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:08.900+0000 7f9285541700 1 -- 172.21.15.88:0/3821470241 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7f928009ffd0 msgr2=0x7f9280141f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:37:08.926 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:08.900+0000 7f9285541700 1 --2- 172.21.15.88:0/3821470241 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7f928009ffd0 0x7f9280141f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:37:08.926 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:08.900+0000 7f9285541700 1 -- 172.21.15.88:0/3821470241 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f92800a3380 msgr2=0x7f928013e9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:37:08.926 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:08.900+0000 7f9285541700 1 --2- 172.21.15.88:0/3821470241 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f92800a3380 0x7f928013e9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:37:08.926 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:08.900+0000 7f9285541700 10 monclient: _finish_hunting 0 2023-09-30T01:37:08.926 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:08.900+0000 7f9285541700 1 monclient: found mon.noname-d 2023-09-30T01:37:08.926 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:08.900+0000 7f9285541700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:37:08.926 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:08.900+0000 7f9285541700 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.88:3303/0 2023-09-30T01:37:08.927 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:08.900+0000 7f9285541700 1 -- 172.21.15.88:0/3821470241 --> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f927c0094f0 con 0x7f92800a03a0 2023-09-30T01:37:08.927 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:08.900+0000 7f9285541700 1 --2- 172.21.15.88:0/3821470241 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7f92800a03a0 0x7f928013e070 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f927400c7e0 tx=0x7f9274007da0 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:37:08.927 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:08.900+0000 7f9271ffb700 1 -- 172.21.15.88:0/3821470241 <== mon.6 v2:172.21.15.88:3303/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f9274011a10 con 0x7f92800a03a0 2023-09-30T01:37:08.927 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:08.900+0000 7f9271ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:37:08.927 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:08.900+0000 7f9271ffb700 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2023-09-30T01:37:08.927 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:08.900+0000 7f9271ffb700 10 monclient: dump: 2023-09-30T01:37:08.928 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:37:08.928 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:37:08.928 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:37:08.928 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:37:08.928 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:37:08.928 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:37:08.928 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:37:08.928 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:37:08.928 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:37:08.928 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:37:08.928 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:37:08.929 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:37:08.929 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:37:08.929 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:37:08.929 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:37:08.929 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:37:08.929 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:08.900+0000 7f9271ffb700 10 monclient: _finish_auth 0 2023-09-30T01:37:08.929 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:08.900+0000 7f9271ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:37:08.929 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:08.900+0000 7f9286d44700 5 monclient: authenticate success, global_id 4668 2023-09-30T01:37:08.929 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:08.900+0000 7f9271ffb700 1 -- 172.21.15.88:0/3821470241 <== mon.6 v2:172.21.15.88:3303/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9274011bb0 con 0x7f92800a03a0 2023-09-30T01:37:08.930 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:08.900+0000 7f9286d44700 10 monclient: _renew_subs 2023-09-30T01:37:08.930 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:08.900+0000 7f9286d44700 10 monclient: _send_mon_message to mon.d at v2:172.21.15.88:3303/0 2023-09-30T01:37:08.930 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:08.900+0000 7f9286d44700 1 -- 172.21.15.88:0/3821470241 --> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9280123410 con 0x7f92800a03a0 2023-09-30T01:37:08.930 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:08.900+0000 7f9271ffb700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:37:08.930 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:08.900+0000 7f9271ffb700 1 -- 172.21.15.88:0/3821470241 <== mon.6 v2:172.21.15.88:3303/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f9274011d20 con 0x7f92800a03a0 2023-09-30T01:37:08.930 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:08.900+0000 7f9271ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:37:08.930 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:08.900+0000 7f9271ffb700 10 monclient: got monmap 1 from mon.d (according to old e1) 2023-09-30T01:37:08.930 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:08.900+0000 7f9273fff700 4 set_mon_vals no callback set 2023-09-30T01:37:08.930 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:08.900+0000 7f9271ffb700 10 monclient: dump: 2023-09-30T01:37:08.930 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:37:08.931 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:37:08.931 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:37:08.931 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:37:08.931 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:37:08.931 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:37:08.931 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:37:08.931 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:37:08.931 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:37:08.931 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:37:08.931 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:37:08.931 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:37:08.931 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:37:08.932 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:37:08.932 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:37:08.932 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:37:08.932 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:08.900+0000 7f9286d44700 10 monclient: _renew_subs 2023-09-30T01:37:08.932 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:08.900+0000 7f9286d44700 10 monclient: _send_mon_message to mon.d at v2:172.21.15.88:3303/0 2023-09-30T01:37:08.932 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:08.900+0000 7f9286d44700 1 -- 172.21.15.88:0/3821470241 --> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f928013e5b0 con 0x7f92800a03a0 2023-09-30T01:37:08.932 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:08.904+0000 7f9271ffb700 1 -- 172.21.15.88:0/3821470241 <== mon.6 v2:172.21.15.88:3303/0 4 ==== mgrmap(e 5) v1 ==== 40144+0+0 (secure 0 0 0) 0x7f9274011590 con 0x7f92800a03a0 2023-09-30T01:37:08.932 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:08.904+0000 7f9286d44700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:37:08.932 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:08.904+0000 7f9286d44700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:37:08.932 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:08.904+0000 7f9286d44700 10 monclient: _send_mon_message to mon.d at v2:172.21.15.88:3303/0 2023-09-30T01:37:08.932 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:08.904+0000 7f9286d44700 1 -- 172.21.15.88:0/3821470241 --> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9280120f80 con 0x7f92800a03a0 2023-09-30T01:37:08.932 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:08.904+0000 7f9271ffb700 1 --2- 172.21.15.88:0/3821470241 >> v2:172.21.15.107:6812/29926 conn(0x7f9264033360 0x7f9264035810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:37:08.933 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:08.904+0000 7f9271ffb700 1 -- 172.21.15.88:0/3821470241 <== mon.6 v2:172.21.15.88:3303/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 3502+0+0 (secure 0 0 0) 0x7f9274042340 con 0x7f92800a03a0 2023-09-30T01:37:08.933 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:08.904+0000 7f9285d42700 1 --2- 172.21.15.88:0/3821470241 >> v2:172.21.15.107:6812/29926 conn(0x7f9264033360 0x7f9264035810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:37:08.933 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:08.904+0000 7f9285d42700 10 monclient: get_auth_request con 0x7f9264033360 auth_method 0 2023-09-30T01:37:08.933 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:08.904+0000 7f9285d42700 1 --2- 172.21.15.88:0/3821470241 >> v2:172.21.15.107:6812/29926 conn(0x7f9264033360 0x7f9264035810 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f927c0099b0 tx=0x7f927c017000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:37:08.933 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:08.908+0000 7f9271ffb700 1 -- 172.21.15.88:0/3821470241 <== mon.6 v2:172.21.15.88:3303/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f9274018c40 con 0x7f92800a03a0 2023-09-30T01:37:08.933 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:08.908+0000 7f9271ffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:37:08.934 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:08.908+0000 7f9271ffb700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:37:09.114 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:09.108+0000 7f9286d44700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T01:37:09.114 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:09.108+0000 7f9286d44700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:37:09.114 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:09.108+0000 7f9286d44700 10 monclient: _send_mon_message to mon.d at v2:172.21.15.88:3303/0 2023-09-30T01:37:09.114 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:09.108+0000 7f9286d44700 1 -- 172.21.15.88:0/3821470241 --> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f928013e7d0 con 0x7f92800a03a0 2023-09-30T01:37:09.115 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:09.108+0000 7f9271ffb700 1 -- 172.21.15.88:0/3821470241 <== mon.6 v2:172.21.15.88:3303/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f928013e7d0 con 0x7f92800a03a0 2023-09-30T01:37:09.115 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:09.108+0000 7f9271ffb700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:37:09.115 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:09.108+0000 7f9271ffb700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:37:09.115 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-09-30T01:37:09.115 INFO:teuthology.orchestra.run.smithi088.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:37:09.118 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:09.112+0000 7f9286d44700 1 -- 172.21.15.88:0/3821470241 >> v2:172.21.15.107:6812/29926 conn(0x7f9264033360 msgr2=0x7f9264035810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:37:09.118 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:09.112+0000 7f9286d44700 1 --2- 172.21.15.88:0/3821470241 >> v2:172.21.15.107:6812/29926 conn(0x7f9264033360 0x7f9264035810 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f927c0099b0 tx=0x7f927c017000 comp rx=0 tx=0).stop 2023-09-30T01:37:09.118 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:09.112+0000 7f9286d44700 10 monclient: shutdown 2023-09-30T01:37:09.118 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:09.112+0000 7f9286d44700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:37:09.118 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:09.112+0000 7f9286d44700 1 -- 172.21.15.88:0/3821470241 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7f92800a03a0 msgr2=0x7f928013e070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:37:09.118 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:09.112+0000 7f9286d44700 1 --2- 172.21.15.88:0/3821470241 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7f92800a03a0 0x7f928013e070 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f927400c7e0 tx=0x7f9274007da0 comp rx=0 tx=0).stop 2023-09-30T01:37:09.118 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:09.112+0000 7f9286d44700 1 -- 172.21.15.88:0/3821470241 shutdown_connections 2023-09-30T01:37:09.119 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:09.112+0000 7f9286d44700 1 --2- 172.21.15.88:0/3821470241 >> v2:172.21.15.107:6812/29926 conn(0x7f9264033360 0x7f9264035810 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:37:09.119 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:09.112+0000 7f9286d44700 1 --2- 172.21.15.88:0/3821470241 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f92800a3380 0x7f928013e9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:37:09.119 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:09.112+0000 7f9286d44700 1 --2- 172.21.15.88:0/3821470241 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7f92800a03a0 0x7f928013e070 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:37:09.119 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:09.112+0000 7f9286d44700 1 --2- 172.21.15.88:0/3821470241 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7f928009ffd0 0x7f9280141f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:37:09.119 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:09.112+0000 7f9286d44700 1 -- 172.21.15.88:0/3821470241 >> 172.21.15.88:0/3821470241 conn(0x7f928006e5e0 msgr2=0x7f92800a2820 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:37:09.119 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:09.112+0000 7f9286d44700 1 -- 172.21.15.88:0/3821470241 shutdown_connections 2023-09-30T01:37:09.119 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:09.112+0000 7f9286d44700 1 -- 172.21.15.88:0/3821470241 wait complete. 2023-09-30T01:37:12.129 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:37:12.237 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:12.232+0000 7f8303e28700 10 monclient: get_monmap_and_config 2023-09-30T01:37:12.237 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:12.232+0000 7f8303e28700 10 monclient: build_initial_monmap 2023-09-30T01:37:12.237 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:12.232+0000 7f8303e28700 10 monclient: monmap: 2023-09-30T01:37:12.238 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2023-09-30T01:37:12.238 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:37:12.238 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:37:12.237420+0000 2023-09-30T01:37:12.238 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:37:12.237420+0000 2023-09-30T01:37:12.238 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2023-09-30T01:37:12.238 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:37:12.238 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a-legacy 2023-09-30T01:37:12.238 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.107:6789/0 mon.noname-f-legacy 2023-09-30T01:37:12.238 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.noname-b 2023-09-30T01:37:12.238 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.noname-g 2023-09-30T01:37:12.238 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.noname-c 2023-09-30T01:37:12.239 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.noname-h 2023-09-30T01:37:12.239 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.noname-d 2023-09-30T01:37:12.239 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.noname-i 2023-09-30T01:37:12.239 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.noname-e 2023-09-30T01:37:12.239 INFO:teuthology.orchestra.run.smithi088.stderr:9: v2:172.21.15.88:3300/0 mon.noname-a 2023-09-30T01:37:12.239 INFO:teuthology.orchestra.run.smithi088.stderr:10: v2:172.21.15.107:3300/0 mon.noname-f 2023-09-30T01:37:12.239 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:37:12.248 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:12.232+0000 7f8303e28700 1 Processor -- start 2023-09-30T01:37:12.248 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:12.232+0000 7f8303e28700 1 -- start start 2023-09-30T01:37:12.248 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:12.232+0000 7f8303e28700 10 monclient: init 2023-09-30T01:37:12.248 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:12.232+0000 7f8303e28700 10 monclient: _reopen_session rank -1 2023-09-30T01:37:12.248 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:12.232+0000 7f8303e28700 10 monclient: _add_conns ranks=[6,8,5,1,4,0,2,3,10,7,9] 2023-09-30T01:37:12.248 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:12.232+0000 7f8303e28700 1 --2- >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7f82fc0a35f0 0x7f82fc133330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:37:12.248 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:12.232+0000 7f8303e28700 10 monclient(hunting): picked mon.noname-d con 0x7f82fc0a35f0 addr [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] 2023-09-30T01:37:12.249 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:12.232+0000 7f8303e28700 1 --2- >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f82fc133870 0x7f82fc133c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:37:12.249 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:12.232+0000 7f8303e28700 10 monclient(hunting): picked mon.noname-e con 0x7f82fc133870 addr [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] 2023-09-30T01:37:12.249 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:12.232+0000 7f8303e28700 1 --2- >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7f82fc0a46f0 0x7f82fc0a4b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:37:12.249 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:12.232+0000 7f8303e28700 10 monclient(hunting): picked mon.noname-h con 0x7f82fc0a46f0 addr [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] 2023-09-30T01:37:12.249 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:12.232+0000 7f8303e28700 10 monclient(hunting): start opening mon connection 2023-09-30T01:37:12.249 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:12.232+0000 7f8303e28700 1 -- --> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] -- mon_getmap magic: 0 v1 -- 0x7f82fc097ad0 con 0x7f82fc0a46f0 2023-09-30T01:37:12.249 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:12.232+0000 7f8303e28700 10 monclient(hunting): start opening mon connection 2023-09-30T01:37:12.249 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:12.232+0000 7f8303e28700 1 -- --> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] -- mon_getmap magic: 0 v1 -- 0x7f82fc1341c0 con 0x7f82fc0a35f0 2023-09-30T01:37:12.249 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:12.232+0000 7f8303e28700 10 monclient(hunting): start opening mon connection 2023-09-30T01:37:12.249 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:12.232+0000 7f8303e28700 1 -- --> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] -- mon_getmap magic: 0 v1 -- 0x7f82fc0a0940 con 0x7f82fc133870 2023-09-30T01:37:12.249 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:12.232+0000 7f8303e28700 10 monclient(hunting): _renew_subs 2023-09-30T01:37:12.249 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:12.232+0000 7f8303e28700 10 monclient(hunting): authenticate will time out at 1104.274927s 2023-09-30T01:37:12.250 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:12.232+0000 7f8301bc4700 1 --2- >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7f82fc0a35f0 0x7f82fc133330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:37:12.250 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:12.232+0000 7f83013c3700 1 --2- >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f82fc133870 0x7f82fc133c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:37:12.250 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:12.232+0000 7f8301bc4700 1 --2- >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7f82fc0a35f0 0x7f82fc133330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3303/0 says I am v2:172.21.15.88:33062/0 (socket says 172.21.15.88:33062) 2023-09-30T01:37:12.250 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:12.232+0000 7f8301bc4700 1 -- 172.21.15.88:0/719699252 learned_addr learned my addr 172.21.15.88:0/719699252 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-09-30T01:37:12.250 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:12.232+0000 7f8301bc4700 10 monclient(hunting): get_auth_request con 0x7f82fc0a35f0 auth_method 0 2023-09-30T01:37:12.250 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:12.232+0000 7f8301bc4700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:37:12.250 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:12.232+0000 7f8301bc4700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:37:12.250 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:12.232+0000 7f8301bc4700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:37:12.250 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:12.232+0000 7f83013c3700 10 monclient(hunting): get_auth_request con 0x7f82fc133870 auth_method 0 2023-09-30T01:37:12.250 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:12.232+0000 7f83013c3700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:37:12.250 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:12.232+0000 7f83013c3700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:37:12.251 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:12.232+0000 7f83013c3700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:37:12.251 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:12.232+0000 7f83023c5700 1 --2- 172.21.15.88:0/719699252 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7f82fc0a46f0 0x7f82fc0a4b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:37:12.251 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:12.232+0000 7f83013c3700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:37:12.251 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:12.232+0000 7f83013c3700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:37:12.251 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:12.232+0000 7f83013c3700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:37:12.251 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:12.232+0000 7f8301bc4700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:37:12.251 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:12.232+0000 7f8301bc4700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:37:12.251 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:12.232+0000 7f8301bc4700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:37:12.251 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:12.232+0000 7f83023c5700 10 monclient(hunting): get_auth_request con 0x7f82fc0a46f0 auth_method 0 2023-09-30T01:37:12.251 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:12.232+0000 7f83023c5700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:37:12.251 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:12.232+0000 7f83023c5700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:37:12.251 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:12.232+0000 7f83023c5700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:37:12.251 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:12.232+0000 7f83013c3700 10 monclient(hunting): handle_auth_done global_id 4769 payload 274 2023-09-30T01:37:12.252 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:12.232+0000 7f83013c3700 1 -- 172.21.15.88:0/719699252 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7f82fc0a35f0 msgr2=0x7f82fc133330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:37:12.252 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:12.232+0000 7f83013c3700 1 --2- 172.21.15.88:0/719699252 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7f82fc0a35f0 0x7f82fc133330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:37:12.252 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:12.232+0000 7f83013c3700 1 -- 172.21.15.88:0/719699252 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7f82fc0a46f0 msgr2=0x7f82fc0a4b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:37:12.252 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:12.232+0000 7f83013c3700 1 --2- 172.21.15.88:0/719699252 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7f82fc0a46f0 0x7f82fc0a4b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:37:12.252 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:12.232+0000 7f83013c3700 10 monclient: _finish_hunting 0 2023-09-30T01:37:12.252 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:12.232+0000 7f83013c3700 1 monclient: found mon.noname-e 2023-09-30T01:37:12.252 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:12.232+0000 7f83013c3700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:37:12.252 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:12.232+0000 7f83013c3700 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.88:3304/0 2023-09-30T01:37:12.252 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:12.232+0000 7f83013c3700 1 -- 172.21.15.88:0/719699252 --> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82fc05ecb0 con 0x7f82fc133870 2023-09-30T01:37:12.252 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:12.232+0000 7f83023c5700 1 --2- 172.21.15.88:0/719699252 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7f82fc0a46f0 0x7f82fc0a4b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-30T01:37:12.252 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:12.236+0000 7f83013c3700 1 --2- 172.21.15.88:0/719699252 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f82fc133870 0x7f82fc133c80 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f82ec009910 tx=0x7f82ec0091a0 comp rx=0 tx=0).ready entity=mon.8 client_cookie=dfb9c9bcbdf37d9c server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:37:12.252 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:12.236+0000 7f8300bc2700 1 -- 172.21.15.88:0/719699252 <== mon.8 v2:172.21.15.88:3304/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f82ec00bb00 con 0x7f82fc133870 2023-09-30T01:37:12.253 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:12.236+0000 7f8300bc2700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:37:12.253 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:12.236+0000 7f8300bc2700 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2023-09-30T01:37:12.253 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:12.236+0000 7f8300bc2700 10 monclient: dump: 2023-09-30T01:37:12.253 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:37:12.253 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:37:12.253 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:37:12.253 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:37:12.253 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:37:12.253 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:37:12.253 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:37:12.253 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:37:12.253 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:37:12.254 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:37:12.254 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:37:12.254 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:37:12.254 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:37:12.254 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:37:12.254 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:37:12.254 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:37:12.254 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:12.236+0000 7f8300bc2700 10 monclient: _finish_auth 0 2023-09-30T01:37:12.254 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:12.236+0000 7f8300bc2700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:37:12.254 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:12.236+0000 7f8303e28700 5 monclient: authenticate success, global_id 4769 2023-09-30T01:37:12.254 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:12.236+0000 7f8303e28700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:37:12.254 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:12.236+0000 7f8300bc2700 1 -- 172.21.15.88:0/719699252 <== mon.8 v2:172.21.15.88:3304/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82ec00bc80 con 0x7f82fc133870 2023-09-30T01:37:12.255 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:12.236+0000 7f8300bc2700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:37:12.255 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:12.236+0000 7f8303e28700 10 monclient: get_monmap_and_config success 2023-09-30T01:37:12.255 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:12.236+0000 7f8303e28700 4 set_mon_vals no callback set 2023-09-30T01:37:12.255 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:12.236+0000 7f8300bc2700 1 -- 172.21.15.88:0/719699252 <== mon.8 v2:172.21.15.88:3304/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f82ec00bdf0 con 0x7f82fc133870 2023-09-30T01:37:12.255 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:12.236+0000 7f8300bc2700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:37:12.255 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:12.236+0000 7f8300bc2700 10 monclient: got monmap 1 from mon.e (according to old e1) 2023-09-30T01:37:12.255 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:12.236+0000 7f8300bc2700 10 monclient: dump: 2023-09-30T01:37:12.255 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:37:12.255 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:37:12.255 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:37:12.255 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:37:12.255 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:37:12.256 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:37:12.256 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:37:12.256 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:37:12.256 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:37:12.256 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:37:12.256 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:37:12.256 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:37:12.256 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:37:12.256 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:37:12.256 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:37:12.256 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:37:12.256 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:12.236+0000 7f8303e28700 10 monclient: shutdown 2023-09-30T01:37:12.257 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:12.236+0000 7f8303e28700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:37:12.257 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:12.236+0000 7f8303e28700 1 -- 172.21.15.88:0/719699252 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f82fc133870 msgr2=0x7f82fc133c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:37:12.257 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:12.236+0000 7f8303e28700 1 --2- 172.21.15.88:0/719699252 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f82fc133870 0x7f82fc133c80 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f82ec009910 tx=0x7f82ec0091a0 comp rx=0 tx=0).stop 2023-09-30T01:37:12.257 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:12.236+0000 7f8303e28700 1 -- 172.21.15.88:0/719699252 shutdown_connections 2023-09-30T01:37:12.257 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:12.236+0000 7f8303e28700 1 --2- 172.21.15.88:0/719699252 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f82fc133870 0x7f82fc133c80 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:37:12.257 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:12.236+0000 7f8303e28700 1 --2- 172.21.15.88:0/719699252 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7f82fc0a46f0 0x7f82fc0a4b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:37:12.257 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:12.236+0000 7f8303e28700 1 --2- 172.21.15.88:0/719699252 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7f82fc0a35f0 0x7f82fc133330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:37:12.257 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:12.236+0000 7f8303e28700 1 -- 172.21.15.88:0/719699252 >> 172.21.15.88:0/719699252 conn(0x7f82fc06e5e0 msgr2=0x7f82fc06e9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:37:12.257 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:12.236+0000 7f8303e28700 1 -- 172.21.15.88:0/719699252 shutdown_connections 2023-09-30T01:37:12.257 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:12.236+0000 7f8303e28700 1 -- 172.21.15.88:0/719699252 wait complete. 2023-09-30T01:37:12.257 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:12.236+0000 7f8303e28700 10 monclient: build_initial_monmap 2023-09-30T01:37:12.257 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:12.236+0000 7f8303e28700 10 monclient: monmap: 2023-09-30T01:37:12.258 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2023-09-30T01:37:12.258 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:37:12.258 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 0.000000 2023-09-30T01:37:12.258 INFO:teuthology.orchestra.run.smithi088.stderr:created 0.000000 2023-09-30T01:37:12.258 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2023-09-30T01:37:12.258 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 1 2023-09-30T01:37:12.258 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.noname-a 2023-09-30T01:37:12.258 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.noname-f 2023-09-30T01:37:12.258 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.noname-b 2023-09-30T01:37:12.258 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.noname-g 2023-09-30T01:37:12.258 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.noname-c 2023-09-30T01:37:12.258 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.noname-h 2023-09-30T01:37:12.259 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.noname-d 2023-09-30T01:37:12.259 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.noname-i 2023-09-30T01:37:12.259 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.noname-e 2023-09-30T01:37:12.259 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:37:12.259 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:12.236+0000 7f8303e28700 1 Processor -- start 2023-09-30T01:37:12.259 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:12.236+0000 7f8303e28700 1 -- start start 2023-09-30T01:37:12.259 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:12.236+0000 7f8303e28700 10 monclient: init 2023-09-30T01:37:12.259 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:12.240+0000 7f8303e28700 10 monclient: _reopen_session rank -1 2023-09-30T01:37:12.259 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:12.240+0000 7f8303e28700 10 monclient: _add_conns ranks=[5,1,7,2,8,4,6,0,3] 2023-09-30T01:37:12.259 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:12.240+0000 7f8303e28700 1 --2- >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7f82fc0a35f0 0x7f82fc130470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:37:12.259 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:12.240+0000 7f8303e28700 10 monclient(hunting): picked mon.noname-h con 0x7f82fc0a35f0 addr [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] 2023-09-30T01:37:12.259 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:12.240+0000 7f8303e28700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f82fc0a46f0 0x7f82fc09d160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:37:12.260 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:12.240+0000 7f8303e28700 10 monclient(hunting): picked mon.noname-f con 0x7f82fc0a46f0 addr [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] 2023-09-30T01:37:12.260 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:12.240+0000 7f8303e28700 1 --2- >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7f82fc09f9d0 0x7f82fc09d6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:37:12.260 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:12.240+0000 7f8303e28700 10 monclient(hunting): picked mon.noname-i con 0x7f82fc09f9d0 addr [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] 2023-09-30T01:37:12.260 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:12.240+0000 7f8303e28700 10 monclient(hunting): start opening mon connection 2023-09-30T01:37:12.260 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:12.240+0000 7f8303e28700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82fc097d90 con 0x7f82fc0a46f0 2023-09-30T01:37:12.260 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:12.240+0000 7f8303e28700 10 monclient(hunting): start opening mon connection 2023-09-30T01:37:12.260 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:12.240+0000 7f8303e28700 1 -- --> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] -- mon_getmap magic: 0 v1 -- 0x7f82fc0a0b70 con 0x7f82fc0a35f0 2023-09-30T01:37:12.260 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:12.240+0000 7f8303e28700 10 monclient(hunting): start opening mon connection 2023-09-30T01:37:12.260 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:12.240+0000 7f8303e28700 1 -- --> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] -- mon_getmap magic: 0 v1 -- 0x7f82fc11d590 con 0x7f82fc09f9d0 2023-09-30T01:37:12.260 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:12.240+0000 7f8303e28700 10 monclient(hunting): _renew_subs 2023-09-30T01:37:12.260 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:12.240+0000 7f8303e28700 10 monclient(hunting): authenticate will time out at 1104.280377s 2023-09-30T01:37:12.260 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:12.240+0000 7f83013c3700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f82fc0a46f0 0x7f82fc09d160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:37:12.261 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:12.240+0000 7f83023c5700 1 --2- >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7f82fc09f9d0 0x7f82fc09d6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:37:12.261 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:12.240+0000 7f8301bc4700 1 --2- >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7f82fc0a35f0 0x7f82fc130470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:37:12.261 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:12.240+0000 7f83013c3700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f82fc0a46f0 0x7f82fc09d160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.88:40706/0 (socket says 172.21.15.88:40706) 2023-09-30T01:37:12.261 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:12.240+0000 7f83013c3700 1 -- 172.21.15.88:0/2938005079 learned_addr learned my addr 172.21.15.88:0/2938005079 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-09-30T01:37:12.261 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:12.240+0000 7f83013c3700 10 monclient(hunting): get_auth_request con 0x7f82fc0a46f0 auth_method 0 2023-09-30T01:37:12.261 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:12.240+0000 7f83013c3700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:37:12.261 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:12.240+0000 7f83013c3700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:37:12.261 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:12.240+0000 7f83013c3700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:37:12.261 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:12.240+0000 7f8301bc4700 10 monclient(hunting): get_auth_request con 0x7f82fc0a35f0 auth_method 0 2023-09-30T01:37:12.261 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:12.240+0000 7f8301bc4700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:37:12.261 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:12.240+0000 7f8301bc4700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:37:12.261 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:12.240+0000 7f8301bc4700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:37:12.262 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:12.240+0000 7f83023c5700 10 monclient(hunting): get_auth_request con 0x7f82fc09f9d0 auth_method 0 2023-09-30T01:37:12.262 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:12.240+0000 7f83023c5700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:37:12.262 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:12.240+0000 7f83023c5700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:37:12.262 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:12.240+0000 7f83023c5700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:37:12.262 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:12.240+0000 7f83013c3700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:37:12.262 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:12.240+0000 7f83013c3700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:37:12.262 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:12.240+0000 7f83013c3700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:37:12.262 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:12.240+0000 7f83023c5700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:37:12.262 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:12.240+0000 7f83023c5700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:37:12.262 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:12.240+0000 7f83023c5700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:37:12.262 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:12.240+0000 7f8301bc4700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:37:12.262 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:12.240+0000 7f8301bc4700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:37:12.263 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:12.240+0000 7f8301bc4700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:37:12.263 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:12.240+0000 7f83013c3700 10 monclient(hunting): handle_auth_done global_id 5095 payload 824 2023-09-30T01:37:12.263 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:12.240+0000 7f83013c3700 1 -- 172.21.15.88:0/2938005079 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7f82fc09f9d0 msgr2=0x7f82fc09d6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:37:12.263 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:12.240+0000 7f83013c3700 1 --2- 172.21.15.88:0/2938005079 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7f82fc09f9d0 0x7f82fc09d6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:37:12.263 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:12.240+0000 7f83013c3700 1 -- 172.21.15.88:0/2938005079 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7f82fc0a35f0 msgr2=0x7f82fc130470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:37:12.263 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:12.240+0000 7f83013c3700 1 --2- 172.21.15.88:0/2938005079 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7f82fc0a35f0 0x7f82fc130470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:37:12.263 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:12.240+0000 7f83013c3700 10 monclient: _finish_hunting 0 2023-09-30T01:37:12.263 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:12.240+0000 7f83013c3700 1 monclient: found mon.noname-f 2023-09-30T01:37:12.263 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:12.240+0000 7f83013c3700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:37:12.263 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:12.240+0000 7f83013c3700 10 monclient: _send_mon_message to mon.noname-f at v2:172.21.15.107:3300/0 2023-09-30T01:37:12.263 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:12.240+0000 7f83013c3700 1 -- 172.21.15.88:0/2938005079 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82ec009420 con 0x7f82fc0a46f0 2023-09-30T01:37:12.263 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:12.240+0000 7f83013c3700 1 --2- 172.21.15.88:0/2938005079 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f82fc0a46f0 0x7f82fc09d160 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f82ec0093f0 tx=0x7f82ec017000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:37:12.264 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:12.240+0000 7f82f1ffb700 1 -- 172.21.15.88:0/2938005079 <== mon.1 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f82ec006ad0 con 0x7f82fc0a46f0 2023-09-30T01:37:12.264 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:12.240+0000 7f82f1ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:37:12.264 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:12.240+0000 7f82f1ffb700 10 monclient: got monmap 1 from mon.noname-f (according to old e1) 2023-09-30T01:37:12.264 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:12.240+0000 7f82f1ffb700 10 monclient: dump: 2023-09-30T01:37:12.264 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:37:12.264 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:37:12.264 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:37:12.264 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:37:12.264 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:37:12.264 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:37:12.264 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:37:12.264 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:37:12.265 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:37:12.265 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:37:12.265 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:37:12.265 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:37:12.265 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:37:12.265 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:37:12.265 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:37:12.265 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:37:12.265 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:12.240+0000 7f82f1ffb700 10 monclient: _finish_auth 0 2023-09-30T01:37:12.265 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:12.240+0000 7f82f1ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:37:12.265 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:12.240+0000 7f82f1ffb700 1 -- 172.21.15.88:0/2938005079 <== mon.1 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82ec01a450 con 0x7f82fc0a46f0 2023-09-30T01:37:12.265 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:12.240+0000 7f82f1ffb700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:37:12.266 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:12.240+0000 7f82f1ffb700 1 -- 172.21.15.88:0/2938005079 <== mon.1 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f82ec01aa40 con 0x7f82fc0a46f0 2023-09-30T01:37:12.266 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:12.240+0000 7f82f1ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:37:12.266 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:12.240+0000 7f82f1ffb700 10 monclient: got monmap 1 from mon.f (according to old e1) 2023-09-30T01:37:12.266 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:12.240+0000 7f82f1ffb700 10 monclient: dump: 2023-09-30T01:37:12.266 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:37:12.266 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:37:12.266 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:37:12.266 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:37:12.266 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:37:12.266 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:37:12.266 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:37:12.266 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:37:12.266 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:37:12.267 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:37:12.267 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:37:12.267 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:37:12.267 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:37:12.267 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:37:12.267 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:37:12.267 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:37:12.267 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:12.240+0000 7f82f3fff700 4 set_mon_vals no callback set 2023-09-30T01:37:12.267 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:12.240+0000 7f8303e28700 5 monclient: authenticate success, global_id 5095 2023-09-30T01:37:12.267 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:12.240+0000 7f8303e28700 10 monclient: _renew_subs 2023-09-30T01:37:12.267 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:12.240+0000 7f8303e28700 10 monclient: _send_mon_message to mon.f at v2:172.21.15.107:3300/0 2023-09-30T01:37:12.267 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:12.240+0000 7f8303e28700 1 -- 172.21.15.88:0/2938005079 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82fc11d870 con 0x7f82fc0a46f0 2023-09-30T01:37:12.268 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:12.240+0000 7f8303e28700 10 monclient: _renew_subs 2023-09-30T01:37:12.268 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:12.240+0000 7f8303e28700 10 monclient: _send_mon_message to mon.f at v2:172.21.15.107:3300/0 2023-09-30T01:37:12.268 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:12.240+0000 7f8303e28700 1 -- 172.21.15.88:0/2938005079 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f82fc1433b0 con 0x7f82fc0a46f0 2023-09-30T01:37:12.268 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:12.240+0000 7f8303e28700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:37:12.268 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:12.240+0000 7f8303e28700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:37:12.268 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:12.240+0000 7f8303e28700 10 monclient: _send_mon_message to mon.f at v2:172.21.15.107:3300/0 2023-09-30T01:37:12.268 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:12.240+0000 7f8303e28700 1 -- 172.21.15.88:0/2938005079 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f82fc11da00 con 0x7f82fc0a46f0 2023-09-30T01:37:12.268 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:12.240+0000 7f82f1ffb700 1 -- 172.21.15.88:0/2938005079 <== mon.1 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 40144+0+0 (secure 0 0 0) 0x7f82ec01aba0 con 0x7f82fc0a46f0 2023-09-30T01:37:12.268 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:12.244+0000 7f82f1ffb700 1 --2- 172.21.15.88:0/2938005079 >> v2:172.21.15.107:6812/29926 conn(0x7f82e0033290 0x7f82e0035740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:37:12.268 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:12.244+0000 7f82f1ffb700 1 -- 172.21.15.88:0/2938005079 <== mon.1 v2:172.21.15.107:3300/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 3502+0+0 (secure 0 0 0) 0x7f82ec044d10 con 0x7f82fc0a46f0 2023-09-30T01:37:12.268 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:12.244+0000 7f8301bc4700 1 --2- 172.21.15.88:0/2938005079 >> v2:172.21.15.107:6812/29926 conn(0x7f82e0033290 0x7f82e0035740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:37:12.268 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:12.244+0000 7f8301bc4700 10 monclient: get_auth_request con 0x7f82e0033290 auth_method 0 2023-09-30T01:37:12.269 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:12.244+0000 7f8301bc4700 1 --2- 172.21.15.88:0/2938005079 >> v2:172.21.15.107:6812/29926 conn(0x7f82e0033290 0x7f82e0035740 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f82f8006d40 tx=0x7f82f8009250 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:37:12.269 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:12.252+0000 7f82f1ffb700 1 -- 172.21.15.88:0/2938005079 <== mon.1 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f82ec006cb0 con 0x7f82fc0a46f0 2023-09-30T01:37:12.269 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:12.252+0000 7f82f1ffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:37:12.269 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:12.252+0000 7f82f1ffb700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:37:12.458 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:12.452+0000 7f8303e28700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T01:37:12.459 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:12.452+0000 7f8303e28700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:37:12.459 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:12.452+0000 7f8303e28700 10 monclient: _send_mon_message to mon.f at v2:172.21.15.107:3300/0 2023-09-30T01:37:12.459 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:12.452+0000 7f8303e28700 1 -- 172.21.15.88:0/2938005079 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f82fc0710d0 con 0x7f82fc0a46f0 2023-09-30T01:37:12.459 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:12.452+0000 7f82f1ffb700 1 -- 172.21.15.88:0/2938005079 <== mon.1 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f82ec015070 con 0x7f82fc0a46f0 2023-09-30T01:37:12.459 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:12.452+0000 7f82f1ffb700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:37:12.460 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:12.452+0000 7f82f1ffb700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:37:12.460 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-09-30T01:37:12.460 INFO:teuthology.orchestra.run.smithi088.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:37:12.462 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:12.456+0000 7f8303e28700 1 -- 172.21.15.88:0/2938005079 >> v2:172.21.15.107:6812/29926 conn(0x7f82e0033290 msgr2=0x7f82e0035740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:37:12.462 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:12.456+0000 7f8303e28700 1 --2- 172.21.15.88:0/2938005079 >> v2:172.21.15.107:6812/29926 conn(0x7f82e0033290 0x7f82e0035740 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f82f8006d40 tx=0x7f82f8009250 comp rx=0 tx=0).stop 2023-09-30T01:37:12.462 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:12.456+0000 7f8303e28700 10 monclient: shutdown 2023-09-30T01:37:12.462 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:12.456+0000 7f8303e28700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:37:12.462 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:12.456+0000 7f8303e28700 1 -- 172.21.15.88:0/2938005079 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f82fc0a46f0 msgr2=0x7f82fc09d160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:37:12.462 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:12.456+0000 7f8303e28700 1 --2- 172.21.15.88:0/2938005079 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f82fc0a46f0 0x7f82fc09d160 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f82ec0093f0 tx=0x7f82ec017000 comp rx=0 tx=0).stop 2023-09-30T01:37:12.462 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:12.456+0000 7f8303e28700 1 -- 172.21.15.88:0/2938005079 shutdown_connections 2023-09-30T01:37:12.463 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:12.456+0000 7f8303e28700 1 --2- 172.21.15.88:0/2938005079 >> v2:172.21.15.107:6812/29926 conn(0x7f82e0033290 0x7f82e0035740 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:37:12.463 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:12.456+0000 7f8303e28700 1 --2- 172.21.15.88:0/2938005079 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7f82fc09f9d0 0x7f82fc09d6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:37:12.463 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:12.456+0000 7f8303e28700 1 --2- 172.21.15.88:0/2938005079 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f82fc0a46f0 0x7f82fc09d160 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:37:12.463 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:12.456+0000 7f8303e28700 1 --2- 172.21.15.88:0/2938005079 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7f82fc0a35f0 0x7f82fc130470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:37:12.463 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:12.456+0000 7f8303e28700 1 -- 172.21.15.88:0/2938005079 >> 172.21.15.88:0/2938005079 conn(0x7f82fc06e5e0 msgr2=0x7f82fc1317d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:37:12.463 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:12.456+0000 7f8303e28700 1 -- 172.21.15.88:0/2938005079 shutdown_connections 2023-09-30T01:37:12.463 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:12.456+0000 7f8303e28700 1 -- 172.21.15.88:0/2938005079 wait complete. 2023-09-30T01:37:15.473 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:37:15.580 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:15.576+0000 7fdfd17ba700 10 monclient: get_monmap_and_config 2023-09-30T01:37:15.580 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:15.576+0000 7fdfd17ba700 10 monclient: build_initial_monmap 2023-09-30T01:37:15.580 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:15.576+0000 7fdfd17ba700 10 monclient: monmap: 2023-09-30T01:37:15.580 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2023-09-30T01:37:15.580 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:37:15.580 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:37:15.580011+0000 2023-09-30T01:37:15.581 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:37:15.580011+0000 2023-09-30T01:37:15.581 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2023-09-30T01:37:15.581 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:37:15.581 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a-legacy 2023-09-30T01:37:15.581 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.107:6789/0 mon.noname-f-legacy 2023-09-30T01:37:15.581 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.noname-b 2023-09-30T01:37:15.581 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.noname-g 2023-09-30T01:37:15.581 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.noname-c 2023-09-30T01:37:15.581 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.noname-h 2023-09-30T01:37:15.581 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.noname-d 2023-09-30T01:37:15.581 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.noname-i 2023-09-30T01:37:15.581 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.noname-e 2023-09-30T01:37:15.582 INFO:teuthology.orchestra.run.smithi088.stderr:9: v2:172.21.15.88:3300/0 mon.noname-a 2023-09-30T01:37:15.582 INFO:teuthology.orchestra.run.smithi088.stderr:10: v2:172.21.15.107:3300/0 mon.noname-f 2023-09-30T01:37:15.582 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:37:15.591 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:15.576+0000 7fdfd17ba700 1 Processor -- start 2023-09-30T01:37:15.591 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:15.576+0000 7fdfd17ba700 1 -- start start 2023-09-30T01:37:15.591 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:15.576+0000 7fdfd17ba700 10 monclient: init 2023-09-30T01:37:15.591 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:15.576+0000 7fdfd17ba700 10 monclient: _reopen_session rank -1 2023-09-30T01:37:15.591 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:15.576+0000 7fdfd17ba700 10 monclient: _add_conns ranks=[9,6,8,4,1,3,0,2,5,7,10] 2023-09-30T01:37:15.591 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:15.576+0000 7fdfd17ba700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fdfcc0a3270 0x7fdfcc0a0160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:37:15.591 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:15.576+0000 7fdfd17ba700 10 monclient(hunting): picked mon.noname-a con 0x7fdfcc0a3270 addr v2:172.21.15.88:3300/0 2023-09-30T01:37:15.592 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:15.576+0000 7fdfd17ba700 1 --2- >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7fdfcc0a4b00 0x7fdfcc0a06a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:37:15.592 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:15.576+0000 7fdfd17ba700 10 monclient(hunting): picked mon.noname-d con 0x7fdfcc0a4b00 addr [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] 2023-09-30T01:37:15.592 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:15.576+0000 7fdfd17ba700 1 --2- >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7fdfcc1333c0 0x7fdfcc1337d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:37:15.592 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:15.576+0000 7fdfd17ba700 10 monclient(hunting): picked mon.noname-e con 0x7fdfcc1333c0 addr [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] 2023-09-30T01:37:15.592 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:15.576+0000 7fdfd17ba700 10 monclient(hunting): start opening mon connection 2023-09-30T01:37:15.592 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:15.576+0000 7fdfd17ba700 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdfcc0a4ed0 con 0x7fdfcc0a3270 2023-09-30T01:37:15.592 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:15.576+0000 7fdfd17ba700 10 monclient(hunting): start opening mon connection 2023-09-30T01:37:15.592 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:15.576+0000 7fdfd17ba700 1 -- --> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] -- mon_getmap magic: 0 v1 -- 0x7fdfcc0a6330 con 0x7fdfcc0a4b00 2023-09-30T01:37:15.592 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:15.576+0000 7fdfd17ba700 10 monclient(hunting): start opening mon connection 2023-09-30T01:37:15.592 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:15.576+0000 7fdfd17ba700 1 -- --> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] -- mon_getmap magic: 0 v1 -- 0x7fdfcc098610 con 0x7fdfcc1333c0 2023-09-30T01:37:15.592 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:15.576+0000 7fdfd17ba700 10 monclient(hunting): _renew_subs 2023-09-30T01:37:15.593 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:15.576+0000 7fdfd17ba700 10 monclient(hunting): authenticate will time out at 1107.617633s 2023-09-30T01:37:15.593 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:15.576+0000 7fdfca7fc700 1 --2- >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7fdfcc0a4b00 0x7fdfcc0a06a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:37:15.593 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:15.576+0000 7fdfcaffd700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fdfcc0a3270 0x7fdfcc0a0160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:37:15.593 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:15.576+0000 7fdfcaffd700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fdfcc0a3270 0x7fdfcc0a0160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:59720/0 (socket says 172.21.15.88:59720) 2023-09-30T01:37:15.593 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:15.576+0000 7fdfcb7fe700 1 --2- >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7fdfcc1333c0 0x7fdfcc1337d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:37:15.593 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:15.576+0000 7fdfca7fc700 1 --2- >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7fdfcc0a4b00 0x7fdfcc0a06a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3303/0 says I am v2:172.21.15.88:33072/0 (socket says 172.21.15.88:33072) 2023-09-30T01:37:15.593 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:15.576+0000 7fdfcaffd700 1 -- 172.21.15.88:0/3829961 learned_addr learned my addr 172.21.15.88:0/3829961 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-09-30T01:37:15.593 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:15.576+0000 7fdfcaffd700 10 monclient(hunting): get_auth_request con 0x7fdfcc0a3270 auth_method 0 2023-09-30T01:37:15.593 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:15.576+0000 7fdfcaffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:37:15.593 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:15.576+0000 7fdfcaffd700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:37:15.593 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:15.576+0000 7fdfcaffd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:37:15.593 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:15.576+0000 7fdfca7fc700 10 monclient(hunting): get_auth_request con 0x7fdfcc0a4b00 auth_method 0 2023-09-30T01:37:15.594 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:15.576+0000 7fdfca7fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:37:15.594 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:15.576+0000 7fdfca7fc700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:37:15.594 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:15.576+0000 7fdfca7fc700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:37:15.594 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:15.576+0000 7fdfcb7fe700 10 monclient(hunting): get_auth_request con 0x7fdfcc1333c0 auth_method 0 2023-09-30T01:37:15.594 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:15.576+0000 7fdfcb7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:37:15.594 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:15.576+0000 7fdfcb7fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:37:15.594 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:15.576+0000 7fdfcb7fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:37:15.594 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:15.576+0000 7fdfcaffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:37:15.594 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:15.576+0000 7fdfcaffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:37:15.594 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:15.576+0000 7fdfcaffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:37:15.594 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:15.576+0000 7fdfcb7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:37:15.594 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:15.576+0000 7fdfcb7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:37:15.595 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:15.576+0000 7fdfcb7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:37:15.595 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:15.576+0000 7fdfca7fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:37:15.595 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:15.576+0000 7fdfca7fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:37:15.595 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:15.576+0000 7fdfca7fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:37:15.595 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:15.576+0000 7fdfcb7fe700 10 monclient(hunting): handle_auth_done global_id 4778 payload 274 2023-09-30T01:37:15.595 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:15.576+0000 7fdfcb7fe700 1 -- 172.21.15.88:0/3829961 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7fdfcc0a4b00 msgr2=0x7fdfcc0a06a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:37:15.595 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:15.576+0000 7fdfcb7fe700 1 --2- 172.21.15.88:0/3829961 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7fdfcc0a4b00 0x7fdfcc0a06a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:37:15.595 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:15.576+0000 7fdfcb7fe700 1 -- 172.21.15.88:0/3829961 >> v2:172.21.15.88:3300/0 conn(0x7fdfcc0a3270 msgr2=0x7fdfcc0a0160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:37:15.595 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:15.576+0000 7fdfcb7fe700 1 --2- 172.21.15.88:0/3829961 >> v2:172.21.15.88:3300/0 conn(0x7fdfcc0a3270 0x7fdfcc0a0160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:37:15.595 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:15.576+0000 7fdfcb7fe700 10 monclient: _finish_hunting 0 2023-09-30T01:37:15.595 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:15.576+0000 7fdfcb7fe700 1 monclient: found mon.noname-e 2023-09-30T01:37:15.595 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:15.576+0000 7fdfcb7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:37:15.596 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:15.576+0000 7fdfcb7fe700 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.88:3304/0 2023-09-30T01:37:15.596 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:15.576+0000 7fdfcb7fe700 1 -- 172.21.15.88:0/3829961 --> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdfcc05ecb0 con 0x7fdfcc1333c0 2023-09-30T01:37:15.596 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:15.576+0000 7fdfcaffd700 1 --2- 172.21.15.88:0/3829961 >> v2:172.21.15.88:3300/0 conn(0x7fdfcc0a3270 0x7fdfcc0a0160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:37:15.596 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:15.576+0000 7fdfca7fc700 1 --2- 172.21.15.88:0/3829961 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7fdfcc0a4b00 0x7fdfcc0a06a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:37:15.596 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:15.576+0000 7fdfcb7fe700 1 --2- 172.21.15.88:0/3829961 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7fdfcc1333c0 0x7fdfcc1337d0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fdfc4009e60 tx=0x7fdfc4009780 comp rx=0 tx=0).ready entity=mon.8 client_cookie=904475103bb36626 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:37:15.596 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:15.576+0000 7fdfc9ffb700 1 -- 172.21.15.88:0/3829961 <== mon.8 v2:172.21.15.88:3304/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7fdfc400e070 con 0x7fdfcc1333c0 2023-09-30T01:37:15.596 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:15.576+0000 7fdfc9ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:37:15.596 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:15.576+0000 7fdfc9ffb700 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2023-09-30T01:37:15.596 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:15.576+0000 7fdfc9ffb700 10 monclient: dump: 2023-09-30T01:37:15.596 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:37:15.597 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:37:15.597 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:37:15.597 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:37:15.597 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:37:15.597 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:37:15.597 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:37:15.597 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:37:15.597 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:37:15.597 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:37:15.597 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:37:15.597 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:37:15.597 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:37:15.598 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:37:15.598 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:37:15.598 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:37:15.598 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:15.576+0000 7fdfc9ffb700 10 monclient: _finish_auth 0 2023-09-30T01:37:15.598 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:15.576+0000 7fdfc9ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:37:15.598 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:15.576+0000 7fdfd17ba700 5 monclient: authenticate success, global_id 4778 2023-09-30T01:37:15.598 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:15.576+0000 7fdfc9ffb700 1 -- 172.21.15.88:0/3829961 <== mon.8 v2:172.21.15.88:3304/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdfc4012430 con 0x7fdfcc1333c0 2023-09-30T01:37:15.598 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:15.576+0000 7fdfd17ba700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:37:15.598 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:15.576+0000 7fdfc9ffb700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:37:15.598 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:15.576+0000 7fdfd17ba700 10 monclient: get_monmap_and_config success 2023-09-30T01:37:15.598 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:15.576+0000 7fdfc9ffb700 1 -- 172.21.15.88:0/3829961 <== mon.8 v2:172.21.15.88:3304/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7fdfc4012a20 con 0x7fdfcc1333c0 2023-09-30T01:37:15.599 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:15.576+0000 7fdfc9ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:37:15.599 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:15.576+0000 7fdfd17ba700 4 set_mon_vals no callback set 2023-09-30T01:37:15.599 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:15.576+0000 7fdfc9ffb700 10 monclient: got monmap 1 from mon.e (according to old e1) 2023-09-30T01:37:15.599 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:15.576+0000 7fdfc9ffb700 10 monclient: dump: 2023-09-30T01:37:15.599 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:37:15.599 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:37:15.599 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:37:15.599 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:37:15.599 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:37:15.599 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:37:15.600 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:37:15.600 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:37:15.600 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:37:15.600 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:37:15.600 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:37:15.600 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:37:15.600 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:37:15.600 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:37:15.600 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:37:15.600 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:37:15.600 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:15.576+0000 7fdfd17ba700 10 monclient: shutdown 2023-09-30T01:37:15.600 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:15.576+0000 7fdfd17ba700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:37:15.601 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:15.576+0000 7fdfd17ba700 1 -- 172.21.15.88:0/3829961 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7fdfcc1333c0 msgr2=0x7fdfcc1337d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:37:15.601 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:15.576+0000 7fdfd17ba700 1 --2- 172.21.15.88:0/3829961 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7fdfcc1333c0 0x7fdfcc1337d0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fdfc4009e60 tx=0x7fdfc4009780 comp rx=0 tx=0).stop 2023-09-30T01:37:15.601 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:15.576+0000 7fdfd17ba700 1 -- 172.21.15.88:0/3829961 shutdown_connections 2023-09-30T01:37:15.601 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:15.576+0000 7fdfd17ba700 1 --2- 172.21.15.88:0/3829961 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7fdfcc1333c0 0x7fdfcc1337d0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:37:15.601 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:15.576+0000 7fdfd17ba700 1 --2- 172.21.15.88:0/3829961 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7fdfcc0a4b00 0x7fdfcc0a06a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:37:15.601 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:15.576+0000 7fdfd17ba700 1 --2- 172.21.15.88:0/3829961 >> v2:172.21.15.88:3300/0 conn(0x7fdfcc0a3270 0x7fdfcc0a0160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:37:15.601 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:15.576+0000 7fdfd17ba700 1 -- 172.21.15.88:0/3829961 >> 172.21.15.88:0/3829961 conn(0x7fdfcc0a67f0 msgr2=0x7fdfcc0a6bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:37:15.601 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:15.576+0000 7fdfd17ba700 1 -- 172.21.15.88:0/3829961 shutdown_connections 2023-09-30T01:37:15.601 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:15.576+0000 7fdfd17ba700 1 -- 172.21.15.88:0/3829961 wait complete. 2023-09-30T01:37:15.601 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:15.580+0000 7fdfd17ba700 10 monclient: build_initial_monmap 2023-09-30T01:37:15.601 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:15.580+0000 7fdfd17ba700 10 monclient: monmap: 2023-09-30T01:37:15.601 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2023-09-30T01:37:15.601 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:37:15.602 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 0.000000 2023-09-30T01:37:15.602 INFO:teuthology.orchestra.run.smithi088.stderr:created 0.000000 2023-09-30T01:37:15.602 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2023-09-30T01:37:15.602 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 1 2023-09-30T01:37:15.602 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.noname-a 2023-09-30T01:37:15.602 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.noname-f 2023-09-30T01:37:15.602 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.noname-b 2023-09-30T01:37:15.602 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.noname-g 2023-09-30T01:37:15.602 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.noname-c 2023-09-30T01:37:15.602 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.noname-h 2023-09-30T01:37:15.602 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.noname-d 2023-09-30T01:37:15.603 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.noname-i 2023-09-30T01:37:15.603 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.noname-e 2023-09-30T01:37:15.603 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:37:15.603 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:15.580+0000 7fdfd17ba700 1 Processor -- start 2023-09-30T01:37:15.603 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:15.580+0000 7fdfd17ba700 1 -- start start 2023-09-30T01:37:15.603 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:15.580+0000 7fdfd17ba700 10 monclient: init 2023-09-30T01:37:15.603 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:15.580+0000 7fdfd17ba700 10 monclient: _reopen_session rank -1 2023-09-30T01:37:15.603 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:15.580+0000 7fdfd17ba700 10 monclient: _add_conns ranks=[6,8,3,5,0,7,2,1,4] 2023-09-30T01:37:15.603 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:15.580+0000 7fdfd17ba700 1 --2- >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7fdfcc0a3270 0x7fdfcc0a0500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:37:15.603 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:15.580+0000 7fdfd17ba700 10 monclient(hunting): picked mon.noname-d con 0x7fdfcc0a3270 addr [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] 2023-09-30T01:37:15.603 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:15.580+0000 7fdfd17ba700 1 --2- >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7fdfcc0a4b00 0x7fdfcc11ce50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:37:15.603 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:15.580+0000 7fdfd17ba700 10 monclient(hunting): picked mon.noname-e con 0x7fdfcc0a4b00 addr [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] 2023-09-30T01:37:15.603 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:15.580+0000 7fdfd17ba700 1 --2- >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7fdfcc1333c0 0x7fdfcc11d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:37:15.604 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:15.580+0000 7fdfd17ba700 10 monclient(hunting): picked mon.noname-g con 0x7fdfcc1333c0 addr [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] 2023-09-30T01:37:15.604 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:15.580+0000 7fdfd17ba700 10 monclient(hunting): start opening mon connection 2023-09-30T01:37:15.604 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:15.580+0000 7fdfd17ba700 1 -- --> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdfcc11ed40 con 0x7fdfcc1333c0 2023-09-30T01:37:15.605 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:15.580+0000 7fdfd17ba700 10 monclient(hunting): start opening mon connection 2023-09-30T01:37:15.605 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:15.580+0000 7fdfd17ba700 1 -- --> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] -- mon_getmap magic: 0 v1 -- 0x7fdfcc097c70 con 0x7fdfcc0a3270 2023-09-30T01:37:15.605 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:15.580+0000 7fdfd17ba700 10 monclient(hunting): start opening mon connection 2023-09-30T01:37:15.605 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:15.580+0000 7fdfd17ba700 1 -- --> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] -- mon_getmap magic: 0 v1 -- 0x7fdfcc0a0e80 con 0x7fdfcc0a4b00 2023-09-30T01:37:15.605 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:15.580+0000 7fdfd17ba700 10 monclient(hunting): _renew_subs 2023-09-30T01:37:15.605 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:15.580+0000 7fdfd17ba700 10 monclient(hunting): authenticate will time out at 1107.621226s 2023-09-30T01:37:15.605 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:15.580+0000 7fdfcaffd700 1 --2- >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7fdfcc0a3270 0x7fdfcc0a0500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:37:15.605 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:15.580+0000 7fdfca7fc700 1 --2- >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7fdfcc0a4b00 0x7fdfcc11ce50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:37:15.605 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:15.580+0000 7fdfcaffd700 1 --2- >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7fdfcc0a3270 0x7fdfcc0a0500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3303/0 says I am v2:172.21.15.88:33074/0 (socket says 172.21.15.88:33074) 2023-09-30T01:37:15.606 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:15.580+0000 7fdfcaffd700 1 -- 172.21.15.88:0/295005856 learned_addr learned my addr 172.21.15.88:0/295005856 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-09-30T01:37:15.606 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:15.580+0000 7fdfcaffd700 10 monclient(hunting): get_auth_request con 0x7fdfcc0a3270 auth_method 0 2023-09-30T01:37:15.606 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:15.580+0000 7fdfcaffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:37:15.606 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:15.580+0000 7fdfcaffd700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:37:15.606 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:15.580+0000 7fdfcaffd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:37:15.606 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:15.580+0000 7fdfca7fc700 10 monclient(hunting): get_auth_request con 0x7fdfcc0a4b00 auth_method 0 2023-09-30T01:37:15.606 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:15.580+0000 7fdfca7fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:37:15.606 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:15.580+0000 7fdfca7fc700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:37:15.606 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:15.580+0000 7fdfca7fc700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:37:15.606 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:15.580+0000 7fdfcaffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:37:15.606 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:15.580+0000 7fdfcaffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:37:15.610 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:15.580+0000 7fdfcaffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:37:15.610 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:15.580+0000 7fdfca7fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:37:15.610 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:15.580+0000 7fdfca7fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:37:15.610 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:15.580+0000 7fdfca7fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:37:15.610 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:15.580+0000 7fdfcb7fe700 1 --2- 172.21.15.88:0/295005856 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7fdfcc1333c0 0x7fdfcc11d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:37:15.610 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:15.580+0000 7fdfcaffd700 10 monclient(hunting): handle_auth_done global_id 4695 payload 824 2023-09-30T01:37:15.610 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:15.580+0000 7fdfcaffd700 1 -- 172.21.15.88:0/295005856 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7fdfcc0a4b00 msgr2=0x7fdfcc11ce50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:37:15.610 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:15.580+0000 7fdfcaffd700 1 --2- 172.21.15.88:0/295005856 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7fdfcc0a4b00 0x7fdfcc11ce50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:37:15.610 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:15.580+0000 7fdfcaffd700 1 -- 172.21.15.88:0/295005856 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7fdfcc1333c0 msgr2=0x7fdfcc11d390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:37:15.611 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:15.580+0000 7fdfcaffd700 1 --2- 172.21.15.88:0/295005856 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7fdfcc1333c0 0x7fdfcc11d390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:37:15.611 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:15.580+0000 7fdfcaffd700 10 monclient: _finish_hunting 0 2023-09-30T01:37:15.611 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:15.580+0000 7fdfcaffd700 1 monclient: found mon.noname-d 2023-09-30T01:37:15.611 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:15.580+0000 7fdfcaffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:37:15.611 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:15.580+0000 7fdfcaffd700 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.88:3303/0 2023-09-30T01:37:15.612 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:15.580+0000 7fdfcaffd700 1 -- 172.21.15.88:0/295005856 --> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdfc00094f0 con 0x7fdfcc0a3270 2023-09-30T01:37:15.612 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:15.580+0000 7fdfcb7fe700 10 monclient: get_auth_request con 0x7fdfcc1333c0 auth_method 0 2023-09-30T01:37:15.612 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:15.580+0000 7fdfcb7fe700 1 --2- 172.21.15.88:0/295005856 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7fdfcc1333c0 0x7fdfcc11d390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-09-30T01:37:15.612 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:15.580+0000 7fdfcaffd700 1 --2- 172.21.15.88:0/295005856 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7fdfcc0a3270 0x7fdfcc0a0500 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fdfc000ec80 tx=0x7fdfc0009250 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:37:15.612 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:15.580+0000 7fdfb2ffd700 1 -- 172.21.15.88:0/295005856 <== mon.6 v2:172.21.15.88:3303/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7fdfc000cda0 con 0x7fdfcc0a3270 2023-09-30T01:37:15.612 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:15.580+0000 7fdfb2ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:37:15.612 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:15.580+0000 7fdfb2ffd700 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2023-09-30T01:37:15.612 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:15.580+0000 7fdfb2ffd700 10 monclient: dump: 2023-09-30T01:37:15.612 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:37:15.612 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:37:15.613 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:37:15.613 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:37:15.613 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:37:15.613 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:37:15.613 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:37:15.613 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:37:15.613 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:37:15.613 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:37:15.613 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:37:15.613 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:37:15.613 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:37:15.613 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:37:15.614 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:37:15.614 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:37:15.614 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:15.580+0000 7fdfb2ffd700 10 monclient: _finish_auth 0 2023-09-30T01:37:15.614 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:15.580+0000 7fdfb2ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:37:15.614 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:15.580+0000 7fdfb2ffd700 1 -- 172.21.15.88:0/295005856 <== mon.6 v2:172.21.15.88:3303/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdfc0013470 con 0x7fdfcc0a3270 2023-09-30T01:37:15.614 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:15.580+0000 7fdfd17ba700 5 monclient: authenticate success, global_id 4695 2023-09-30T01:37:15.614 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:15.580+0000 7fdfd17ba700 10 monclient: _renew_subs 2023-09-30T01:37:15.614 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:15.580+0000 7fdfd17ba700 10 monclient: _send_mon_message to mon.d at v2:172.21.15.88:3303/0 2023-09-30T01:37:15.615 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:15.580+0000 7fdfd17ba700 1 -- 172.21.15.88:0/295005856 --> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdfbc009420 con 0x7fdfcc0a3270 2023-09-30T01:37:15.615 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:15.580+0000 7fdfb2ffd700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:37:15.615 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:15.580+0000 7fdfb2ffd700 1 -- 172.21.15.88:0/295005856 <== mon.6 v2:172.21.15.88:3303/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7fdfc0013a60 con 0x7fdfcc0a3270 2023-09-30T01:37:15.615 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:15.580+0000 7fdfb2ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:37:15.615 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:15.580+0000 7fdfb2ffd700 10 monclient: got monmap 1 from mon.d (according to old e1) 2023-09-30T01:37:15.615 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:15.580+0000 7fdfc97fa700 4 set_mon_vals no callback set 2023-09-30T01:37:15.615 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:15.580+0000 7fdfb2ffd700 10 monclient: dump: 2023-09-30T01:37:15.615 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:37:15.615 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:37:15.615 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:37:15.615 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:37:15.616 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:37:15.616 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:37:15.616 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:37:15.616 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:37:15.616 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:37:15.616 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:37:15.616 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:37:15.616 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:37:15.616 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:37:15.616 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:37:15.616 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:37:15.617 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:37:15.617 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:15.580+0000 7fdfd17ba700 10 monclient: _renew_subs 2023-09-30T01:37:15.617 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:15.580+0000 7fdfd17ba700 10 monclient: _send_mon_message to mon.d at v2:172.21.15.88:3303/0 2023-09-30T01:37:15.617 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:15.580+0000 7fdfd17ba700 1 -- 172.21.15.88:0/295005856 --> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdfc4009a20 con 0x7fdfcc0a3270 2023-09-30T01:37:15.617 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:15.580+0000 7fdfb2ffd700 1 -- 172.21.15.88:0/295005856 <== mon.6 v2:172.21.15.88:3303/0 4 ==== mgrmap(e 5) v1 ==== 40144+0+0 (secure 0 0 0) 0x7fdfc0013bc0 con 0x7fdfcc0a3270 2023-09-30T01:37:15.617 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:15.580+0000 7fdfd17ba700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:37:15.617 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:15.580+0000 7fdfd17ba700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:37:15.617 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:15.580+0000 7fdfd17ba700 10 monclient: _send_mon_message to mon.d at v2:172.21.15.88:3303/0 2023-09-30T01:37:15.617 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:15.580+0000 7fdfd17ba700 1 -- 172.21.15.88:0/295005856 --> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdfcc1200d0 con 0x7fdfcc0a3270 2023-09-30T01:37:15.617 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:15.580+0000 7fdfb2ffd700 1 --2- 172.21.15.88:0/295005856 >> v2:172.21.15.107:6812/29926 conn(0x7fdfb80332c0 0x7fdfb8035770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:37:15.617 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:15.580+0000 7fdfb2ffd700 1 -- 172.21.15.88:0/295005856 <== mon.6 v2:172.21.15.88:3303/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 3502+0+0 (secure 0 0 0) 0x7fdfc0043ab0 con 0x7fdfcc0a3270 2023-09-30T01:37:15.618 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:15.584+0000 7fdfca7fc700 1 --2- 172.21.15.88:0/295005856 >> v2:172.21.15.107:6812/29926 conn(0x7fdfb80332c0 0x7fdfb8035770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:37:15.618 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:15.584+0000 7fdfca7fc700 10 monclient: get_auth_request con 0x7fdfb80332c0 auth_method 0 2023-09-30T01:37:15.618 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:15.584+0000 7fdfca7fc700 1 --2- 172.21.15.88:0/295005856 >> v2:172.21.15.107:6812/29926 conn(0x7fdfb80332c0 0x7fdfb8035770 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fdfbc009f40 tx=0x7fdfbc0091a0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:37:15.618 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:15.592+0000 7fdfb2ffd700 1 -- 172.21.15.88:0/295005856 <== mon.6 v2:172.21.15.88:3303/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fdfc0013870 con 0x7fdfcc0a3270 2023-09-30T01:37:15.618 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:15.592+0000 7fdfb2ffd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:37:15.619 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:15.592+0000 7fdfb2ffd700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:37:15.792 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:15.788+0000 7fdfd17ba700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T01:37:15.792 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:15.788+0000 7fdfd17ba700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:37:15.792 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:15.788+0000 7fdfd17ba700 10 monclient: _send_mon_message to mon.d at v2:172.21.15.88:3303/0 2023-09-30T01:37:15.792 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:15.788+0000 7fdfd17ba700 1 -- 172.21.15.88:0/295005856 --> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fdfcc09cbe0 con 0x7fdfcc0a3270 2023-09-30T01:37:15.793 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:15.788+0000 7fdfb2ffd700 1 -- 172.21.15.88:0/295005856 <== mon.6 v2:172.21.15.88:3303/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7fdfc001b3b0 con 0x7fdfcc0a3270 2023-09-30T01:37:15.793 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:15.788+0000 7fdfb2ffd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:37:15.793 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:15.788+0000 7fdfb2ffd700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:37:15.793 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-09-30T01:37:15.794 INFO:teuthology.orchestra.run.smithi088.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:37:15.798 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:15.792+0000 7fdfd17ba700 1 -- 172.21.15.88:0/295005856 >> v2:172.21.15.107:6812/29926 conn(0x7fdfb80332c0 msgr2=0x7fdfb8035770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:37:15.798 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:15.792+0000 7fdfd17ba700 1 --2- 172.21.15.88:0/295005856 >> v2:172.21.15.107:6812/29926 conn(0x7fdfb80332c0 0x7fdfb8035770 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fdfbc009f40 tx=0x7fdfbc0091a0 comp rx=0 tx=0).stop 2023-09-30T01:37:15.798 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:15.792+0000 7fdfd17ba700 10 monclient: shutdown 2023-09-30T01:37:15.798 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:15.792+0000 7fdfd17ba700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:37:15.798 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:15.792+0000 7fdfd17ba700 1 -- 172.21.15.88:0/295005856 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7fdfcc0a3270 msgr2=0x7fdfcc0a0500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:37:15.798 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:15.792+0000 7fdfd17ba700 1 --2- 172.21.15.88:0/295005856 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7fdfcc0a3270 0x7fdfcc0a0500 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fdfc000ec80 tx=0x7fdfc0009250 comp rx=0 tx=0).stop 2023-09-30T01:37:15.798 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:15.792+0000 7fdfd17ba700 1 -- 172.21.15.88:0/295005856 shutdown_connections 2023-09-30T01:37:15.799 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:15.792+0000 7fdfd17ba700 1 --2- 172.21.15.88:0/295005856 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7fdfcc1333c0 0x7fdfcc11d390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:37:15.799 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:15.792+0000 7fdfd17ba700 1 --2- 172.21.15.88:0/295005856 >> v2:172.21.15.107:6812/29926 conn(0x7fdfb80332c0 0x7fdfb8035770 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:37:15.799 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:15.792+0000 7fdfd17ba700 1 --2- 172.21.15.88:0/295005856 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7fdfcc0a4b00 0x7fdfcc11ce50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:37:15.799 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:15.792+0000 7fdfd17ba700 1 --2- 172.21.15.88:0/295005856 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7fdfcc0a3270 0x7fdfcc0a0500 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:37:15.799 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:15.792+0000 7fdfd17ba700 1 -- 172.21.15.88:0/295005856 >> 172.21.15.88:0/295005856 conn(0x7fdfcc0a67f0 msgr2=0x7fdfcc130920 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:37:15.799 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:15.792+0000 7fdfd17ba700 1 -- 172.21.15.88:0/295005856 shutdown_connections 2023-09-30T01:37:15.799 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:15.792+0000 7fdfd17ba700 1 -- 172.21.15.88:0/295005856 wait complete. 2023-09-30T01:37:18.814 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:37:18.927 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:18.920+0000 7fe9b1d70700 10 monclient: get_monmap_and_config 2023-09-30T01:37:18.927 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:18.920+0000 7fe9b1d70700 10 monclient: build_initial_monmap 2023-09-30T01:37:18.927 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:18.920+0000 7fe9b1d70700 10 monclient: monmap: 2023-09-30T01:37:18.927 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2023-09-30T01:37:18.928 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:37:18.928 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:37:18.926759+0000 2023-09-30T01:37:18.928 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:37:18.926759+0000 2023-09-30T01:37:18.928 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2023-09-30T01:37:18.928 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:37:18.928 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a-legacy 2023-09-30T01:37:18.928 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.107:6789/0 mon.noname-f-legacy 2023-09-30T01:37:18.929 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.noname-b 2023-09-30T01:37:18.929 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.noname-g 2023-09-30T01:37:18.929 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.noname-c 2023-09-30T01:37:18.929 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.noname-h 2023-09-30T01:37:18.929 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.noname-d 2023-09-30T01:37:18.929 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.noname-i 2023-09-30T01:37:18.929 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.noname-e 2023-09-30T01:37:18.929 INFO:teuthology.orchestra.run.smithi088.stderr:9: v2:172.21.15.88:3300/0 mon.noname-a 2023-09-30T01:37:18.929 INFO:teuthology.orchestra.run.smithi088.stderr:10: v2:172.21.15.107:3300/0 mon.noname-f 2023-09-30T01:37:18.929 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:37:18.933 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:18.924+0000 7fe9b1d70700 1 Processor -- start 2023-09-30T01:37:18.934 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:18.924+0000 7fe9b1d70700 1 -- start start 2023-09-30T01:37:18.934 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:18.924+0000 7fe9b1d70700 10 monclient: init 2023-09-30T01:37:18.934 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:18.924+0000 7fe9b1d70700 10 monclient: _reopen_session rank -1 2023-09-30T01:37:18.934 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:18.924+0000 7fe9b1d70700 10 monclient: _add_conns ranks=[4,10,0,2,5,8,6,7,9,3,1] 2023-09-30T01:37:18.935 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:18.924+0000 7fe9b1d70700 1 --2- >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7fe9ac0a3380 0x7fe9ac0a48b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:37:18.935 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:18.924+0000 7fe9b1d70700 10 monclient(hunting): picked mon.noname-c con 0x7fe9ac0a3380 addr [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] 2023-09-30T01:37:18.935 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:18.924+0000 7fe9b1d70700 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fe9ac09ffd0 0x7fe9ac0a03a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:37:18.935 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:18.924+0000 7fe9b1d70700 10 monclient(hunting): picked mon.noname-f con 0x7fe9ac09ffd0 addr v2:172.21.15.107:3300/0 2023-09-30T01:37:18.935 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:18.924+0000 7fe9b1d70700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fe9ac1333b0 addr v1:172.21.15.88:6789/0 2023-09-30T01:37:18.935 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:18.924+0000 7fe9b1d70700 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9ac04bb20 con 0x7fe9ac1333b0 2023-09-30T01:37:18.935 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:18.924+0000 7fe9b1d70700 10 monclient(hunting): start opening mon connection 2023-09-30T01:37:18.935 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:18.924+0000 7fe9b1d70700 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe9ac097ab0 con 0x7fe9ac09ffd0 2023-09-30T01:37:18.935 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:18.924+0000 7fe9b1d70700 10 monclient(hunting): start opening mon connection 2023-09-30T01:37:18.935 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:18.924+0000 7fe9b1d70700 1 -- --> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] -- mon_getmap magic: 0 v1 -- 0x7fe9ac0a08e0 con 0x7fe9ac0a3380 2023-09-30T01:37:18.935 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:18.924+0000 7fe9b1d70700 10 monclient(hunting): _renew_subs 2023-09-30T01:37:18.936 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:18.924+0000 7fe9b1d70700 10 monclient(hunting): authenticate will time out at 1110.964723s 2023-09-30T01:37:18.936 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:18.924+0000 7fe9b0d6e700 1 --2- >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7fe9ac0a3380 0x7fe9ac0a48b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:37:18.936 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:18.924+0000 7fe9b0d6e700 1 --2- >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7fe9ac0a3380 0x7fe9ac0a48b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3302/0 says I am v2:172.21.15.88:58946/0 (socket says 172.21.15.88:58946) 2023-09-30T01:37:18.936 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:18.924+0000 7fe9b0d6e700 1 -- 172.21.15.88:0/950068152 learned_addr learned my addr 172.21.15.88:0/950068152 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-09-30T01:37:18.936 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:18.924+0000 7fe9b0d6e700 10 monclient(hunting): get_auth_request con 0x7fe9ac0a3380 auth_method 0 2023-09-30T01:37:18.936 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:18.924+0000 7fe9b0d6e700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:37:18.936 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:18.924+0000 7fe9b0d6e700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:37:18.936 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:18.924+0000 7fe9b0d6e700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:37:18.936 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:18.924+0000 7fe9b0d6e700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:37:18.936 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:18.924+0000 7fe9b0d6e700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:37:18.936 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:18.924+0000 7fe9b0d6e700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:37:18.937 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:18.924+0000 7fe9abfff700 1 --2- 172.21.15.88:0/950068152 >> v2:172.21.15.107:3300/0 conn(0x7fe9ac09ffd0 0x7fe9ac0a03a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:37:18.937 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:18.924+0000 7fe9abfff700 10 monclient(hunting): get_auth_request con 0x7fe9ac09ffd0 auth_method 0 2023-09-30T01:37:18.937 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:18.924+0000 7fe9abfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:37:18.937 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:18.924+0000 7fe9abfff700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:37:18.937 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:18.924+0000 7fe9abfff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:37:18.938 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:18.924+0000 7fe9b0d6e700 10 monclient(hunting): handle_auth_done global_id 4792 payload 274 2023-09-30T01:37:18.938 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:18.924+0000 7fe9b0d6e700 1 -- 172.21.15.88:0/950068152 >> v2:172.21.15.107:3300/0 conn(0x7fe9ac09ffd0 msgr2=0x7fe9ac0a03a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:37:18.938 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:18.924+0000 7fe9b0d6e700 1 --2- 172.21.15.88:0/950068152 >> v2:172.21.15.107:3300/0 conn(0x7fe9ac09ffd0 0x7fe9ac0a03a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:37:18.938 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:18.924+0000 7fe9b0d6e700 1 -- 172.21.15.88:0/950068152 >> v1:172.21.15.88:6789/0 conn(0x7fe9ac1333b0 legacy=0x7fe9ac133820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:37:18.938 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:18.924+0000 7fe9b0d6e700 10 monclient: _finish_hunting 0 2023-09-30T01:37:18.938 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:18.924+0000 7fe9b0d6e700 1 monclient: found mon.noname-c 2023-09-30T01:37:18.938 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:18.924+0000 7fe9b0d6e700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:37:18.938 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:18.924+0000 7fe9b0d6e700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.88:3302/0 2023-09-30T01:37:18.939 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:18.924+0000 7fe9b0d6e700 1 -- 172.21.15.88:0/950068152 --> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9ac098480 con 0x7fe9ac0a3380 2023-09-30T01:37:18.939 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:18.924+0000 7fe9b0d6e700 1 --2- 172.21.15.88:0/950068152 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7fe9ac0a3380 0x7fe9ac0a48b0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fe99c0099b0 tx=0x7fe99c009250 comp rx=0 tx=0).ready entity=mon.4 client_cookie=e7efa7e1ae32abba server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:37:18.939 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:18.924+0000 7fe9ab7fe700 1 -- 172.21.15.88:0/950068152 <== mon.4 v2:172.21.15.88:3302/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7fe99c00bce0 con 0x7fe9ac0a3380 2023-09-30T01:37:18.939 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:18.924+0000 7fe9ab7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:37:18.939 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:18.924+0000 7fe9ab7fe700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T01:37:18.939 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:18.924+0000 7fe9ab7fe700 10 monclient: dump: 2023-09-30T01:37:18.939 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:37:18.939 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:37:18.939 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:37:18.940 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:37:18.940 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:37:18.940 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:37:18.940 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:37:18.940 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:37:18.940 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:37:18.940 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:37:18.940 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:37:18.941 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:37:18.941 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:37:18.941 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:37:18.941 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:37:18.941 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:37:18.941 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:18.924+0000 7fe9ab7fe700 10 monclient: _finish_auth 0 2023-09-30T01:37:18.941 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:18.924+0000 7fe9ab7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:37:18.941 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:18.924+0000 7fe9ab7fe700 1 -- 172.21.15.88:0/950068152 <== mon.4 v2:172.21.15.88:3302/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe99c00be60 con 0x7fe9ac0a3380 2023-09-30T01:37:18.941 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:18.924+0000 7fe9ab7fe700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:37:18.941 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:18.924+0000 7fe9b1d70700 5 monclient: authenticate success, global_id 4792 2023-09-30T01:37:18.942 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:18.924+0000 7fe9b1d70700 10 monclient: get_monmap_and_config success 2023-09-30T01:37:18.942 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:18.924+0000 7fe9b1d70700 4 set_mon_vals no callback set 2023-09-30T01:37:18.942 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:18.924+0000 7fe9ab7fe700 1 -- 172.21.15.88:0/950068152 <== mon.4 v2:172.21.15.88:3302/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7fe99c011420 con 0x7fe9ac0a3380 2023-09-30T01:37:18.942 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:18.924+0000 7fe9ab7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:37:18.942 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:18.924+0000 7fe9ab7fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T01:37:18.942 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:18.924+0000 7fe9ab7fe700 10 monclient: dump: 2023-09-30T01:37:18.942 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:37:18.942 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:37:18.942 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:37:18.942 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:37:18.942 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:37:18.942 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:37:18.943 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:37:18.943 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:37:18.943 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:37:18.943 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:37:18.943 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:37:18.943 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:37:18.943 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:37:18.943 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:37:18.943 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:37:18.943 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:37:18.943 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:18.924+0000 7fe9b1d70700 10 monclient: shutdown 2023-09-30T01:37:18.943 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:18.924+0000 7fe9b1d70700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:37:18.944 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:18.924+0000 7fe9b1d70700 1 -- 172.21.15.88:0/950068152 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7fe9ac0a3380 msgr2=0x7fe9ac0a48b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:37:18.944 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:18.924+0000 7fe9b1d70700 1 --2- 172.21.15.88:0/950068152 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7fe9ac0a3380 0x7fe9ac0a48b0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fe99c0099b0 tx=0x7fe99c009250 comp rx=0 tx=0).stop 2023-09-30T01:37:18.944 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:18.924+0000 7fe9b1d70700 1 -- 172.21.15.88:0/950068152 shutdown_connections 2023-09-30T01:37:18.944 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:18.924+0000 7fe9b1d70700 1 --2- 172.21.15.88:0/950068152 >> v2:172.21.15.107:3300/0 conn(0x7fe9ac09ffd0 0x7fe9ac0a03a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:37:18.944 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:18.924+0000 7fe9b1d70700 1 --2- 172.21.15.88:0/950068152 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7fe9ac0a3380 0x7fe9ac0a48b0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:37:18.944 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:18.924+0000 7fe9b1d70700 1 -- 172.21.15.88:0/950068152 >> 172.21.15.88:0/950068152 conn(0x7fe9ac06e5e0 msgr2=0x7fe9ac06e9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:37:18.944 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:18.924+0000 7fe9b1d70700 1 -- 172.21.15.88:0/950068152 shutdown_connections 2023-09-30T01:37:18.944 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:18.924+0000 7fe9b1d70700 1 -- 172.21.15.88:0/950068152 wait complete. 2023-09-30T01:37:18.944 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:18.928+0000 7fe9b1d70700 10 monclient: build_initial_monmap 2023-09-30T01:37:18.944 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:18.928+0000 7fe9b1d70700 10 monclient: monmap: 2023-09-30T01:37:18.944 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2023-09-30T01:37:18.944 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:37:18.945 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 0.000000 2023-09-30T01:37:18.945 INFO:teuthology.orchestra.run.smithi088.stderr:created 0.000000 2023-09-30T01:37:18.945 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2023-09-30T01:37:18.945 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 1 2023-09-30T01:37:18.945 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.noname-a 2023-09-30T01:37:18.945 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.noname-f 2023-09-30T01:37:18.945 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.noname-b 2023-09-30T01:37:18.945 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.noname-g 2023-09-30T01:37:18.945 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.noname-c 2023-09-30T01:37:18.946 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.noname-h 2023-09-30T01:37:18.946 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.noname-d 2023-09-30T01:37:18.946 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.noname-i 2023-09-30T01:37:18.946 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.noname-e 2023-09-30T01:37:18.946 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:37:18.946 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:18.928+0000 7fe9b1d70700 1 Processor -- start 2023-09-30T01:37:18.946 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:18.928+0000 7fe9b1d70700 1 -- start start 2023-09-30T01:37:18.946 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:18.928+0000 7fe9b1d70700 10 monclient: init 2023-09-30T01:37:18.952 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:18.928+0000 7fe9b1d70700 10 monclient: _reopen_session rank -1 2023-09-30T01:37:18.952 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:18.928+0000 7fe9b1d70700 10 monclient: _add_conns ranks=[7,3,5,6,8,1,2,4,0] 2023-09-30T01:37:18.952 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:18.928+0000 7fe9b1d70700 1 --2- >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7fe9ac09ffd0 0x7fe9ac11d850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:37:18.952 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:18.928+0000 7fe9b1d70700 10 monclient(hunting): picked mon.noname-i con 0x7fe9ac09ffd0 addr [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] 2023-09-30T01:37:18.952 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:18.928+0000 7fe9b1d70700 1 --2- >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7fe9ac0a3380 0x7fe9ac11dd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:37:18.952 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:18.928+0000 7fe9b1d70700 10 monclient(hunting): picked mon.noname-g con 0x7fe9ac0a3380 addr [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] 2023-09-30T01:37:18.952 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:18.928+0000 7fe9b1d70700 1 --2- >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7fe9ac1333b0 0x7fe9ac11e2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:37:18.952 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:18.928+0000 7fe9b1d70700 10 monclient(hunting): picked mon.noname-h con 0x7fe9ac1333b0 addr [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] 2023-09-30T01:37:18.953 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:18.928+0000 7fe9b1d70700 10 monclient(hunting): start opening mon connection 2023-09-30T01:37:18.953 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:18.928+0000 7fe9b1d70700 1 -- --> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe9ac121200 con 0x7fe9ac0a3380 2023-09-30T01:37:18.953 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:18.928+0000 7fe9b1d70700 10 monclient(hunting): start opening mon connection 2023-09-30T01:37:18.953 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:18.928+0000 7fe9b1d70700 1 -- --> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] -- mon_getmap magic: 0 v1 -- 0x7fe9ac130fb0 con 0x7fe9ac1333b0 2023-09-30T01:37:18.953 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:18.928+0000 7fe9b1d70700 10 monclient(hunting): start opening mon connection 2023-09-30T01:37:18.953 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:18.928+0000 7fe9b1d70700 1 -- --> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] -- mon_getmap magic: 0 v1 -- 0x7fe9ac0a0ac0 con 0x7fe9ac09ffd0 2023-09-30T01:37:18.953 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:18.928+0000 7fe9b1d70700 10 monclient(hunting): _renew_subs 2023-09-30T01:37:18.953 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:18.928+0000 7fe9b1d70700 10 monclient(hunting): authenticate will time out at 1110.970932s 2023-09-30T01:37:18.953 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:18.928+0000 7fe9b0d6e700 1 --2- >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7fe9ac09ffd0 0x7fe9ac11d850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:37:18.953 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:18.928+0000 7fe9abfff700 1 --2- >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7fe9ac0a3380 0x7fe9ac11dd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:37:18.953 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:18.928+0000 7fe9b0d6e700 1 --2- >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7fe9ac09ffd0 0x7fe9ac11d850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3303/0 says I am v2:172.21.15.88:48026/0 (socket says 172.21.15.88:48026) 2023-09-30T01:37:18.953 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:18.928+0000 7fe9b156f700 1 --2- >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7fe9ac1333b0 0x7fe9ac11e2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:37:18.954 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:18.928+0000 7fe9b0d6e700 1 -- 172.21.15.88:0/1262697114 learned_addr learned my addr 172.21.15.88:0/1262697114 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-09-30T01:37:18.954 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:18.928+0000 7fe9b0d6e700 10 monclient(hunting): get_auth_request con 0x7fe9ac09ffd0 auth_method 0 2023-09-30T01:37:18.954 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:18.928+0000 7fe9b0d6e700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:37:18.954 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:18.928+0000 7fe9b0d6e700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:37:18.954 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:18.928+0000 7fe9b0d6e700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:37:18.954 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:18.928+0000 7fe9abfff700 10 monclient(hunting): get_auth_request con 0x7fe9ac0a3380 auth_method 0 2023-09-30T01:37:18.954 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:18.928+0000 7fe9abfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:37:18.954 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:18.928+0000 7fe9abfff700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:37:18.954 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:18.928+0000 7fe9abfff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:37:18.954 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:18.928+0000 7fe9b156f700 10 monclient(hunting): get_auth_request con 0x7fe9ac1333b0 auth_method 0 2023-09-30T01:37:18.954 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:18.928+0000 7fe9b156f700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:37:18.955 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:18.928+0000 7fe9b156f700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:37:18.955 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:18.928+0000 7fe9b156f700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:37:18.955 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:18.928+0000 7fe9b0d6e700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:37:18.955 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:18.928+0000 7fe9b0d6e700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:37:18.955 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:18.928+0000 7fe9b0d6e700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:37:18.955 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:18.928+0000 7fe9abfff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:37:18.955 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:18.928+0000 7fe9abfff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:37:18.955 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:18.928+0000 7fe9abfff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:37:18.955 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:18.928+0000 7fe9b156f700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:37:18.955 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:18.928+0000 7fe9b156f700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:37:18.955 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:18.928+0000 7fe9b156f700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:37:18.955 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:18.928+0000 7fe9abfff700 10 monclient(hunting): handle_auth_done global_id 4620 payload 824 2023-09-30T01:37:18.956 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:18.928+0000 7fe9abfff700 1 -- 172.21.15.88:0/1262697114 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7fe9ac09ffd0 msgr2=0x7fe9ac11d850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:37:18.956 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:18.928+0000 7fe9abfff700 1 --2- 172.21.15.88:0/1262697114 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7fe9ac09ffd0 0x7fe9ac11d850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:37:18.956 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:18.928+0000 7fe9abfff700 1 -- 172.21.15.88:0/1262697114 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7fe9ac1333b0 msgr2=0x7fe9ac11e2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:37:18.956 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:18.928+0000 7fe9abfff700 1 --2- 172.21.15.88:0/1262697114 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7fe9ac1333b0 0x7fe9ac11e2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:37:18.956 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:18.928+0000 7fe9abfff700 10 monclient: _finish_hunting 0 2023-09-30T01:37:18.956 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:18.928+0000 7fe9abfff700 1 monclient: found mon.noname-g 2023-09-30T01:37:18.956 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:18.928+0000 7fe9abfff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:37:18.956 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:18.928+0000 7fe9abfff700 10 monclient: _send_mon_message to mon.noname-g at v2:172.21.15.107:3301/0 2023-09-30T01:37:18.956 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:18.928+0000 7fe9abfff700 1 -- 172.21.15.88:0/1262697114 --> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe99c0094f0 con 0x7fe9ac0a3380 2023-09-30T01:37:18.956 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:18.932+0000 7fe9b0d6e700 1 --2- 172.21.15.88:0/1262697114 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7fe9ac09ffd0 0x7fe9ac11d850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:37:18.956 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:18.932+0000 7fe9abfff700 1 --2- 172.21.15.88:0/1262697114 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7fe9ac0a3380 0x7fe9ac11dd90 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fe99400d890 tx=0x7fe994008040 comp rx=0 tx=0).ready entity=mon.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:37:18.957 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:18.932+0000 7fe9a8ff9700 1 -- 172.21.15.88:0/1262697114 <== mon.3 v2:172.21.15.107:3301/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7fe9940073c0 con 0x7fe9ac0a3380 2023-09-30T01:37:18.957 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:18.932+0000 7fe9a8ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:37:18.957 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:18.932+0000 7fe9a8ff9700 10 monclient: got monmap 1 from mon.noname-g (according to old e1) 2023-09-30T01:37:18.957 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:18.932+0000 7fe9a8ff9700 10 monclient: dump: 2023-09-30T01:37:18.957 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:37:18.957 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:37:18.957 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:37:18.957 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:37:18.957 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:37:18.957 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:37:18.958 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:37:18.958 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:37:18.958 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:37:18.958 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:37:18.958 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:37:18.958 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:37:18.958 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:37:18.958 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:37:18.959 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:37:18.959 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:37:18.959 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:18.932+0000 7fe9a8ff9700 10 monclient: _finish_auth 0 2023-09-30T01:37:18.959 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:18.932+0000 7fe9a8ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:37:18.959 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:18.932+0000 7fe9b1d70700 5 monclient: authenticate success, global_id 4620 2023-09-30T01:37:18.959 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:18.932+0000 7fe9a8ff9700 1 -- 172.21.15.88:0/1262697114 <== mon.3 v2:172.21.15.107:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe994012a70 con 0x7fe9ac0a3380 2023-09-30T01:37:18.959 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:18.932+0000 7fe9a8ff9700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:37:18.960 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:18.932+0000 7fe9a8ff9700 1 -- 172.21.15.88:0/1262697114 <== mon.3 v2:172.21.15.107:3301/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7fe994012be0 con 0x7fe9ac0a3380 2023-09-30T01:37:18.960 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:18.932+0000 7fe9b1d70700 10 monclient: _renew_subs 2023-09-30T01:37:18.960 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:18.932+0000 7fe9b1d70700 10 monclient: _send_mon_message to mon.g at v2:172.21.15.107:3301/0 2023-09-30T01:37:18.960 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:18.932+0000 7fe9b1d70700 1 -- 172.21.15.88:0/1262697114 --> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe9ac097ce0 con 0x7fe9ac0a3380 2023-09-30T01:37:18.960 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:18.932+0000 7fe9a8ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:37:18.960 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:18.932+0000 7fe9aaffd700 4 set_mon_vals no callback set 2023-09-30T01:37:18.960 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:18.932+0000 7fe9a8ff9700 10 monclient: got monmap 1 from mon.g (according to old e1) 2023-09-30T01:37:18.960 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:18.932+0000 7fe9a8ff9700 10 monclient: dump: 2023-09-30T01:37:18.960 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:37:18.960 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:37:18.960 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:37:18.961 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:37:18.961 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:37:18.961 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:37:18.961 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:37:18.961 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:37:18.961 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:37:18.961 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:37:18.961 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:37:18.961 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:37:18.961 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:37:18.962 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:37:18.962 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:37:18.962 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:37:18.962 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:18.932+0000 7fe9b1d70700 10 monclient: _renew_subs 2023-09-30T01:37:18.962 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:18.932+0000 7fe9b1d70700 10 monclient: _send_mon_message to mon.g at v2:172.21.15.107:3301/0 2023-09-30T01:37:18.962 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:18.932+0000 7fe9b1d70700 1 -- 172.21.15.88:0/1262697114 --> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9ac11f3f0 con 0x7fe9ac0a3380 2023-09-30T01:37:18.962 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:18.932+0000 7fe9b1d70700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:37:18.962 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:18.932+0000 7fe9b1d70700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:37:18.962 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:18.932+0000 7fe9b1d70700 10 monclient: _send_mon_message to mon.g at v2:172.21.15.107:3301/0 2023-09-30T01:37:18.962 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:18.932+0000 7fe9b1d70700 1 -- 172.21.15.88:0/1262697114 --> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe9ac0710d0 con 0x7fe9ac0a3380 2023-09-30T01:37:18.963 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:18.932+0000 7fe9a8ff9700 1 -- 172.21.15.88:0/1262697114 <== mon.3 v2:172.21.15.107:3301/0 4 ==== mgrmap(e 5) v1 ==== 40144+0+0 (secure 0 0 0) 0x7fe994012d40 con 0x7fe9ac0a3380 2023-09-30T01:37:18.963 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:18.932+0000 7fe9a8ff9700 1 --2- 172.21.15.88:0/1262697114 >> v2:172.21.15.107:6812/29926 conn(0x7fe98c033360 0x7fe98c035810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:37:18.963 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:18.932+0000 7fe9a8ff9700 1 -- 172.21.15.88:0/1262697114 <== mon.3 v2:172.21.15.107:3301/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 3502+0+0 (secure 0 0 0) 0x7fe9940437d0 con 0x7fe9ac0a3380 2023-09-30T01:37:18.963 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:18.932+0000 7fe9b0d6e700 1 --2- 172.21.15.88:0/1262697114 >> v2:172.21.15.107:6812/29926 conn(0x7fe98c033360 0x7fe98c035810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:37:18.963 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:18.932+0000 7fe9b0d6e700 10 monclient: get_auth_request con 0x7fe98c033360 auth_method 0 2023-09-30T01:37:18.963 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:18.936+0000 7fe9b0d6e700 1 --2- 172.21.15.88:0/1262697114 >> v2:172.21.15.107:6812/29926 conn(0x7fe98c033360 0x7fe98c035810 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fe99c009340 tx=0x7fe99c017000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:37:18.963 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:18.944+0000 7fe9a8ff9700 1 -- 172.21.15.88:0/1262697114 <== mon.3 v2:172.21.15.107:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fe9940125f0 con 0x7fe9ac0a3380 2023-09-30T01:37:18.963 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:18.944+0000 7fe9a8ff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:37:18.963 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:18.944+0000 7fe9a8ff9700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:37:19.140 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:19.136+0000 7fe9b1d70700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T01:37:19.141 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:19.136+0000 7fe9b1d70700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:37:19.141 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:19.136+0000 7fe9b1d70700 10 monclient: _send_mon_message to mon.g at v2:172.21.15.107:3301/0 2023-09-30T01:37:19.141 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:19.136+0000 7fe9b1d70700 1 -- 172.21.15.88:0/1262697114 --> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fe9ac067b60 con 0x7fe9ac0a3380 2023-09-30T01:37:19.141 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:19.136+0000 7fe9a8ff9700 1 -- 172.21.15.88:0/1262697114 <== mon.3 v2:172.21.15.107:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7fe994011e60 con 0x7fe9ac0a3380 2023-09-30T01:37:19.142 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:19.136+0000 7fe9a8ff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:37:19.142 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:19.136+0000 7fe9a8ff9700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:37:19.142 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-09-30T01:37:19.142 INFO:teuthology.orchestra.run.smithi088.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:37:19.144 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:19.136+0000 7fe9b1d70700 1 -- 172.21.15.88:0/1262697114 >> v2:172.21.15.107:6812/29926 conn(0x7fe98c033360 msgr2=0x7fe98c035810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:37:19.144 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:19.136+0000 7fe9b1d70700 1 --2- 172.21.15.88:0/1262697114 >> v2:172.21.15.107:6812/29926 conn(0x7fe98c033360 0x7fe98c035810 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fe99c009340 tx=0x7fe99c017000 comp rx=0 tx=0).stop 2023-09-30T01:37:19.145 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:19.136+0000 7fe9b1d70700 10 monclient: shutdown 2023-09-30T01:37:19.145 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:19.136+0000 7fe9b1d70700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:37:19.145 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:19.136+0000 7fe9b1d70700 1 -- 172.21.15.88:0/1262697114 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7fe9ac0a3380 msgr2=0x7fe9ac11dd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:37:19.145 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:19.136+0000 7fe9b1d70700 1 --2- 172.21.15.88:0/1262697114 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7fe9ac0a3380 0x7fe9ac11dd90 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fe99400d890 tx=0x7fe994008040 comp rx=0 tx=0).stop 2023-09-30T01:37:19.145 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:19.140+0000 7fe9b1d70700 1 -- 172.21.15.88:0/1262697114 shutdown_connections 2023-09-30T01:37:19.145 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:19.140+0000 7fe9b1d70700 1 --2- 172.21.15.88:0/1262697114 >> v2:172.21.15.107:6812/29926 conn(0x7fe98c033360 0x7fe98c035810 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:37:19.145 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:19.140+0000 7fe9b1d70700 1 --2- 172.21.15.88:0/1262697114 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7fe9ac1333b0 0x7fe9ac11e2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:37:19.145 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:19.140+0000 7fe9b1d70700 1 --2- 172.21.15.88:0/1262697114 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7fe9ac0a3380 0x7fe9ac11dd90 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:37:19.146 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:19.140+0000 7fe9b1d70700 1 --2- 172.21.15.88:0/1262697114 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7fe9ac09ffd0 0x7fe9ac11d850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:37:19.146 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:19.140+0000 7fe9b1d70700 1 -- 172.21.15.88:0/1262697114 >> 172.21.15.88:0/1262697114 conn(0x7fe9ac06e5e0 msgr2=0x7fe9ac1322e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:37:19.146 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:19.140+0000 7fe9b1d70700 1 -- 172.21.15.88:0/1262697114 shutdown_connections 2023-09-30T01:37:19.146 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:19.140+0000 7fe9b1d70700 1 -- 172.21.15.88:0/1262697114 wait complete. 2023-09-30T01:37:22.157 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:37:22.266 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:22.260+0000 7f2d94445700 10 monclient: get_monmap_and_config 2023-09-30T01:37:22.266 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:22.260+0000 7f2d94445700 10 monclient: build_initial_monmap 2023-09-30T01:37:22.267 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:22.260+0000 7f2d94445700 10 monclient: monmap: 2023-09-30T01:37:22.267 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2023-09-30T01:37:22.267 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:37:22.267 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:37:22.266460+0000 2023-09-30T01:37:22.267 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:37:22.266460+0000 2023-09-30T01:37:22.267 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2023-09-30T01:37:22.267 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:37:22.267 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a-legacy 2023-09-30T01:37:22.267 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.107:6789/0 mon.noname-f-legacy 2023-09-30T01:37:22.267 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.noname-b 2023-09-30T01:37:22.267 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.noname-g 2023-09-30T01:37:22.267 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.noname-c 2023-09-30T01:37:22.268 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.noname-h 2023-09-30T01:37:22.268 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.noname-d 2023-09-30T01:37:22.268 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.noname-i 2023-09-30T01:37:22.268 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.noname-e 2023-09-30T01:37:22.268 INFO:teuthology.orchestra.run.smithi088.stderr:9: v2:172.21.15.88:3300/0 mon.noname-a 2023-09-30T01:37:22.268 INFO:teuthology.orchestra.run.smithi088.stderr:10: v2:172.21.15.107:3300/0 mon.noname-f 2023-09-30T01:37:22.268 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:37:22.273 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:22.260+0000 7f2d94445700 1 Processor -- start 2023-09-30T01:37:22.273 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:22.260+0000 7f2d94445700 1 -- start start 2023-09-30T01:37:22.274 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:22.260+0000 7f2d94445700 10 monclient: init 2023-09-30T01:37:22.274 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:22.264+0000 7f2d94445700 10 monclient: _reopen_session rank -1 2023-09-30T01:37:22.274 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:22.264+0000 7f2d94445700 10 monclient: _add_conns ranks=[7,4,5,8,1,0,2,9,10,3,6] 2023-09-30T01:37:22.274 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:22.264+0000 7f2d94445700 1 --2- >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7f2d8c062c70 0x7f2d8c09ac40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:37:22.274 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:22.264+0000 7f2d94445700 10 monclient(hunting): picked mon.noname-i con 0x7f2d8c062c70 addr [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] 2023-09-30T01:37:22.274 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:22.264+0000 7f2d94445700 1 --2- >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f2d8c0988b0 0x7f2d8c063040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:37:22.274 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:22.264+0000 7f2d94445700 10 monclient(hunting): picked mon.noname-c con 0x7f2d8c0988b0 addr [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] 2023-09-30T01:37:22.274 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:22.264+0000 7f2d94445700 1 --2- >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7f2d8c063580 0x7f2d8c137660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:37:22.274 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:22.264+0000 7f2d94445700 10 monclient(hunting): picked mon.noname-h con 0x7f2d8c063580 addr [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] 2023-09-30T01:37:22.274 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:22.264+0000 7f2d94445700 10 monclient(hunting): start opening mon connection 2023-09-30T01:37:22.275 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:22.264+0000 7f2d94445700 1 -- --> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] -- mon_getmap magic: 0 v1 -- 0x7f2d8c09a820 con 0x7f2d8c0988b0 2023-09-30T01:37:22.275 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:22.264+0000 7f2d94445700 10 monclient(hunting): start opening mon connection 2023-09-30T01:37:22.275 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:22.264+0000 7f2d94445700 1 -- --> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] -- mon_getmap magic: 0 v1 -- 0x7f2d8c099f50 con 0x7f2d8c063580 2023-09-30T01:37:22.275 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:22.264+0000 7f2d94445700 10 monclient(hunting): start opening mon connection 2023-09-30T01:37:22.275 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:22.264+0000 7f2d94445700 1 -- --> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] -- mon_getmap magic: 0 v1 -- 0x7f2d8c09b180 con 0x7f2d8c062c70 2023-09-30T01:37:22.275 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:22.264+0000 7f2d94445700 10 monclient(hunting): _renew_subs 2023-09-30T01:37:22.275 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:22.264+0000 7f2d94445700 10 monclient(hunting): authenticate will time out at 1114.304153s 2023-09-30T01:37:22.275 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:22.264+0000 7f2d919e0700 1 --2- >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f2d8c0988b0 0x7f2d8c063040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:37:22.275 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:22.264+0000 7f2d919e0700 1 --2- >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f2d8c0988b0 0x7f2d8c063040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3302/0 says I am v2:172.21.15.88:36720/0 (socket says 172.21.15.88:36720) 2023-09-30T01:37:22.275 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:22.264+0000 7f2d929e2700 1 --2- >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7f2d8c063580 0x7f2d8c137660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:37:22.275 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:22.264+0000 7f2d919e0700 1 -- 172.21.15.88:0/3831994791 learned_addr learned my addr 172.21.15.88:0/3831994791 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-09-30T01:37:22.275 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:22.264+0000 7f2d919e0700 10 monclient(hunting): get_auth_request con 0x7f2d8c0988b0 auth_method 0 2023-09-30T01:37:22.276 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:22.264+0000 7f2d919e0700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:37:22.276 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:22.264+0000 7f2d919e0700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:37:22.276 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:22.264+0000 7f2d919e0700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:37:22.276 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:22.264+0000 7f2d921e1700 1 --2- 172.21.15.88:0/3831994791 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7f2d8c062c70 0x7f2d8c09ac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:37:22.276 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:22.264+0000 7f2d929e2700 10 monclient(hunting): get_auth_request con 0x7f2d8c063580 auth_method 0 2023-09-30T01:37:22.276 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:22.264+0000 7f2d929e2700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:37:22.276 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:22.264+0000 7f2d929e2700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:37:22.276 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:22.264+0000 7f2d929e2700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:37:22.276 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:22.264+0000 7f2d921e1700 10 monclient(hunting): get_auth_request con 0x7f2d8c062c70 auth_method 0 2023-09-30T01:37:22.276 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:22.264+0000 7f2d921e1700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:37:22.276 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:22.264+0000 7f2d921e1700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:37:22.276 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:22.264+0000 7f2d921e1700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:37:22.276 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:22.264+0000 7f2d919e0700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:37:22.277 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:22.264+0000 7f2d919e0700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:37:22.277 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:22.264+0000 7f2d919e0700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:37:22.277 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:22.264+0000 7f2d929e2700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:37:22.277 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:22.264+0000 7f2d929e2700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:37:22.277 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:22.264+0000 7f2d929e2700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:37:22.277 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:22.264+0000 7f2d921e1700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:37:22.277 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:22.264+0000 7f2d921e1700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:37:22.277 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:22.264+0000 7f2d921e1700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:37:22.277 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:22.264+0000 7f2d919e0700 10 monclient(hunting): handle_auth_done global_id 4801 payload 274 2023-09-30T01:37:22.277 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:22.264+0000 7f2d919e0700 1 -- 172.21.15.88:0/3831994791 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7f2d8c062c70 msgr2=0x7f2d8c09ac40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:37:22.277 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:22.264+0000 7f2d919e0700 1 --2- 172.21.15.88:0/3831994791 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7f2d8c062c70 0x7f2d8c09ac40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:37:22.277 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:22.264+0000 7f2d919e0700 1 -- 172.21.15.88:0/3831994791 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7f2d8c063580 msgr2=0x7f2d8c137660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:37:22.278 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:22.264+0000 7f2d919e0700 1 --2- 172.21.15.88:0/3831994791 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7f2d8c063580 0x7f2d8c137660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:37:22.278 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:22.264+0000 7f2d919e0700 10 monclient: _finish_hunting 0 2023-09-30T01:37:22.278 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:22.264+0000 7f2d919e0700 1 monclient: found mon.noname-c 2023-09-30T01:37:22.278 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:22.264+0000 7f2d919e0700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:37:22.278 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:22.264+0000 7f2d919e0700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.88:3302/0 2023-09-30T01:37:22.278 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:22.264+0000 7f2d919e0700 1 -- 172.21.15.88:0/3831994791 --> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d8c05ecb0 con 0x7f2d8c0988b0 2023-09-30T01:37:22.278 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:22.264+0000 7f2d929e2700 1 --2- 172.21.15.88:0/3831994791 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7f2d8c063580 0x7f2d8c137660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:37:22.278 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:22.264+0000 7f2d919e0700 1 --2- 172.21.15.88:0/3831994791 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f2d8c0988b0 0x7f2d8c063040 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f2d7c0099e0 tx=0x7f2d7c009250 comp rx=0 tx=0).ready entity=mon.4 client_cookie=6520e899928b78d9 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:37:22.278 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:22.264+0000 7f2d911df700 1 -- 172.21.15.88:0/3831994791 <== mon.4 v2:172.21.15.88:3302/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f2d7c00bb00 con 0x7f2d8c0988b0 2023-09-30T01:37:22.278 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:22.264+0000 7f2d911df700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:37:22.278 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:22.264+0000 7f2d911df700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T01:37:22.278 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:22.264+0000 7f2d911df700 10 monclient: dump: 2023-09-30T01:37:22.279 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:37:22.279 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:37:22.279 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:37:22.279 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:37:22.279 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:37:22.279 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:37:22.279 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:37:22.279 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:37:22.279 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:37:22.279 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:37:22.279 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:37:22.279 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:37:22.280 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:37:22.280 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:37:22.280 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:37:22.280 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:37:22.280 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:22.264+0000 7f2d911df700 10 monclient: _finish_auth 0 2023-09-30T01:37:22.280 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:22.264+0000 7f2d911df700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:37:22.280 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:22.264+0000 7f2d94445700 5 monclient: authenticate success, global_id 4801 2023-09-30T01:37:22.280 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:22.264+0000 7f2d94445700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:37:22.280 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:22.264+0000 7f2d911df700 1 -- 172.21.15.88:0/3831994791 <== mon.4 v2:172.21.15.88:3302/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d7c00bc60 con 0x7f2d8c0988b0 2023-09-30T01:37:22.280 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:22.264+0000 7f2d911df700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:37:22.280 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:22.264+0000 7f2d94445700 10 monclient: get_monmap_and_config success 2023-09-30T01:37:22.280 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:22.264+0000 7f2d94445700 4 set_mon_vals no callback set 2023-09-30T01:37:22.281 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:22.264+0000 7f2d911df700 1 -- 172.21.15.88:0/3831994791 <== mon.4 v2:172.21.15.88:3302/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f2d7c00bdd0 con 0x7f2d8c0988b0 2023-09-30T01:37:22.281 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:22.264+0000 7f2d911df700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:37:22.281 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:22.264+0000 7f2d911df700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T01:37:22.281 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:22.264+0000 7f2d911df700 10 monclient: dump: 2023-09-30T01:37:22.281 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:37:22.281 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:37:22.281 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:37:22.281 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:37:22.281 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:37:22.281 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:37:22.281 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:37:22.282 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:37:22.282 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:37:22.282 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:37:22.282 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:37:22.282 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:37:22.282 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:37:22.282 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:37:22.282 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:37:22.282 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:37:22.283 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:22.264+0000 7f2d94445700 10 monclient: shutdown 2023-09-30T01:37:22.283 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:22.264+0000 7f2d94445700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:37:22.283 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:22.264+0000 7f2d94445700 1 -- 172.21.15.88:0/3831994791 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f2d8c0988b0 msgr2=0x7f2d8c063040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:37:22.283 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:22.264+0000 7f2d94445700 1 --2- 172.21.15.88:0/3831994791 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f2d8c0988b0 0x7f2d8c063040 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f2d7c0099e0 tx=0x7f2d7c009250 comp rx=0 tx=0).stop 2023-09-30T01:37:22.283 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:22.264+0000 7f2d94445700 1 -- 172.21.15.88:0/3831994791 shutdown_connections 2023-09-30T01:37:22.283 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:22.264+0000 7f2d94445700 1 --2- 172.21.15.88:0/3831994791 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7f2d8c063580 0x7f2d8c137660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:37:22.283 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:22.264+0000 7f2d94445700 1 --2- 172.21.15.88:0/3831994791 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f2d8c0988b0 0x7f2d8c063040 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:37:22.283 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:22.264+0000 7f2d94445700 1 --2- 172.21.15.88:0/3831994791 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7f2d8c062c70 0x7f2d8c09ac40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:37:22.283 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:22.264+0000 7f2d94445700 1 -- 172.21.15.88:0/3831994791 >> 172.21.15.88:0/3831994791 conn(0x7f2d8c0a2660 msgr2=0x7f2d8c0a2a30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:37:22.283 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:22.264+0000 7f2d94445700 1 -- 172.21.15.88:0/3831994791 shutdown_connections 2023-09-30T01:37:22.283 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:22.264+0000 7f2d94445700 1 -- 172.21.15.88:0/3831994791 wait complete. 2023-09-30T01:37:22.283 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:22.268+0000 7f2d94445700 10 monclient: build_initial_monmap 2023-09-30T01:37:22.284 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:22.268+0000 7f2d94445700 10 monclient: monmap: 2023-09-30T01:37:22.284 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2023-09-30T01:37:22.284 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:37:22.284 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 0.000000 2023-09-30T01:37:22.284 INFO:teuthology.orchestra.run.smithi088.stderr:created 0.000000 2023-09-30T01:37:22.284 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2023-09-30T01:37:22.284 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 1 2023-09-30T01:37:22.284 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.noname-a 2023-09-30T01:37:22.284 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.noname-f 2023-09-30T01:37:22.284 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.noname-b 2023-09-30T01:37:22.284 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.noname-g 2023-09-30T01:37:22.284 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.noname-c 2023-09-30T01:37:22.284 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.noname-h 2023-09-30T01:37:22.285 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.noname-d 2023-09-30T01:37:22.285 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.noname-i 2023-09-30T01:37:22.285 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.noname-e 2023-09-30T01:37:22.285 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:37:22.285 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:22.268+0000 7f2d94445700 1 Processor -- start 2023-09-30T01:37:22.285 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:22.268+0000 7f2d94445700 1 -- start start 2023-09-30T01:37:22.285 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:22.268+0000 7f2d94445700 10 monclient: init 2023-09-30T01:37:22.290 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:22.268+0000 7f2d94445700 10 monclient: _reopen_session rank -1 2023-09-30T01:37:22.290 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:22.268+0000 7f2d94445700 10 monclient: _add_conns ranks=[3,7,4,6,1,8,0,2,5] 2023-09-30T01:37:22.290 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:22.268+0000 7f2d94445700 1 --2- >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7f2d8c062c70 0x7f2d8c12df00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:37:22.290 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:22.268+0000 7f2d94445700 10 monclient(hunting): picked mon.noname-g con 0x7f2d8c062c70 addr [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] 2023-09-30T01:37:22.290 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:22.268+0000 7f2d94445700 1 --2- >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7f2d8c063580 0x7f2d8c1422c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:37:22.290 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:22.268+0000 7f2d94445700 10 monclient(hunting): picked mon.noname-i con 0x7f2d8c063580 addr [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] 2023-09-30T01:37:22.290 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:22.268+0000 7f2d94445700 1 --2- >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f2d8c0988b0 0x7f2d8c142800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:37:22.290 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:22.268+0000 7f2d94445700 10 monclient(hunting): picked mon.noname-c con 0x7f2d8c0988b0 addr [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] 2023-09-30T01:37:22.290 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:22.268+0000 7f2d94445700 10 monclient(hunting): start opening mon connection 2023-09-30T01:37:22.291 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:22.268+0000 7f2d94445700 1 -- --> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d8c12e440 con 0x7f2d8c062c70 2023-09-30T01:37:22.291 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:22.268+0000 7f2d94445700 10 monclient(hunting): start opening mon connection 2023-09-30T01:37:22.291 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:22.268+0000 7f2d94445700 1 -- --> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] -- mon_getmap magic: 0 v1 -- 0x7f2d8c098fe0 con 0x7f2d8c0988b0 2023-09-30T01:37:22.291 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:22.268+0000 7f2d94445700 10 monclient(hunting): start opening mon connection 2023-09-30T01:37:22.291 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:22.268+0000 7f2d94445700 1 -- --> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] -- mon_getmap magic: 0 v1 -- 0x7f2d8c096130 con 0x7f2d8c063580 2023-09-30T01:37:22.291 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:22.268+0000 7f2d94445700 10 monclient(hunting): _renew_subs 2023-09-30T01:37:22.291 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:22.268+0000 7f2d94445700 10 monclient(hunting): authenticate will time out at 1114.310806s 2023-09-30T01:37:22.291 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:22.268+0000 7f2d929e2700 1 --2- >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f2d8c0988b0 0x7f2d8c142800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:37:22.291 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:22.268+0000 7f2d929e2700 1 --2- >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f2d8c0988b0 0x7f2d8c142800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3302/0 says I am v2:172.21.15.88:36730/0 (socket says 172.21.15.88:36730) 2023-09-30T01:37:22.291 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:22.268+0000 7f2d929e2700 1 -- 172.21.15.88:0/2511722821 learned_addr learned my addr 172.21.15.88:0/2511722821 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-09-30T01:37:22.291 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:22.268+0000 7f2d929e2700 10 monclient(hunting): get_auth_request con 0x7f2d8c0988b0 auth_method 0 2023-09-30T01:37:22.291 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:22.268+0000 7f2d929e2700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:37:22.292 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:22.268+0000 7f2d929e2700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:37:22.292 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:22.268+0000 7f2d929e2700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:37:22.292 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:22.268+0000 7f2d919e0700 1 --2- 172.21.15.88:0/2511722821 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7f2d8c063580 0x7f2d8c1422c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:37:22.292 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:22.268+0000 7f2d921e1700 1 --2- 172.21.15.88:0/2511722821 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7f2d8c062c70 0x7f2d8c12df00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:37:22.292 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:22.268+0000 7f2d919e0700 10 monclient(hunting): get_auth_request con 0x7f2d8c063580 auth_method 0 2023-09-30T01:37:22.292 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:22.268+0000 7f2d919e0700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:37:22.292 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:22.268+0000 7f2d919e0700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:37:22.292 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:22.268+0000 7f2d919e0700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:37:22.292 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:22.268+0000 7f2d921e1700 10 monclient(hunting): get_auth_request con 0x7f2d8c062c70 auth_method 0 2023-09-30T01:37:22.292 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:22.268+0000 7f2d921e1700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:37:22.292 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:22.268+0000 7f2d921e1700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:37:22.292 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:22.268+0000 7f2d921e1700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:37:22.293 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:22.268+0000 7f2d929e2700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:37:22.293 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:22.268+0000 7f2d929e2700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:37:22.293 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:22.268+0000 7f2d929e2700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:37:22.293 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:22.268+0000 7f2d919e0700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:37:22.293 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:22.268+0000 7f2d919e0700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:37:22.293 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:22.268+0000 7f2d919e0700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:37:22.293 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:22.268+0000 7f2d921e1700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:37:22.293 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:22.268+0000 7f2d921e1700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:37:22.293 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:22.268+0000 7f2d921e1700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:37:22.293 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:22.268+0000 7f2d929e2700 10 monclient(hunting): handle_auth_done global_id 4810 payload 824 2023-09-30T01:37:22.293 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:22.268+0000 7f2d929e2700 1 -- 172.21.15.88:0/2511722821 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7f2d8c063580 msgr2=0x7f2d8c1422c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:37:22.293 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:22.268+0000 7f2d929e2700 1 --2- 172.21.15.88:0/2511722821 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7f2d8c063580 0x7f2d8c1422c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:37:22.293 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:22.268+0000 7f2d929e2700 1 -- 172.21.15.88:0/2511722821 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7f2d8c062c70 msgr2=0x7f2d8c12df00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:37:22.294 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:22.268+0000 7f2d929e2700 1 --2- 172.21.15.88:0/2511722821 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7f2d8c062c70 0x7f2d8c12df00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:37:22.294 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:22.268+0000 7f2d929e2700 10 monclient: _finish_hunting 0 2023-09-30T01:37:22.294 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:22.268+0000 7f2d929e2700 1 monclient: found mon.noname-c 2023-09-30T01:37:22.294 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:22.268+0000 7f2d929e2700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:37:22.294 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:22.268+0000 7f2d929e2700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.88:3302/0 2023-09-30T01:37:22.294 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:22.268+0000 7f2d929e2700 1 -- 172.21.15.88:0/2511722821 --> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d84009a20 con 0x7f2d8c0988b0 2023-09-30T01:37:22.294 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:22.272+0000 7f2d929e2700 1 --2- 172.21.15.88:0/2511722821 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f2d8c0988b0 0x7f2d8c142800 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f2d84009f90 tx=0x7f2d8400b040 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:37:22.294 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:22.272+0000 7f2d827fc700 1 -- 172.21.15.88:0/2511722821 <== mon.4 v2:172.21.15.88:3302/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f2d8400cbd0 con 0x7f2d8c0988b0 2023-09-30T01:37:22.294 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:22.272+0000 7f2d827fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:37:22.294 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:22.272+0000 7f2d827fc700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T01:37:22.294 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:22.272+0000 7f2d827fc700 10 monclient: dump: 2023-09-30T01:37:22.295 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:37:22.295 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:37:22.295 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:37:22.295 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:37:22.295 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:37:22.295 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:37:22.295 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:37:22.295 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:37:22.295 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:37:22.295 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:37:22.295 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:37:22.295 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:37:22.295 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:37:22.296 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:37:22.296 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:37:22.296 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:37:22.296 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:22.272+0000 7f2d827fc700 10 monclient: _finish_auth 0 2023-09-30T01:37:22.296 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:22.272+0000 7f2d827fc700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:37:22.296 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:22.272+0000 7f2d827fc700 1 -- 172.21.15.88:0/2511722821 <== mon.4 v2:172.21.15.88:3302/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d8400f040 con 0x7f2d8c0988b0 2023-09-30T01:37:22.296 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:22.272+0000 7f2d94445700 5 monclient: authenticate success, global_id 4810 2023-09-30T01:37:22.296 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:22.272+0000 7f2d94445700 10 monclient: _renew_subs 2023-09-30T01:37:22.296 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:22.272+0000 7f2d94445700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.88:3302/0 2023-09-30T01:37:22.296 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:22.272+0000 7f2d94445700 1 -- 172.21.15.88:0/2511722821 --> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d7c0094f0 con 0x7f2d8c0988b0 2023-09-30T01:37:22.296 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:22.272+0000 7f2d827fc700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:37:22.296 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:22.272+0000 7f2d827fc700 1 -- 172.21.15.88:0/2511722821 <== mon.4 v2:172.21.15.88:3302/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f2d84014d80 con 0x7f2d8c0988b0 2023-09-30T01:37:22.297 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:22.272+0000 7f2d827fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:37:22.297 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:22.272+0000 7f2d827fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T01:37:22.297 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:22.272+0000 7f2d909de700 4 set_mon_vals no callback set 2023-09-30T01:37:22.297 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:22.272+0000 7f2d827fc700 10 monclient: dump: 2023-09-30T01:37:22.297 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:37:22.297 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:37:22.297 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:37:22.297 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:37:22.297 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:37:22.297 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:37:22.297 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:37:22.297 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:37:22.298 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:37:22.298 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:37:22.298 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:37:22.298 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:37:22.298 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:37:22.298 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:37:22.298 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:37:22.298 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:37:22.298 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:22.272+0000 7f2d94445700 10 monclient: _renew_subs 2023-09-30T01:37:22.298 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:22.272+0000 7f2d94445700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.88:3302/0 2023-09-30T01:37:22.298 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:22.272+0000 7f2d94445700 1 -- 172.21.15.88:0/2511722821 --> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d8c12cf10 con 0x7f2d8c0988b0 2023-09-30T01:37:22.298 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:22.272+0000 7f2d827fc700 1 -- 172.21.15.88:0/2511722821 <== mon.4 v2:172.21.15.88:3302/0 4 ==== mgrmap(e 5) v1 ==== 40144+0+0 (secure 0 0 0) 0x7f2d84014480 con 0x7f2d8c0988b0 2023-09-30T01:37:22.298 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:22.272+0000 7f2d94445700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:37:22.299 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:22.272+0000 7f2d94445700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:37:22.299 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:22.272+0000 7f2d94445700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.88:3302/0 2023-09-30T01:37:22.299 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:22.272+0000 7f2d94445700 1 -- 172.21.15.88:0/2511722821 --> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2d8c12d3a0 con 0x7f2d8c0988b0 2023-09-30T01:37:22.299 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:22.272+0000 7f2d827fc700 1 --2- 172.21.15.88:0/2511722821 >> v2:172.21.15.107:6812/29926 conn(0x7f2d70033360 0x7f2d70035810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:37:22.299 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:22.272+0000 7f2d827fc700 1 -- 172.21.15.88:0/2511722821 <== mon.4 v2:172.21.15.88:3302/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 3502+0+0 (secure 0 0 0) 0x7f2d84044190 con 0x7f2d8c0988b0 2023-09-30T01:37:22.299 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:22.272+0000 7f2d921e1700 1 --2- 172.21.15.88:0/2511722821 >> v2:172.21.15.107:6812/29926 conn(0x7f2d70033360 0x7f2d70035810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:37:22.299 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:22.272+0000 7f2d921e1700 10 monclient: get_auth_request con 0x7f2d70033360 auth_method 0 2023-09-30T01:37:22.299 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:22.272+0000 7f2d921e1700 1 --2- 172.21.15.88:0/2511722821 >> v2:172.21.15.107:6812/29926 conn(0x7f2d70033360 0x7f2d70035810 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f2d88009d00 tx=0x7f2d88009120 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:37:22.299 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:22.280+0000 7f2d827fc700 1 -- 172.21.15.88:0/2511722821 <== mon.4 v2:172.21.15.88:3302/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f2d8c12d3a0 con 0x7f2d8c0988b0 2023-09-30T01:37:22.299 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:22.280+0000 7f2d827fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:37:22.299 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:22.280+0000 7f2d827fc700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:37:22.483 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:22.476+0000 7f2d94445700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T01:37:22.483 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:22.476+0000 7f2d94445700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:37:22.483 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:22.476+0000 7f2d94445700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.88:3302/0 2023-09-30T01:37:22.483 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:22.476+0000 7f2d94445700 1 -- 172.21.15.88:0/2511722821 --> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f2d8c12d3a0 con 0x7f2d8c0988b0 2023-09-30T01:37:22.483 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:22.476+0000 7f2d827fc700 1 -- 172.21.15.88:0/2511722821 <== mon.4 v2:172.21.15.88:3302/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f2d8c12d3a0 con 0x7f2d8c0988b0 2023-09-30T01:37:22.483 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:22.476+0000 7f2d827fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:37:22.483 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:22.476+0000 7f2d827fc700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:37:22.484 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-09-30T01:37:22.484 INFO:teuthology.orchestra.run.smithi088.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:37:22.485 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:22.480+0000 7f2d94445700 1 -- 172.21.15.88:0/2511722821 >> v2:172.21.15.107:6812/29926 conn(0x7f2d70033360 msgr2=0x7f2d70035810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:37:22.486 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:22.480+0000 7f2d94445700 1 --2- 172.21.15.88:0/2511722821 >> v2:172.21.15.107:6812/29926 conn(0x7f2d70033360 0x7f2d70035810 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f2d88009d00 tx=0x7f2d88009120 comp rx=0 tx=0).stop 2023-09-30T01:37:22.486 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:22.480+0000 7f2d94445700 10 monclient: shutdown 2023-09-30T01:37:22.486 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:22.480+0000 7f2d94445700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:37:22.486 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:22.480+0000 7f2d94445700 1 -- 172.21.15.88:0/2511722821 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f2d8c0988b0 msgr2=0x7f2d8c142800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:37:22.486 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:22.480+0000 7f2d94445700 1 --2- 172.21.15.88:0/2511722821 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f2d8c0988b0 0x7f2d8c142800 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f2d84009f90 tx=0x7f2d8400b040 comp rx=0 tx=0).stop 2023-09-30T01:37:22.486 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:22.480+0000 7f2d94445700 1 -- 172.21.15.88:0/2511722821 shutdown_connections 2023-09-30T01:37:22.486 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:22.480+0000 7f2d94445700 1 --2- 172.21.15.88:0/2511722821 >> v2:172.21.15.107:6812/29926 conn(0x7f2d70033360 0x7f2d70035810 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:37:22.486 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:22.480+0000 7f2d94445700 1 --2- 172.21.15.88:0/2511722821 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f2d8c0988b0 0x7f2d8c142800 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:37:22.486 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:22.480+0000 7f2d94445700 1 --2- 172.21.15.88:0/2511722821 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7f2d8c063580 0x7f2d8c1422c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:37:22.486 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:22.480+0000 7f2d94445700 1 --2- 172.21.15.88:0/2511722821 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7f2d8c062c70 0x7f2d8c12df00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:37:22.487 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:22.480+0000 7f2d94445700 1 -- 172.21.15.88:0/2511722821 >> 172.21.15.88:0/2511722821 conn(0x7f2d8c0a2660 msgr2=0x7f2d8c133c60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:37:22.487 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:22.480+0000 7f2d94445700 1 -- 172.21.15.88:0/2511722821 shutdown_connections 2023-09-30T01:37:22.487 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:22.480+0000 7f2d94445700 1 -- 172.21.15.88:0/2511722821 wait complete. 2023-09-30T01:37:25.497 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:37:25.602 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:25.596+0000 7f9d7f59e700 10 monclient: get_monmap_and_config 2023-09-30T01:37:25.602 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:25.596+0000 7f9d7f59e700 10 monclient: build_initial_monmap 2023-09-30T01:37:25.602 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:25.596+0000 7f9d7f59e700 10 monclient: monmap: 2023-09-30T01:37:25.602 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2023-09-30T01:37:25.602 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:37:25.603 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:37:25.602165+0000 2023-09-30T01:37:25.603 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:37:25.602165+0000 2023-09-30T01:37:25.603 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2023-09-30T01:37:25.603 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:37:25.603 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a-legacy 2023-09-30T01:37:25.603 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.107:6789/0 mon.noname-f-legacy 2023-09-30T01:37:25.603 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.noname-b 2023-09-30T01:37:25.603 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.noname-g 2023-09-30T01:37:25.603 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.noname-c 2023-09-30T01:37:25.603 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.noname-h 2023-09-30T01:37:25.604 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.noname-d 2023-09-30T01:37:25.604 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.noname-i 2023-09-30T01:37:25.604 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.noname-e 2023-09-30T01:37:25.604 INFO:teuthology.orchestra.run.smithi088.stderr:9: v2:172.21.15.88:3300/0 mon.noname-a 2023-09-30T01:37:25.604 INFO:teuthology.orchestra.run.smithi088.stderr:10: v2:172.21.15.107:3300/0 mon.noname-f 2023-09-30T01:37:25.604 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:37:25.614 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:25.596+0000 7f9d7f59e700 1 Processor -- start 2023-09-30T01:37:25.614 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:25.596+0000 7f9d7f59e700 1 -- start start 2023-09-30T01:37:25.614 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:25.596+0000 7f9d7f59e700 10 monclient: init 2023-09-30T01:37:25.614 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:25.600+0000 7f9d7f59e700 10 monclient: _reopen_session rank -1 2023-09-30T01:37:25.614 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:25.600+0000 7f9d7f59e700 10 monclient: _add_conns ranks=[7,9,2,0,10,6,5,3,8,4,1] 2023-09-30T01:37:25.614 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:25.600+0000 7f9d7f59e700 1 --2- >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7f9d800976f0 0x7f9d80098de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:37:25.614 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:25.600+0000 7f9d7f59e700 10 monclient(hunting): picked mon.noname-i con 0x7f9d800976f0 addr [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] 2023-09-30T01:37:25.614 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:25.600+0000 7f9d7f59e700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f9d8009ed80 0x7f9d8009f150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:37:25.615 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:25.600+0000 7f9d7f59e700 10 monclient(hunting): picked mon.noname-a con 0x7f9d8009ed80 addr v2:172.21.15.88:3300/0 2023-09-30T01:37:25.615 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:25.600+0000 7f9d7f59e700 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f9d8009f690 0x7f9d80130570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:37:25.615 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:25.600+0000 7f9d7f59e700 10 monclient(hunting): picked mon.noname-b con 0x7f9d8009f690 addr [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] 2023-09-30T01:37:25.615 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:25.600+0000 7f9d7f59e700 10 monclient(hunting): start opening mon connection 2023-09-30T01:37:25.615 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:25.600+0000 7f9d7f59e700 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9d80099320 con 0x7f9d8009ed80 2023-09-30T01:37:25.615 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:25.600+0000 7f9d7f59e700 10 monclient(hunting): start opening mon connection 2023-09-30T01:37:25.615 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:25.600+0000 7f9d7f59e700 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d80098400 con 0x7f9d8009f690 2023-09-30T01:37:25.615 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:25.600+0000 7f9d7f59e700 10 monclient(hunting): start opening mon connection 2023-09-30T01:37:25.615 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:25.600+0000 7f9d7f59e700 1 -- --> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] -- mon_getmap magic: 0 v1 -- 0x7f9d80098a30 con 0x7f9d800976f0 2023-09-30T01:37:25.616 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:25.600+0000 7f9d7f59e700 10 monclient(hunting): _renew_subs 2023-09-30T01:37:25.616 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:25.600+0000 7f9d7f59e700 10 monclient(hunting): authenticate will time out at 1117.640474s 2023-09-30T01:37:25.616 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:25.600+0000 7f9d7ed9d700 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f9d8009f690 0x7f9d80130570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:37:25.616 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:25.600+0000 7f9d75d9b700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f9d8009ed80 0x7f9d8009f150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:37:25.616 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:25.600+0000 7f9d7ed9d700 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f9d8009f690 0x7f9d80130570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3301/0 says I am v2:172.21.15.88:34020/0 (socket says 172.21.15.88:34020) 2023-09-30T01:37:25.616 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:25.600+0000 7f9d7ed9d700 1 -- 172.21.15.88:0/1338394990 learned_addr learned my addr 172.21.15.88:0/1338394990 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-09-30T01:37:25.616 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:25.600+0000 7f9d7ed9d700 10 monclient(hunting): get_auth_request con 0x7f9d8009f690 auth_method 0 2023-09-30T01:37:25.616 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:25.600+0000 7f9d7ed9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:37:25.616 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:25.600+0000 7f9d7ed9d700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:37:25.616 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:25.600+0000 7f9d7ed9d700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:37:25.616 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:25.600+0000 7f9d75d9b700 10 monclient(hunting): get_auth_request con 0x7f9d8009ed80 auth_method 0 2023-09-30T01:37:25.616 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:25.600+0000 7f9d75d9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:37:25.617 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:25.600+0000 7f9d75d9b700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:37:25.617 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:25.600+0000 7f9d75d9b700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:37:25.617 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:25.600+0000 7f9d7e59c700 1 --2- 172.21.15.88:0/1338394990 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7f9d800976f0 0x7f9d80098de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:37:25.617 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:25.600+0000 7f9d7ed9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:37:25.617 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:25.600+0000 7f9d7ed9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:37:25.617 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:25.600+0000 7f9d7ed9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:37:25.617 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:25.600+0000 7f9d75d9b700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:37:25.617 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:25.600+0000 7f9d75d9b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:37:25.617 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:25.600+0000 7f9d75d9b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:37:25.617 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:25.600+0000 7f9d7e59c700 10 monclient(hunting): get_auth_request con 0x7f9d800976f0 auth_method 0 2023-09-30T01:37:25.617 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:25.600+0000 7f9d7e59c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:37:25.617 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:25.600+0000 7f9d7e59c700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:37:25.618 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:25.600+0000 7f9d7e59c700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:37:25.618 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:25.600+0000 7f9d7e59c700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:37:25.618 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:25.600+0000 7f9d7e59c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:37:25.618 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:25.600+0000 7f9d7e59c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:37:25.618 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:25.600+0000 7f9d75d9b700 10 monclient(hunting): handle_auth_done global_id 5085 payload 274 2023-09-30T01:37:25.618 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:25.600+0000 7f9d75d9b700 1 -- 172.21.15.88:0/1338394990 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7f9d800976f0 msgr2=0x7f9d80098de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:37:25.618 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:25.600+0000 7f9d75d9b700 1 --2- 172.21.15.88:0/1338394990 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7f9d800976f0 0x7f9d80098de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:37:25.618 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:25.600+0000 7f9d75d9b700 1 -- 172.21.15.88:0/1338394990 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f9d8009f690 msgr2=0x7f9d80130570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:37:25.618 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:25.600+0000 7f9d75d9b700 1 --2- 172.21.15.88:0/1338394990 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f9d8009f690 0x7f9d80130570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:37:25.618 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:25.600+0000 7f9d75d9b700 10 monclient: _finish_hunting 0 2023-09-30T01:37:25.618 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:25.600+0000 7f9d75d9b700 1 monclient: found mon.noname-a 2023-09-30T01:37:25.619 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:25.600+0000 7f9d75d9b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:37:25.619 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:25.600+0000 7f9d75d9b700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.88:3300/0 2023-09-30T01:37:25.619 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:25.600+0000 7f9d75d9b700 1 -- 172.21.15.88:0/1338394990 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d8004bb20 con 0x7f9d8009ed80 2023-09-30T01:37:25.619 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:25.600+0000 7f9d7ed9d700 1 --2- 172.21.15.88:0/1338394990 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f9d8009f690 0x7f9d80130570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:37:25.619 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:25.600+0000 7f9d75d9b700 1 --2- 172.21.15.88:0/1338394990 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9d8009ed80 0x7f9d8009f150 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f9d6800dd10 tx=0x7f9d6800d630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e9a461f0a7a622d0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:37:25.619 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:25.600+0000 7f9d7dd9b700 1 -- 172.21.15.88:0/1338394990 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f9d68012070 con 0x7f9d8009ed80 2023-09-30T01:37:25.619 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:25.600+0000 7f9d7dd9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:37:25.619 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:25.600+0000 7f9d7dd9b700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T01:37:25.619 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:25.600+0000 7f9d7dd9b700 10 monclient: dump: 2023-09-30T01:37:25.619 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:37:25.619 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:37:25.620 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:37:25.620 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:37:25.620 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:37:25.620 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:37:25.620 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:37:25.620 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:37:25.620 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:37:25.620 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:37:25.620 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:37:25.621 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:37:25.621 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:37:25.621 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:37:25.621 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:37:25.621 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:37:25.621 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:25.600+0000 7f9d7dd9b700 10 monclient: _finish_auth 0 2023-09-30T01:37:25.621 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:25.600+0000 7f9d7dd9b700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:37:25.621 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:25.600+0000 7f9d7dd9b700 1 -- 172.21.15.88:0/1338394990 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d6800de70 con 0x7f9d8009ed80 2023-09-30T01:37:25.621 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:25.600+0000 7f9d7f59e700 5 monclient: authenticate success, global_id 5085 2023-09-30T01:37:25.622 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:25.600+0000 7f9d7f59e700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:37:25.622 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:25.600+0000 7f9d7dd9b700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:37:25.622 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:25.600+0000 7f9d7f59e700 10 monclient: get_monmap_and_config success 2023-09-30T01:37:25.622 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:25.600+0000 7f9d7dd9b700 1 -- 172.21.15.88:0/1338394990 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f9d6800d2e0 con 0x7f9d8009ed80 2023-09-30T01:37:25.622 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:25.600+0000 7f9d7dd9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:37:25.622 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:25.600+0000 7f9d7f59e700 4 set_mon_vals no callback set 2023-09-30T01:37:25.622 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:25.600+0000 7f9d7dd9b700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:37:25.622 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:25.600+0000 7f9d7dd9b700 10 monclient: dump: 2023-09-30T01:37:25.622 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:37:25.622 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:37:25.622 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:37:25.623 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:37:25.623 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:37:25.623 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:37:25.623 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:37:25.623 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:37:25.623 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:37:25.623 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:37:25.623 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:37:25.623 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:37:25.623 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:37:25.624 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:37:25.624 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:37:25.624 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:37:25.624 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:25.600+0000 7f9d7f59e700 10 monclient: shutdown 2023-09-30T01:37:25.624 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:25.600+0000 7f9d7f59e700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:37:25.624 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:25.600+0000 7f9d7f59e700 1 -- 172.21.15.88:0/1338394990 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9d8009ed80 msgr2=0x7f9d8009f150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:37:25.624 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:25.600+0000 7f9d7f59e700 1 --2- 172.21.15.88:0/1338394990 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9d8009ed80 0x7f9d8009f150 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f9d6800dd10 tx=0x7f9d6800d630 comp rx=0 tx=0).stop 2023-09-30T01:37:25.624 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:25.600+0000 7f9d7f59e700 1 -- 172.21.15.88:0/1338394990 shutdown_connections 2023-09-30T01:37:25.624 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:25.600+0000 7f9d7f59e700 1 --2- 172.21.15.88:0/1338394990 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f9d8009f690 0x7f9d80130570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:37:25.624 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:25.600+0000 7f9d7f59e700 1 --2- 172.21.15.88:0/1338394990 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9d8009ed80 0x7f9d8009f150 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:37:25.624 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:25.600+0000 7f9d7f59e700 1 --2- 172.21.15.88:0/1338394990 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7f9d800976f0 0x7f9d80098de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:37:25.624 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:25.600+0000 7f9d7f59e700 1 -- 172.21.15.88:0/1338394990 >> 172.21.15.88:0/1338394990 conn(0x7f9d800a0910 msgr2=0x7f9d800a33c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:37:25.625 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:25.600+0000 7f9d7f59e700 1 -- 172.21.15.88:0/1338394990 shutdown_connections 2023-09-30T01:37:25.625 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:25.600+0000 7f9d7f59e700 1 -- 172.21.15.88:0/1338394990 wait complete. 2023-09-30T01:37:25.625 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:25.604+0000 7f9d7f59e700 10 monclient: build_initial_monmap 2023-09-30T01:37:25.625 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:25.604+0000 7f9d7f59e700 10 monclient: monmap: 2023-09-30T01:37:25.625 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2023-09-30T01:37:25.625 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:37:25.625 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 0.000000 2023-09-30T01:37:25.625 INFO:teuthology.orchestra.run.smithi088.stderr:created 0.000000 2023-09-30T01:37:25.625 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2023-09-30T01:37:25.625 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 1 2023-09-30T01:37:25.625 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.noname-a 2023-09-30T01:37:25.625 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.noname-f 2023-09-30T01:37:25.626 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.noname-b 2023-09-30T01:37:25.626 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.noname-g 2023-09-30T01:37:25.626 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.noname-c 2023-09-30T01:37:25.626 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.noname-h 2023-09-30T01:37:25.626 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.noname-d 2023-09-30T01:37:25.626 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.noname-i 2023-09-30T01:37:25.626 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.noname-e 2023-09-30T01:37:25.626 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:37:25.626 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:25.604+0000 7f9d7f59e700 1 Processor -- start 2023-09-30T01:37:25.626 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:25.604+0000 7f9d7f59e700 1 -- start start 2023-09-30T01:37:25.627 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:25.604+0000 7f9d7f59e700 10 monclient: init 2023-09-30T01:37:25.627 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:25.604+0000 7f9d7f59e700 10 monclient: _reopen_session rank -1 2023-09-30T01:37:25.627 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:25.604+0000 7f9d7f59e700 10 monclient: _add_conns ranks=[1,2,0,6,3,5,8,4,7] 2023-09-30T01:37:25.627 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:25.604+0000 7f9d7f59e700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9d800976f0 0x7f9d80096130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:37:25.627 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:25.604+0000 7f9d7f59e700 10 monclient(hunting): picked mon.noname-f con 0x7f9d800976f0 addr [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] 2023-09-30T01:37:25.627 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:25.604+0000 7f9d7f59e700 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f9d8009ed80 0x7f9d80096670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:37:25.627 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:25.604+0000 7f9d7f59e700 10 monclient(hunting): picked mon.noname-b con 0x7f9d8009ed80 addr [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] 2023-09-30T01:37:25.627 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:25.604+0000 7f9d7f59e700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9d8009f690 0x7f9d80096bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:37:25.627 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:25.604+0000 7f9d7f59e700 10 monclient(hunting): picked mon.noname-a con 0x7f9d8009f690 addr [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] 2023-09-30T01:37:25.627 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:25.604+0000 7f9d7f59e700 10 monclient(hunting): start opening mon connection 2023-09-30T01:37:25.627 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:25.604+0000 7f9d7f59e700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d8012aaa0 con 0x7f9d8009f690 2023-09-30T01:37:25.627 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:25.604+0000 7f9d7f59e700 10 monclient(hunting): start opening mon connection 2023-09-30T01:37:25.628 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:25.604+0000 7f9d7f59e700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d80129650 con 0x7f9d800976f0 2023-09-30T01:37:25.628 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:25.604+0000 7f9d7f59e700 10 monclient(hunting): start opening mon connection 2023-09-30T01:37:25.628 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:25.604+0000 7f9d7f59e700 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d8012a7c0 con 0x7f9d8009ed80 2023-09-30T01:37:25.628 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:25.604+0000 7f9d7f59e700 10 monclient(hunting): _renew_subs 2023-09-30T01:37:25.628 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:25.604+0000 7f9d7f59e700 10 monclient(hunting): authenticate will time out at 1117.646570s 2023-09-30T01:37:25.628 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:25.604+0000 7f9d75d9b700 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f9d8009ed80 0x7f9d80096670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:37:25.628 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:25.604+0000 7f9d7ed9d700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9d8009f690 0x7f9d80096bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:37:25.628 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:25.604+0000 7f9d7ed9d700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9d8009f690 0x7f9d80096bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:51096/0 (socket says 172.21.15.88:51096) 2023-09-30T01:37:25.628 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:25.604+0000 7f9d7ed9d700 1 -- 172.21.15.88:0/412283420 learned_addr learned my addr 172.21.15.88:0/412283420 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-09-30T01:37:25.628 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:25.604+0000 7f9d75d9b700 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f9d8009ed80 0x7f9d80096670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3301/0 says I am v2:172.21.15.88:34034/0 (socket says 172.21.15.88:34034) 2023-09-30T01:37:25.628 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:25.604+0000 7f9d7ed9d700 10 monclient(hunting): get_auth_request con 0x7f9d8009f690 auth_method 0 2023-09-30T01:37:25.628 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:25.604+0000 7f9d7ed9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:37:25.629 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:25.604+0000 7f9d7ed9d700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:37:25.629 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:25.604+0000 7f9d7ed9d700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:37:25.629 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:25.604+0000 7f9d75d9b700 10 monclient(hunting): get_auth_request con 0x7f9d8009ed80 auth_method 0 2023-09-30T01:37:25.629 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:25.604+0000 7f9d75d9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:37:25.629 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:25.604+0000 7f9d75d9b700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:37:25.629 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:25.604+0000 7f9d75d9b700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:37:25.629 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:25.604+0000 7f9d7e59c700 1 --2- 172.21.15.88:0/412283420 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9d800976f0 0x7f9d80096130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:37:25.629 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:25.604+0000 7f9d7ed9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:37:25.629 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:25.604+0000 7f9d7ed9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:37:25.629 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:25.604+0000 7f9d7ed9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:37:25.629 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:25.604+0000 7f9d7e59c700 10 monclient(hunting): get_auth_request con 0x7f9d800976f0 auth_method 0 2023-09-30T01:37:25.629 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:25.604+0000 7f9d7e59c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:37:25.630 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:25.604+0000 7f9d7e59c700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:37:25.630 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:25.604+0000 7f9d7e59c700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:37:25.630 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:25.604+0000 7f9d75d9b700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:37:25.630 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:25.604+0000 7f9d75d9b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:37:25.630 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:25.604+0000 7f9d75d9b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:37:25.630 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:25.604+0000 7f9d7e59c700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:37:25.630 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:25.604+0000 7f9d7e59c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:37:25.630 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:25.604+0000 7f9d7e59c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:37:25.630 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:25.604+0000 7f9d75d9b700 10 monclient(hunting): handle_auth_done global_id 4718 payload 824 2023-09-30T01:37:25.630 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:25.604+0000 7f9d75d9b700 1 -- 172.21.15.88:0/412283420 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9d800976f0 msgr2=0x7f9d80096130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:37:25.630 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:25.604+0000 7f9d75d9b700 1 --2- 172.21.15.88:0/412283420 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9d800976f0 0x7f9d80096130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:37:25.630 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:25.604+0000 7f9d75d9b700 1 -- 172.21.15.88:0/412283420 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9d8009f690 msgr2=0x7f9d80096bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:37:25.631 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:25.604+0000 7f9d75d9b700 1 --2- 172.21.15.88:0/412283420 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9d8009f690 0x7f9d80096bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:37:25.631 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:25.604+0000 7f9d75d9b700 10 monclient: _finish_hunting 0 2023-09-30T01:37:25.631 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:25.604+0000 7f9d75d9b700 1 monclient: found mon.noname-b 2023-09-30T01:37:25.631 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:25.604+0000 7f9d75d9b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:37:25.631 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:25.604+0000 7f9d75d9b700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.88:3301/0 2023-09-30T01:37:25.631 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:25.604+0000 7f9d75d9b700 1 -- 172.21.15.88:0/412283420 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d7800bb00 con 0x7f9d8009ed80 2023-09-30T01:37:25.631 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:25.604+0000 7f9d7ed9d700 1 --2- 172.21.15.88:0/412283420 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9d8009f690 0x7f9d80096bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:37:25.631 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:25.604+0000 7f9d75d9b700 1 --2- 172.21.15.88:0/412283420 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f9d8009ed80 0x7f9d80096670 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f9d6800dd10 tx=0x7f9d6801c000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:37:25.631 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:25.608+0000 7f9d76ffd700 1 -- 172.21.15.88:0/412283420 <== mon.2 v2:172.21.15.88:3301/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f9d6800ccc0 con 0x7f9d8009ed80 2023-09-30T01:37:25.631 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:25.608+0000 7f9d76ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:37:25.631 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:25.608+0000 7f9d76ffd700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T01:37:25.631 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:25.608+0000 7f9d76ffd700 10 monclient: dump: 2023-09-30T01:37:25.632 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:37:25.632 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:37:25.632 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:37:25.632 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:37:25.632 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:37:25.632 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:37:25.632 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:37:25.632 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:37:25.632 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:37:25.632 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:37:25.632 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:37:25.633 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:37:25.633 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:37:25.633 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:37:25.633 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:37:25.633 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:37:25.633 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:25.608+0000 7f9d76ffd700 10 monclient: _finish_auth 0 2023-09-30T01:37:25.633 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:25.608+0000 7f9d76ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:37:25.633 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:25.608+0000 7f9d76ffd700 1 -- 172.21.15.88:0/412283420 <== mon.2 v2:172.21.15.88:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d6800ce60 con 0x7f9d8009ed80 2023-09-30T01:37:25.633 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:25.608+0000 7f9d7f59e700 5 monclient: authenticate success, global_id 4718 2023-09-30T01:37:25.633 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:25.608+0000 7f9d7f59e700 10 monclient: _renew_subs 2023-09-30T01:37:25.633 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:25.608+0000 7f9d7f59e700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.88:3301/0 2023-09-30T01:37:25.633 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:25.608+0000 7f9d7f59e700 1 -- 172.21.15.88:0/412283420 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d6800d8d0 con 0x7f9d8009ed80 2023-09-30T01:37:25.634 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:25.608+0000 7f9d76ffd700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:37:25.634 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:25.608+0000 7f9d76ffd700 1 -- 172.21.15.88:0/412283420 <== mon.2 v2:172.21.15.88:3301/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f9d68020890 con 0x7f9d8009ed80 2023-09-30T01:37:25.634 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:25.608+0000 7f9d76ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:37:25.634 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:25.608+0000 7f9d76ffd700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T01:37:25.634 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:25.608+0000 7f9d76ffd700 10 monclient: dump: 2023-09-30T01:37:25.634 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:37:25.634 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:37:25.634 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:37:25.634 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:37:25.634 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:37:25.634 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:37:25.634 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:37:25.635 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:37:25.635 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:37:25.635 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:37:25.635 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:37:25.635 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:37:25.635 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:37:25.635 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:37:25.635 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:37:25.635 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:37:25.635 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:25.608+0000 7f9d7f59e700 10 monclient: _renew_subs 2023-09-30T01:37:25.635 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:25.608+0000 7f9d7f59e700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.88:3301/0 2023-09-30T01:37:25.635 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:25.608+0000 7f9d7f59e700 1 -- 172.21.15.88:0/412283420 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d80097d80 con 0x7f9d8009ed80 2023-09-30T01:37:25.636 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:25.608+0000 7f9d7d59a700 4 set_mon_vals no callback set 2023-09-30T01:37:25.636 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:25.608+0000 7f9d7f59e700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:37:25.636 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:25.608+0000 7f9d7f59e700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:37:25.636 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:25.608+0000 7f9d7f59e700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.88:3301/0 2023-09-30T01:37:25.636 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:25.608+0000 7f9d7f59e700 1 -- 172.21.15.88:0/412283420 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9d8009f4f0 con 0x7f9d8009ed80 2023-09-30T01:37:25.636 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:25.608+0000 7f9d76ffd700 1 -- 172.21.15.88:0/412283420 <== mon.2 v2:172.21.15.88:3301/0 4 ==== mgrmap(e 5) v1 ==== 40144+0+0 (secure 0 0 0) 0x7f9d68020410 con 0x7f9d8009ed80 2023-09-30T01:37:25.636 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:25.608+0000 7f9d76ffd700 1 --2- 172.21.15.88:0/412283420 >> v2:172.21.15.107:6812/29926 conn(0x7f9d64033360 0x7f9d64035810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:37:25.636 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:25.608+0000 7f9d76ffd700 1 -- 172.21.15.88:0/412283420 <== mon.2 v2:172.21.15.88:3301/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 3502+0+0 (secure 0 0 0) 0x7f9d680484b0 con 0x7f9d8009ed80 2023-09-30T01:37:25.636 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:25.608+0000 7f9d7e59c700 1 --2- 172.21.15.88:0/412283420 >> v2:172.21.15.107:6812/29926 conn(0x7f9d64033360 0x7f9d64035810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:37:25.636 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:25.608+0000 7f9d7e59c700 10 monclient: get_auth_request con 0x7f9d64033360 auth_method 0 2023-09-30T01:37:25.637 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:25.608+0000 7f9d7e59c700 1 --2- 172.21.15.88:0/412283420 >> v2:172.21.15.107:6812/29926 conn(0x7f9d64033360 0x7f9d64035810 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f9d70009da0 tx=0x7f9d70009120 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:37:25.637 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:25.616+0000 7f9d76ffd700 1 -- 172.21.15.88:0/412283420 <== mon.2 v2:172.21.15.88:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f9d8009f4f0 con 0x7f9d8009ed80 2023-09-30T01:37:25.637 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:25.616+0000 7f9d76ffd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:37:25.637 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:25.616+0000 7f9d76ffd700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:37:25.812 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:25.808+0000 7f9d7f59e700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T01:37:25.812 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:25.808+0000 7f9d7f59e700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:37:25.812 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:25.808+0000 7f9d7f59e700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.88:3301/0 2023-09-30T01:37:25.812 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:25.808+0000 7f9d7f59e700 1 -- 172.21.15.88:0/412283420 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f9d8009f4f0 con 0x7f9d8009ed80 2023-09-30T01:37:25.813 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:25.808+0000 7f9d76ffd700 1 -- 172.21.15.88:0/412283420 <== mon.2 v2:172.21.15.88:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f9d8009f4f0 con 0x7f9d8009ed80 2023-09-30T01:37:25.813 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:25.808+0000 7f9d76ffd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:37:25.813 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:25.808+0000 7f9d76ffd700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:37:25.813 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-09-30T01:37:25.813 INFO:teuthology.orchestra.run.smithi088.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:37:25.815 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:25.808+0000 7f9d7f59e700 1 -- 172.21.15.88:0/412283420 >> v2:172.21.15.107:6812/29926 conn(0x7f9d64033360 msgr2=0x7f9d64035810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:37:25.815 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:25.808+0000 7f9d7f59e700 1 --2- 172.21.15.88:0/412283420 >> v2:172.21.15.107:6812/29926 conn(0x7f9d64033360 0x7f9d64035810 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f9d70009da0 tx=0x7f9d70009120 comp rx=0 tx=0).stop 2023-09-30T01:37:25.815 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:25.808+0000 7f9d7f59e700 10 monclient: shutdown 2023-09-30T01:37:25.816 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:25.808+0000 7f9d7f59e700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:37:25.816 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:25.808+0000 7f9d7f59e700 1 -- 172.21.15.88:0/412283420 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f9d8009ed80 msgr2=0x7f9d80096670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:37:25.816 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:25.808+0000 7f9d7f59e700 1 --2- 172.21.15.88:0/412283420 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f9d8009ed80 0x7f9d80096670 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f9d6800dd10 tx=0x7f9d6801c000 comp rx=0 tx=0).stop 2023-09-30T01:37:25.816 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:25.808+0000 7f9d7f59e700 1 -- 172.21.15.88:0/412283420 shutdown_connections 2023-09-30T01:37:25.816 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:25.808+0000 7f9d7f59e700 1 --2- 172.21.15.88:0/412283420 >> v2:172.21.15.107:6812/29926 conn(0x7f9d64033360 0x7f9d64035810 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:37:25.816 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:25.808+0000 7f9d7f59e700 1 --2- 172.21.15.88:0/412283420 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9d8009f690 0x7f9d80096bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:37:25.816 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:25.808+0000 7f9d7f59e700 1 --2- 172.21.15.88:0/412283420 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f9d8009ed80 0x7f9d80096670 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:37:25.816 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:25.808+0000 7f9d7f59e700 1 --2- 172.21.15.88:0/412283420 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9d800976f0 0x7f9d80096130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:37:25.816 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:25.808+0000 7f9d7f59e700 1 -- 172.21.15.88:0/412283420 >> 172.21.15.88:0/412283420 conn(0x7f9d800a0910 msgr2=0x7f9d80131b80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:37:25.816 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:25.808+0000 7f9d7f59e700 1 -- 172.21.15.88:0/412283420 shutdown_connections 2023-09-30T01:37:25.816 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:25.808+0000 7f9d7f59e700 1 -- 172.21.15.88:0/412283420 wait complete. 2023-09-30T01:37:28.826 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:37:28.937 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:28.932+0000 7f3530715700 10 monclient: get_monmap_and_config 2023-09-30T01:37:28.937 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:28.932+0000 7f3530715700 10 monclient: build_initial_monmap 2023-09-30T01:37:28.937 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:28.932+0000 7f3530715700 10 monclient: monmap: 2023-09-30T01:37:28.937 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2023-09-30T01:37:28.937 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:37:28.937 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:37:28.936885+0000 2023-09-30T01:37:28.937 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:37:28.936885+0000 2023-09-30T01:37:28.937 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2023-09-30T01:37:28.937 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:37:28.938 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a-legacy 2023-09-30T01:37:28.938 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.107:6789/0 mon.noname-f-legacy 2023-09-30T01:37:28.938 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.noname-b 2023-09-30T01:37:28.938 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.noname-g 2023-09-30T01:37:28.938 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.noname-c 2023-09-30T01:37:28.938 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.noname-h 2023-09-30T01:37:28.938 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.noname-d 2023-09-30T01:37:28.938 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.noname-i 2023-09-30T01:37:28.938 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.noname-e 2023-09-30T01:37:28.938 INFO:teuthology.orchestra.run.smithi088.stderr:9: v2:172.21.15.88:3300/0 mon.noname-a 2023-09-30T01:37:28.938 INFO:teuthology.orchestra.run.smithi088.stderr:10: v2:172.21.15.107:3300/0 mon.noname-f 2023-09-30T01:37:28.939 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:37:28.942 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:28.932+0000 7f3530715700 1 Processor -- start 2023-09-30T01:37:28.942 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:28.932+0000 7f3530715700 1 -- start start 2023-09-30T01:37:28.942 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:28.932+0000 7f3530715700 10 monclient: init 2023-09-30T01:37:28.942 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:28.932+0000 7f3530715700 10 monclient: _reopen_session rank -1 2023-09-30T01:37:28.942 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:28.932+0000 7f3530715700 10 monclient: _add_conns ranks=[8,10,5,0,2,3,6,9,7,1,4] 2023-09-30T01:37:28.942 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:28.932+0000 7f3530715700 1 --2- >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f35280a3270 0x7f35280a0160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:37:28.942 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:28.932+0000 7f3530715700 10 monclient(hunting): picked mon.noname-e con 0x7f35280a3270 addr [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] 2023-09-30T01:37:28.943 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:28.932+0000 7f3530715700 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f35280a4b00 0x7f35280a06a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:37:28.943 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:28.932+0000 7f3530715700 10 monclient(hunting): picked mon.noname-f con 0x7f35280a4b00 addr v2:172.21.15.107:3300/0 2023-09-30T01:37:28.943 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:28.932+0000 7f3530715700 1 --2- >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7f35281333c0 0x7f35281337d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:37:28.943 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:28.932+0000 7f3530715700 10 monclient(hunting): picked mon.noname-h con 0x7f35281333c0 addr [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] 2023-09-30T01:37:28.943 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:28.932+0000 7f3530715700 10 monclient(hunting): start opening mon connection 2023-09-30T01:37:28.943 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:28.932+0000 7f3530715700 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f35280a4ed0 con 0x7f35280a4b00 2023-09-30T01:37:28.943 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:28.932+0000 7f3530715700 10 monclient(hunting): start opening mon connection 2023-09-30T01:37:28.943 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:28.932+0000 7f3530715700 1 -- --> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] -- mon_getmap magic: 0 v1 -- 0x7f35280a6330 con 0x7f35281333c0 2023-09-30T01:37:28.943 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:28.932+0000 7f3530715700 10 monclient(hunting): start opening mon connection 2023-09-30T01:37:28.943 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:28.932+0000 7f3530715700 1 -- --> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] -- mon_getmap magic: 0 v1 -- 0x7f3528098610 con 0x7f35280a3270 2023-09-30T01:37:28.943 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:28.932+0000 7f3530715700 10 monclient(hunting): _renew_subs 2023-09-30T01:37:28.943 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:28.932+0000 7f3530715700 10 monclient(hunting): authenticate will time out at 1120.974319s 2023-09-30T01:37:28.944 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:28.932+0000 7f352e4b1700 1 --2- >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f35280a3270 0x7f35280a0160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:37:28.944 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:28.932+0000 7f352ecb2700 1 --2- >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7f35281333c0 0x7f35281337d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:37:28.944 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:28.932+0000 7f352e4b1700 1 --2- >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f35280a3270 0x7f35280a0160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3304/0 says I am v2:172.21.15.88:39030/0 (socket says 172.21.15.88:39030) 2023-09-30T01:37:28.944 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:28.932+0000 7f352e4b1700 1 -- 172.21.15.88:0/2793414374 learned_addr learned my addr 172.21.15.88:0/2793414374 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-09-30T01:37:28.944 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:28.932+0000 7f352e4b1700 10 monclient(hunting): get_auth_request con 0x7f35280a3270 auth_method 0 2023-09-30T01:37:28.944 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:28.932+0000 7f352e4b1700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:37:28.944 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:28.932+0000 7f352e4b1700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:37:28.944 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:28.932+0000 7f352e4b1700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:37:28.944 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:28.932+0000 7f352dcb0700 1 --2- 172.21.15.88:0/2793414374 >> v2:172.21.15.107:3300/0 conn(0x7f35280a4b00 0x7f35280a06a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:37:28.944 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:28.932+0000 7f352ecb2700 10 monclient(hunting): get_auth_request con 0x7f35281333c0 auth_method 0 2023-09-30T01:37:28.944 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:28.932+0000 7f352ecb2700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:37:28.944 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:28.932+0000 7f352ecb2700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:37:28.945 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:28.932+0000 7f352ecb2700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:37:28.945 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:28.932+0000 7f352dcb0700 10 monclient(hunting): get_auth_request con 0x7f35280a4b00 auth_method 0 2023-09-30T01:37:28.945 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:28.932+0000 7f352dcb0700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:37:28.945 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:28.932+0000 7f352dcb0700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:37:28.945 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:28.932+0000 7f352dcb0700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:37:28.945 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:28.932+0000 7f352e4b1700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:37:28.945 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:28.932+0000 7f352e4b1700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:37:28.945 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:28.932+0000 7f352e4b1700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:37:28.945 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:28.932+0000 7f352ecb2700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:37:28.945 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:28.932+0000 7f352ecb2700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:37:28.945 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:28.932+0000 7f352ecb2700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:37:28.945 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:28.932+0000 7f352dcb0700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:37:28.945 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:28.932+0000 7f352dcb0700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:37:28.946 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:28.932+0000 7f352dcb0700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:37:28.946 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:28.932+0000 7f352e4b1700 10 monclient(hunting): handle_auth_done global_id 4796 payload 274 2023-09-30T01:37:28.946 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:28.932+0000 7f352e4b1700 1 -- 172.21.15.88:0/2793414374 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7f35281333c0 msgr2=0x7f35281337d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:37:28.946 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:28.932+0000 7f352e4b1700 1 --2- 172.21.15.88:0/2793414374 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7f35281333c0 0x7f35281337d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:37:28.946 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:28.932+0000 7f352e4b1700 1 -- 172.21.15.88:0/2793414374 >> v2:172.21.15.107:3300/0 conn(0x7f35280a4b00 msgr2=0x7f35280a06a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:37:28.946 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:28.932+0000 7f352e4b1700 1 --2- 172.21.15.88:0/2793414374 >> v2:172.21.15.107:3300/0 conn(0x7f35280a4b00 0x7f35280a06a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:37:28.946 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:28.932+0000 7f352e4b1700 10 monclient: _finish_hunting 0 2023-09-30T01:37:28.946 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:28.932+0000 7f352e4b1700 1 monclient: found mon.noname-e 2023-09-30T01:37:28.946 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:28.932+0000 7f352e4b1700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:37:28.946 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:28.932+0000 7f352e4b1700 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.88:3304/0 2023-09-30T01:37:28.946 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:28.932+0000 7f352e4b1700 1 -- 172.21.15.88:0/2793414374 --> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f352805ecb0 con 0x7f35280a3270 2023-09-30T01:37:28.946 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:28.932+0000 7f352e4b1700 1 --2- 172.21.15.88:0/2793414374 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f35280a3270 0x7f35280a0160 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f35240099b0 tx=0x7f3524009250 comp rx=0 tx=0).ready entity=mon.8 client_cookie=2689409c5fbd2beb server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:37:28.947 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:28.932+0000 7f352d4af700 1 -- 172.21.15.88:0/2793414374 <== mon.8 v2:172.21.15.88:3304/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f352400bb00 con 0x7f35280a3270 2023-09-30T01:37:28.947 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:28.936+0000 7f352d4af700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:37:28.947 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:28.936+0000 7f352d4af700 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2023-09-30T01:37:28.947 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:28.936+0000 7f352d4af700 10 monclient: dump: 2023-09-30T01:37:28.947 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:37:28.947 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:37:28.947 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:37:28.947 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:37:28.947 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:37:28.947 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:37:28.947 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:37:28.947 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:37:28.948 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:37:28.948 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:37:28.948 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:37:28.948 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:37:28.948 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:37:28.948 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:37:28.948 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:37:28.948 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:37:28.948 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:28.936+0000 7f352d4af700 10 monclient: _finish_auth 0 2023-09-30T01:37:28.948 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:28.936+0000 7f352d4af700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:37:28.949 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:28.936+0000 7f3530715700 5 monclient: authenticate success, global_id 4796 2023-09-30T01:37:28.949 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:28.936+0000 7f3530715700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:37:28.949 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:28.936+0000 7f352d4af700 1 -- 172.21.15.88:0/2793414374 <== mon.8 v2:172.21.15.88:3304/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f352400bc60 con 0x7f35280a3270 2023-09-30T01:37:28.949 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:28.936+0000 7f352d4af700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:37:28.949 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:28.936+0000 7f3530715700 10 monclient: get_monmap_and_config success 2023-09-30T01:37:28.949 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:28.936+0000 7f3530715700 4 set_mon_vals no callback set 2023-09-30T01:37:28.949 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:28.936+0000 7f352d4af700 1 -- 172.21.15.88:0/2793414374 <== mon.8 v2:172.21.15.88:3304/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f352400bdd0 con 0x7f35280a3270 2023-09-30T01:37:28.949 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:28.936+0000 7f352d4af700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:37:28.949 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:28.936+0000 7f352d4af700 10 monclient: got monmap 1 from mon.e (according to old e1) 2023-09-30T01:37:28.949 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:28.936+0000 7f352d4af700 10 monclient: dump: 2023-09-30T01:37:28.949 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:37:28.950 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:37:28.950 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:37:28.950 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:37:28.950 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:37:28.950 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:37:28.950 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:37:28.950 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:37:28.950 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:37:28.950 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:37:28.950 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:37:28.950 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:37:28.950 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:37:28.951 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:37:28.951 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:37:28.951 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:37:28.951 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:28.936+0000 7f3530715700 10 monclient: shutdown 2023-09-30T01:37:28.951 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:28.936+0000 7f3530715700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:37:28.951 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:28.936+0000 7f3530715700 1 -- 172.21.15.88:0/2793414374 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f35280a3270 msgr2=0x7f35280a0160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:37:28.951 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:28.936+0000 7f3530715700 1 --2- 172.21.15.88:0/2793414374 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f35280a3270 0x7f35280a0160 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f35240099b0 tx=0x7f3524009250 comp rx=0 tx=0).stop 2023-09-30T01:37:28.951 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:28.936+0000 7f3530715700 1 -- 172.21.15.88:0/2793414374 shutdown_connections 2023-09-30T01:37:28.951 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:28.936+0000 7f3530715700 1 --2- 172.21.15.88:0/2793414374 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7f35281333c0 0x7f35281337d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:37:28.951 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:28.936+0000 7f3530715700 1 --2- 172.21.15.88:0/2793414374 >> v2:172.21.15.107:3300/0 conn(0x7f35280a4b00 0x7f35280a06a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:37:28.951 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:28.936+0000 7f3530715700 1 --2- 172.21.15.88:0/2793414374 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f35280a3270 0x7f35280a0160 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:37:28.952 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:28.936+0000 7f3530715700 1 -- 172.21.15.88:0/2793414374 >> 172.21.15.88:0/2793414374 conn(0x7f35280a67f0 msgr2=0x7f35280a6bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:37:28.952 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:28.936+0000 7f3530715700 1 -- 172.21.15.88:0/2793414374 shutdown_connections 2023-09-30T01:37:28.952 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:28.936+0000 7f3530715700 1 -- 172.21.15.88:0/2793414374 wait complete. 2023-09-30T01:37:28.952 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:28.936+0000 7f3530715700 10 monclient: build_initial_monmap 2023-09-30T01:37:28.952 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:28.936+0000 7f3530715700 10 monclient: monmap: 2023-09-30T01:37:28.952 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2023-09-30T01:37:28.952 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:37:28.952 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 0.000000 2023-09-30T01:37:28.952 INFO:teuthology.orchestra.run.smithi088.stderr:created 0.000000 2023-09-30T01:37:28.952 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2023-09-30T01:37:28.952 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 1 2023-09-30T01:37:28.952 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.noname-a 2023-09-30T01:37:28.953 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.noname-f 2023-09-30T01:37:28.953 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.noname-b 2023-09-30T01:37:28.953 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.noname-g 2023-09-30T01:37:28.953 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.noname-c 2023-09-30T01:37:28.953 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.noname-h 2023-09-30T01:37:28.953 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.noname-d 2023-09-30T01:37:28.953 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.noname-i 2023-09-30T01:37:28.953 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.noname-e 2023-09-30T01:37:28.953 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:37:28.959 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:28.936+0000 7f3530715700 1 Processor -- start 2023-09-30T01:37:28.959 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:28.936+0000 7f3530715700 1 -- start start 2023-09-30T01:37:28.959 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:28.936+0000 7f3530715700 10 monclient: init 2023-09-30T01:37:28.959 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:28.936+0000 7f3530715700 10 monclient: _reopen_session rank -1 2023-09-30T01:37:28.959 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:28.936+0000 7f3530715700 10 monclient: _add_conns ranks=[5,7,3,1,8,0,2,6,4] 2023-09-30T01:37:28.959 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:28.936+0000 7f3530715700 1 --2- >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7f35280a3270 0x7f352811d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:37:28.959 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:28.936+0000 7f3530715700 10 monclient(hunting): picked mon.noname-h con 0x7f35280a3270 addr [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] 2023-09-30T01:37:28.959 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:28.936+0000 7f3530715700 1 --2- >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7f35280a4b00 0x7f352811df20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:37:28.959 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:28.936+0000 7f3530715700 10 monclient(hunting): picked mon.noname-i con 0x7f35280a4b00 addr [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] 2023-09-30T01:37:28.959 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:28.936+0000 7f3530715700 1 --2- >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7f35281333c0 0x7f352811e460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:37:28.960 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:28.936+0000 7f3530715700 10 monclient(hunting): picked mon.noname-g con 0x7f35281333c0 addr [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] 2023-09-30T01:37:28.960 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:28.936+0000 7f3530715700 10 monclient(hunting): start opening mon connection 2023-09-30T01:37:28.960 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:28.936+0000 7f3530715700 1 -- --> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35281304c0 con 0x7f35281333c0 2023-09-30T01:37:28.960 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:28.936+0000 7f3530715700 10 monclient(hunting): start opening mon connection 2023-09-30T01:37:28.960 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:28.936+0000 7f3530715700 1 -- --> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] -- mon_getmap magic: 0 v1 -- 0x7f3528097ca0 con 0x7f35280a3270 2023-09-30T01:37:28.960 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:28.936+0000 7f3530715700 10 monclient(hunting): start opening mon connection 2023-09-30T01:37:28.960 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:28.936+0000 7f3530715700 1 -- --> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] -- mon_getmap magic: 0 v1 -- 0x7f35280a0eb0 con 0x7f35280a4b00 2023-09-30T01:37:28.960 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:28.936+0000 7f3530715700 10 monclient(hunting): _renew_subs 2023-09-30T01:37:28.960 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:28.936+0000 7f3530715700 10 monclient(hunting): authenticate will time out at 1120.979988s 2023-09-30T01:37:28.960 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:28.940+0000 7f352dcb0700 1 --2- >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7f35280a4b00 0x7f352811df20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:37:28.960 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:28.940+0000 7f352ecb2700 1 --2- >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7f35281333c0 0x7f352811e460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:37:28.961 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:28.940+0000 7f352dcb0700 1 --2- >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7f35280a4b00 0x7f352811df20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3303/0 says I am v2:172.21.15.88:60402/0 (socket says 172.21.15.88:60402) 2023-09-30T01:37:28.961 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:28.940+0000 7f352e4b1700 1 --2- >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7f35280a3270 0x7f352811d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:37:28.961 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:28.940+0000 7f352dcb0700 1 -- 172.21.15.88:0/3868197435 learned_addr learned my addr 172.21.15.88:0/3868197435 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-09-30T01:37:28.961 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:28.940+0000 7f352dcb0700 10 monclient(hunting): get_auth_request con 0x7f35280a4b00 auth_method 0 2023-09-30T01:37:28.961 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:28.940+0000 7f352dcb0700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:37:28.961 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:28.940+0000 7f352dcb0700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:37:28.961 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:28.940+0000 7f352dcb0700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:37:28.961 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:28.940+0000 7f352e4b1700 10 monclient(hunting): get_auth_request con 0x7f35280a3270 auth_method 0 2023-09-30T01:37:28.961 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:28.940+0000 7f352e4b1700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:37:28.961 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:28.940+0000 7f352e4b1700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:37:28.961 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:28.940+0000 7f352e4b1700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:37:28.962 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:28.940+0000 7f352ecb2700 10 monclient(hunting): get_auth_request con 0x7f35281333c0 auth_method 0 2023-09-30T01:37:28.962 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:28.940+0000 7f352ecb2700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:37:28.962 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:28.940+0000 7f352ecb2700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:37:28.962 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:28.940+0000 7f352ecb2700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:37:28.962 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:28.940+0000 7f352dcb0700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:37:28.962 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:28.940+0000 7f352dcb0700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:37:28.962 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:28.940+0000 7f352dcb0700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:37:28.962 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:28.940+0000 7f352ecb2700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:37:28.962 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:28.940+0000 7f352ecb2700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:37:28.962 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:28.940+0000 7f352ecb2700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:37:28.962 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:28.940+0000 7f352e4b1700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:37:28.962 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:28.940+0000 7f352e4b1700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:37:28.963 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:28.940+0000 7f352e4b1700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:37:28.963 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:28.940+0000 7f352dcb0700 10 monclient(hunting): handle_auth_done global_id 4831 payload 824 2023-09-30T01:37:28.963 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:28.940+0000 7f352dcb0700 1 -- 172.21.15.88:0/3868197435 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7f35280a3270 msgr2=0x7f352811d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:37:28.963 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:28.940+0000 7f352dcb0700 1 --2- 172.21.15.88:0/3868197435 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7f35280a3270 0x7f352811d9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:37:28.963 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:28.940+0000 7f352dcb0700 1 -- 172.21.15.88:0/3868197435 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7f35281333c0 msgr2=0x7f352811e460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:37:28.963 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:28.940+0000 7f352dcb0700 1 --2- 172.21.15.88:0/3868197435 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7f35281333c0 0x7f352811e460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:37:28.963 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:28.940+0000 7f352dcb0700 10 monclient: _finish_hunting 0 2023-09-30T01:37:28.963 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:28.940+0000 7f352dcb0700 1 monclient: found mon.noname-i 2023-09-30T01:37:28.963 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:28.940+0000 7f352dcb0700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:37:28.963 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:28.940+0000 7f352dcb0700 10 monclient: _send_mon_message to mon.noname-i at v2:172.21.15.107:3303/0 2023-09-30T01:37:28.963 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:28.940+0000 7f352dcb0700 1 -- 172.21.15.88:0/3868197435 --> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35240094f0 con 0x7f35280a4b00 2023-09-30T01:37:28.963 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:28.940+0000 7f352ecb2700 1 --2- 172.21.15.88:0/3868197435 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7f35281333c0 0x7f352811e460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:37:28.964 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:28.940+0000 7f352dcb0700 1 --2- 172.21.15.88:0/3868197435 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7f35280a4b00 0x7f352811df20 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f351800ec80 tx=0x7f35180091a0 comp rx=0 tx=0).ready entity=mon.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:37:28.964 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:28.940+0000 7f351e7fc700 1 -- 172.21.15.88:0/3868197435 <== mon.7 v2:172.21.15.107:3303/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f3518009690 con 0x7f35280a4b00 2023-09-30T01:37:28.964 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:28.940+0000 7f351e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:37:28.964 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:28.940+0000 7f351e7fc700 10 monclient: got monmap 1 from mon.noname-i (according to old e1) 2023-09-30T01:37:28.964 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:28.940+0000 7f351e7fc700 10 monclient: dump: 2023-09-30T01:37:28.964 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:37:28.964 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:37:28.964 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:37:28.964 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:37:28.964 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:37:28.964 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:37:28.964 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:37:28.965 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:37:28.965 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:37:28.965 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:37:28.965 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:37:28.965 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:37:28.965 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:37:28.965 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:37:28.965 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:37:28.965 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:37:28.965 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:28.940+0000 7f351e7fc700 10 monclient: _finish_auth 0 2023-09-30T01:37:28.965 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:28.940+0000 7f351e7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:37:28.965 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:28.940+0000 7f3530715700 5 monclient: authenticate success, global_id 4831 2023-09-30T01:37:28.965 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:28.940+0000 7f351e7fc700 1 -- 172.21.15.88:0/3868197435 <== mon.7 v2:172.21.15.107:3303/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f351800ccd0 con 0x7f35280a4b00 2023-09-30T01:37:28.966 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:28.940+0000 7f351e7fc700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:37:28.966 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:28.940+0000 7f351e7fc700 1 -- 172.21.15.88:0/3868197435 <== mon.7 v2:172.21.15.107:3303/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f351800ce40 con 0x7f35280a4b00 2023-09-30T01:37:28.966 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:28.940+0000 7f3530715700 10 monclient: _renew_subs 2023-09-30T01:37:28.966 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:28.940+0000 7f3530715700 10 monclient: _send_mon_message to mon.i at v2:172.21.15.107:3303/0 2023-09-30T01:37:28.966 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:28.940+0000 7f3530715700 1 -- 172.21.15.88:0/3868197435 --> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3528121170 con 0x7f35280a4b00 2023-09-30T01:37:28.966 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:28.940+0000 7f351e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:37:28.966 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:28.940+0000 7f351e7fc700 10 monclient: got monmap 1 from mon.i (according to old e1) 2023-09-30T01:37:28.966 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:28.940+0000 7f352ccae700 4 set_mon_vals no callback set 2023-09-30T01:37:28.966 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:28.940+0000 7f351e7fc700 10 monclient: dump: 2023-09-30T01:37:28.966 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:37:28.966 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:37:28.966 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:37:28.967 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:37:28.967 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:37:28.967 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:37:28.967 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:37:28.967 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:37:28.967 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:37:28.967 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:37:28.967 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:37:28.967 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:37:28.967 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:37:28.967 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:37:28.967 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:37:28.968 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:37:28.968 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:28.940+0000 7f3530715700 10 monclient: _renew_subs 2023-09-30T01:37:28.968 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:28.940+0000 7f3530715700 10 monclient: _send_mon_message to mon.i at v2:172.21.15.107:3303/0 2023-09-30T01:37:28.968 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:28.940+0000 7f3530715700 1 -- 172.21.15.88:0/3868197435 --> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3528064b80 con 0x7f35280a4b00 2023-09-30T01:37:28.968 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:28.940+0000 7f351e7fc700 1 -- 172.21.15.88:0/3868197435 <== mon.7 v2:172.21.15.107:3303/0 4 ==== mgrmap(e 5) v1 ==== 40144+0+0 (secure 0 0 0) 0x7f351800c850 con 0x7f35280a4b00 2023-09-30T01:37:28.968 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:28.940+0000 7f351e7fc700 1 --2- 172.21.15.88:0/3868197435 >> v2:172.21.15.107:6812/29926 conn(0x7f350c033360 0x7f350c035810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:37:28.968 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:28.940+0000 7f351e7fc700 1 -- 172.21.15.88:0/3868197435 <== mon.7 v2:172.21.15.107:3303/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 3502+0+0 (secure 0 0 0) 0x7f35180436c0 con 0x7f35280a4b00 2023-09-30T01:37:28.968 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:28.940+0000 7f3530715700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:37:28.968 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:28.940+0000 7f3530715700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:37:28.968 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:28.940+0000 7f3530715700 10 monclient: _send_mon_message to mon.i at v2:172.21.15.107:3303/0 2023-09-30T01:37:28.969 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:28.940+0000 7f3530715700 1 -- 172.21.15.88:0/3868197435 --> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f352806f0d0 con 0x7f35280a4b00 2023-09-30T01:37:28.969 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:28.940+0000 7f352e4b1700 1 --2- 172.21.15.88:0/3868197435 >> v2:172.21.15.107:6812/29926 conn(0x7f350c033360 0x7f350c035810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:37:28.969 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:28.944+0000 7f352e4b1700 10 monclient: get_auth_request con 0x7f350c033360 auth_method 0 2023-09-30T01:37:28.969 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:28.944+0000 7f352e4b1700 1 --2- 172.21.15.88:0/3868197435 >> v2:172.21.15.107:6812/29926 conn(0x7f350c033360 0x7f350c035810 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f3524009fd0 tx=0x7f3524017000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:37:28.969 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:28.952+0000 7f351e7fc700 1 -- 172.21.15.88:0/3868197435 <== mon.7 v2:172.21.15.107:3303/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f351800cb00 con 0x7f35280a4b00 2023-09-30T01:37:28.969 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:28.952+0000 7f351e7fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:37:28.969 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:28.952+0000 7f351e7fc700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:37:29.158 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:29.152+0000 7f3530715700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T01:37:29.158 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:29.152+0000 7f3530715700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:37:29.158 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:29.152+0000 7f3530715700 10 monclient: _send_mon_message to mon.i at v2:172.21.15.107:3303/0 2023-09-30T01:37:29.158 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:29.152+0000 7f3530715700 1 -- 172.21.15.88:0/3868197435 --> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f3528071960 con 0x7f35280a4b00 2023-09-30T01:37:29.159 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:29.152+0000 7f351e7fc700 1 -- 172.21.15.88:0/3868197435 <== mon.7 v2:172.21.15.107:3303/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f3518029100 con 0x7f35280a4b00 2023-09-30T01:37:29.159 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:29.152+0000 7f351e7fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:37:29.159 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:29.152+0000 7f351e7fc700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:37:29.159 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-09-30T01:37:29.159 INFO:teuthology.orchestra.run.smithi088.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:37:29.161 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:29.156+0000 7f3530715700 1 -- 172.21.15.88:0/3868197435 >> v2:172.21.15.107:6812/29926 conn(0x7f350c033360 msgr2=0x7f350c035810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:37:29.161 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:29.156+0000 7f3530715700 1 --2- 172.21.15.88:0/3868197435 >> v2:172.21.15.107:6812/29926 conn(0x7f350c033360 0x7f350c035810 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f3524009fd0 tx=0x7f3524017000 comp rx=0 tx=0).stop 2023-09-30T01:37:29.161 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:29.156+0000 7f3530715700 10 monclient: shutdown 2023-09-30T01:37:29.161 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:29.156+0000 7f3530715700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:37:29.161 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:29.156+0000 7f3530715700 1 -- 172.21.15.88:0/3868197435 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7f35280a4b00 msgr2=0x7f352811df20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:37:29.162 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:29.156+0000 7f3530715700 1 --2- 172.21.15.88:0/3868197435 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7f35280a4b00 0x7f352811df20 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f351800ec80 tx=0x7f35180091a0 comp rx=0 tx=0).stop 2023-09-30T01:37:29.162 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:29.156+0000 7f3530715700 1 -- 172.21.15.88:0/3868197435 shutdown_connections 2023-09-30T01:37:29.162 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:29.156+0000 7f3530715700 1 --2- 172.21.15.88:0/3868197435 >> v2:172.21.15.107:6812/29926 conn(0x7f350c033360 0x7f350c035810 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:37:29.162 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:29.156+0000 7f3530715700 1 --2- 172.21.15.88:0/3868197435 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7f35281333c0 0x7f352811e460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:37:29.162 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:29.156+0000 7f3530715700 1 --2- 172.21.15.88:0/3868197435 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7f35280a4b00 0x7f352811df20 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:37:29.162 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:29.156+0000 7f3530715700 1 --2- 172.21.15.88:0/3868197435 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7f35280a3270 0x7f352811d9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:37:29.162 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:29.156+0000 7f3530715700 1 -- 172.21.15.88:0/3868197435 >> 172.21.15.88:0/3868197435 conn(0x7f35280a67f0 msgr2=0x7f35280a20e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:37:29.162 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:29.156+0000 7f3530715700 1 -- 172.21.15.88:0/3868197435 shutdown_connections 2023-09-30T01:37:29.162 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:29.156+0000 7f3530715700 1 -- 172.21.15.88:0/3868197435 wait complete. 2023-09-30T01:37:32.173 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:37:32.283 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:32.276+0000 7f820a752700 10 monclient: get_monmap_and_config 2023-09-30T01:37:32.283 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:32.276+0000 7f820a752700 10 monclient: build_initial_monmap 2023-09-30T01:37:32.283 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:32.276+0000 7f820a752700 10 monclient: monmap: 2023-09-30T01:37:32.283 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2023-09-30T01:37:32.283 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:37:32.283 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:37:32.283119+0000 2023-09-30T01:37:32.283 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:37:32.283119+0000 2023-09-30T01:37:32.284 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2023-09-30T01:37:32.284 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:37:32.284 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a-legacy 2023-09-30T01:37:32.284 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.107:6789/0 mon.noname-f-legacy 2023-09-30T01:37:32.284 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.noname-b 2023-09-30T01:37:32.284 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.noname-g 2023-09-30T01:37:32.284 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.noname-c 2023-09-30T01:37:32.284 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.noname-h 2023-09-30T01:37:32.284 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.noname-d 2023-09-30T01:37:32.284 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.noname-i 2023-09-30T01:37:32.284 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.noname-e 2023-09-30T01:37:32.284 INFO:teuthology.orchestra.run.smithi088.stderr:9: v2:172.21.15.88:3300/0 mon.noname-a 2023-09-30T01:37:32.285 INFO:teuthology.orchestra.run.smithi088.stderr:10: v2:172.21.15.107:3300/0 mon.noname-f 2023-09-30T01:37:32.285 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:37:32.288 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:32.276+0000 7f820a752700 1 Processor -- start 2023-09-30T01:37:32.288 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:32.280+0000 7f820a752700 1 -- start start 2023-09-30T01:37:32.288 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:32.280+0000 7f820a752700 10 monclient: init 2023-09-30T01:37:32.288 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:32.280+0000 7f820a752700 10 monclient: _reopen_session rank -1 2023-09-30T01:37:32.288 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:32.280+0000 7f820a752700 10 monclient: _add_conns ranks=[1,2,8,0,4,9,7,10,5,3,6] 2023-09-30T01:37:32.288 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:32.280+0000 7f820a752700 10 monclient(hunting): picked mon.noname-f-legacy con 0x7f82040a3380 addr v1:172.21.15.107:6789/0 2023-09-30T01:37:32.288 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:32.280+0000 7f820a752700 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8204097a40 0x7f82040a5320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:37:32.289 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:32.280+0000 7f820a752700 10 monclient(hunting): picked mon.noname-b con 0x7f8204097a40 addr [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] 2023-09-30T01:37:32.289 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:32.280+0000 7f820a752700 1 --2- >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f8204138670 0x7f820413aa50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:37:32.289 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:32.280+0000 7f820a752700 10 monclient(hunting): picked mon.noname-e con 0x7f8204138670 addr [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] 2023-09-30T01:37:32.289 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:32.280+0000 7f820a752700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f820404bb20 con 0x7f82040a3380 2023-09-30T01:37:32.289 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:32.280+0000 7f820a752700 10 monclient(hunting): start opening mon connection 2023-09-30T01:37:32.289 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:32.280+0000 7f820a752700 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82040a6750 con 0x7f8204097a40 2023-09-30T01:37:32.289 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:32.280+0000 7f820a752700 10 monclient(hunting): start opening mon connection 2023-09-30T01:37:32.289 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:32.280+0000 7f820a752700 1 -- --> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] -- mon_getmap magic: 0 v1 -- 0x7f82040a05e0 con 0x7f8204138670 2023-09-30T01:37:32.289 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:32.280+0000 7f820a752700 10 monclient(hunting): _renew_subs 2023-09-30T01:37:32.289 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:32.280+0000 7f820a752700 10 monclient(hunting): authenticate will time out at 1124.320653s 2023-09-30T01:37:32.289 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:32.280+0000 7f8208f4f700 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8204097a40 0x7f82040a5320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:37:32.290 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:32.280+0000 7f8208f4f700 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8204097a40 0x7f82040a5320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3301/0 says I am v2:172.21.15.88:38626/0 (socket says 172.21.15.88:38626) 2023-09-30T01:37:32.290 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:32.280+0000 7f8208f4f700 1 -- 172.21.15.88:0/884398363 learned_addr learned my addr 172.21.15.88:0/884398363 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-09-30T01:37:32.290 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:32.280+0000 7f8208f4f700 10 monclient(hunting): get_auth_request con 0x7f8204097a40 auth_method 0 2023-09-30T01:37:32.290 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:32.280+0000 7f8208f4f700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:37:32.290 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:32.280+0000 7f8208f4f700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:37:32.290 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:32.280+0000 7f8208f4f700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:37:32.290 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:32.280+0000 7f8209f51700 1 --2- 172.21.15.88:0/884398363 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f8204138670 0x7f820413aa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:37:32.290 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:32.280+0000 7f8208f4f700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:37:32.290 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:32.280+0000 7f8208f4f700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:37:32.290 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:32.280+0000 7f8208f4f700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:37:32.290 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:32.280+0000 7f8209f51700 10 monclient(hunting): get_auth_request con 0x7f8204138670 auth_method 0 2023-09-30T01:37:32.290 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:32.280+0000 7f8209f51700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:37:32.291 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:32.280+0000 7f8209f51700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:37:32.291 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:32.280+0000 7f8209f51700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:37:32.291 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:32.280+0000 7f8209f51700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:37:32.291 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:32.280+0000 7f8209f51700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:37:32.291 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:32.280+0000 7f8209f51700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:37:32.291 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:32.280+0000 7f8208f4f700 10 monclient(hunting): handle_auth_done global_id 4727 payload 274 2023-09-30T01:37:32.291 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:32.280+0000 7f8208f4f700 1 -- 172.21.15.88:0/884398363 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f8204138670 msgr2=0x7f820413aa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:37:32.291 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:32.280+0000 7f8208f4f700 1 --2- 172.21.15.88:0/884398363 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f8204138670 0x7f820413aa50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:37:32.291 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:32.280+0000 7f8208f4f700 1 -- 172.21.15.88:0/884398363 >> v1:172.21.15.107:6789/0 conn(0x7f82040a3380 legacy=0x7f82040a4c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:37:32.291 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:32.280+0000 7f8208f4f700 10 monclient: _finish_hunting 0 2023-09-30T01:37:32.291 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:32.280+0000 7f8208f4f700 1 monclient: found mon.noname-b 2023-09-30T01:37:32.291 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:32.280+0000 7f8208f4f700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:37:32.292 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:32.280+0000 7f8208f4f700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.88:3301/0 2023-09-30T01:37:32.292 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:32.280+0000 7f8208f4f700 1 -- 172.21.15.88:0/884398363 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82040a3750 con 0x7f8204097a40 2023-09-30T01:37:32.292 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:32.280+0000 7f8208f4f700 1 --2- 172.21.15.88:0/884398363 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8204097a40 0x7f82040a5320 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f8200009f50 tx=0x7f8200009840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5e11f61b62229d8a server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:37:32.292 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:32.280+0000 7f81fbfff700 1 -- 172.21.15.88:0/884398363 <== mon.2 v2:172.21.15.88:3301/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f820000e070 con 0x7f8204097a40 2023-09-30T01:37:32.292 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:32.280+0000 7f81fbfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:37:32.292 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:32.280+0000 7f81fbfff700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T01:37:32.292 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:32.280+0000 7f81fbfff700 10 monclient: dump: 2023-09-30T01:37:32.292 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:37:32.292 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:37:32.292 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:37:32.292 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:37:32.292 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:37:32.293 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:37:32.293 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:37:32.293 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:37:32.293 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:37:32.293 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:37:32.293 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:37:32.293 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:37:32.293 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:37:32.293 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:37:32.293 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:37:32.293 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:37:32.293 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:32.280+0000 7f81fbfff700 10 monclient: _finish_auth 0 2023-09-30T01:37:32.293 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:32.280+0000 7f81fbfff700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:37:32.294 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:32.280+0000 7f820a752700 5 monclient: authenticate success, global_id 4727 2023-09-30T01:37:32.294 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:32.280+0000 7f820a752700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:37:32.294 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:32.280+0000 7f81fbfff700 1 -- 172.21.15.88:0/884398363 <== mon.2 v2:172.21.15.88:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f820000be80 con 0x7f8204097a40 2023-09-30T01:37:32.294 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:32.280+0000 7f81fbfff700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:37:32.294 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:32.280+0000 7f820a752700 10 monclient: get_monmap_and_config success 2023-09-30T01:37:32.294 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:32.280+0000 7f820a752700 4 set_mon_vals no callback set 2023-09-30T01:37:32.294 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:32.280+0000 7f81fbfff700 1 -- 172.21.15.88:0/884398363 <== mon.2 v2:172.21.15.88:3301/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f82000092e0 con 0x7f8204097a40 2023-09-30T01:37:32.294 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:32.280+0000 7f81fbfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:37:32.294 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:32.280+0000 7f81fbfff700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T01:37:32.294 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:32.280+0000 7f81fbfff700 10 monclient: dump: 2023-09-30T01:37:32.294 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:37:32.294 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:37:32.295 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:37:32.295 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:37:32.295 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:37:32.295 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:37:32.295 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:37:32.295 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:37:32.295 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:37:32.295 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:37:32.295 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:37:32.295 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:37:32.295 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:37:32.295 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:37:32.295 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:37:32.296 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:37:32.296 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:32.280+0000 7f820a752700 10 monclient: shutdown 2023-09-30T01:37:32.296 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:32.280+0000 7f820a752700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:37:32.296 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:32.280+0000 7f820a752700 1 -- 172.21.15.88:0/884398363 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8204097a40 msgr2=0x7f82040a5320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:37:32.296 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:32.280+0000 7f820a752700 1 --2- 172.21.15.88:0/884398363 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8204097a40 0x7f82040a5320 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f8200009f50 tx=0x7f8200009840 comp rx=0 tx=0).stop 2023-09-30T01:37:32.296 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:32.280+0000 7f820a752700 1 -- 172.21.15.88:0/884398363 shutdown_connections 2023-09-30T01:37:32.296 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:32.280+0000 7f820a752700 1 --2- 172.21.15.88:0/884398363 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f8204138670 0x7f820413aa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:37:32.296 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:32.280+0000 7f820a752700 1 --2- 172.21.15.88:0/884398363 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8204097a40 0x7f82040a5320 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:37:32.296 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:32.280+0000 7f820a752700 1 -- 172.21.15.88:0/884398363 >> 172.21.15.88:0/884398363 conn(0x7f8204069310 msgr2=0x7f82040a6900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:37:32.296 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:32.280+0000 7f820a752700 1 -- 172.21.15.88:0/884398363 shutdown_connections 2023-09-30T01:37:32.296 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:32.280+0000 7f820a752700 1 -- 172.21.15.88:0/884398363 wait complete. 2023-09-30T01:37:32.296 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:32.284+0000 7f820a752700 10 monclient: build_initial_monmap 2023-09-30T01:37:32.296 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:32.284+0000 7f820a752700 10 monclient: monmap: 2023-09-30T01:37:32.297 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2023-09-30T01:37:32.297 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:37:32.297 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 0.000000 2023-09-30T01:37:32.297 INFO:teuthology.orchestra.run.smithi088.stderr:created 0.000000 2023-09-30T01:37:32.297 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2023-09-30T01:37:32.297 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 1 2023-09-30T01:37:32.297 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.noname-a 2023-09-30T01:37:32.297 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.noname-f 2023-09-30T01:37:32.297 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.noname-b 2023-09-30T01:37:32.297 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.noname-g 2023-09-30T01:37:32.297 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.noname-c 2023-09-30T01:37:32.297 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.noname-h 2023-09-30T01:37:32.297 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.noname-d 2023-09-30T01:37:32.298 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.noname-i 2023-09-30T01:37:32.298 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.noname-e 2023-09-30T01:37:32.298 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:37:32.302 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:32.284+0000 7f820a752700 1 Processor -- start 2023-09-30T01:37:32.302 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:32.284+0000 7f820a752700 1 -- start start 2023-09-30T01:37:32.302 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:32.284+0000 7f820a752700 10 monclient: init 2023-09-30T01:37:32.302 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:32.284+0000 7f820a752700 10 monclient: _reopen_session rank -1 2023-09-30T01:37:32.302 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:32.284+0000 7f820a752700 10 monclient: _add_conns ranks=[4,5,7,0,8,3,1,6,2] 2023-09-30T01:37:32.302 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:32.284+0000 7f820a752700 1 --2- >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f8204097a40 0x7f8204142cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:37:32.302 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:32.284+0000 7f820a752700 10 monclient(hunting): picked mon.noname-c con 0x7f8204097a40 addr [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] 2023-09-30T01:37:32.302 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:32.284+0000 7f820a752700 1 --2- >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7f82040a3380 0x7f82041431f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:37:32.302 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:32.284+0000 7f820a752700 10 monclient(hunting): picked mon.noname-h con 0x7f82040a3380 addr [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] 2023-09-30T01:37:32.302 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:32.284+0000 7f820a752700 1 --2- >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7f8204138670 0x7f820409bb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:37:32.303 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:32.284+0000 7f820a752700 10 monclient(hunting): picked mon.noname-i con 0x7f8204138670 addr [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] 2023-09-30T01:37:32.303 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:32.284+0000 7f820a752700 10 monclient(hunting): start opening mon connection 2023-09-30T01:37:32.303 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:32.284+0000 7f820a752700 1 -- --> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] -- mon_getmap magic: 0 v1 -- 0x7f820409f8f0 con 0x7f8204097a40 2023-09-30T01:37:32.303 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:32.284+0000 7f820a752700 10 monclient(hunting): start opening mon connection 2023-09-30T01:37:32.303 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:32.284+0000 7f820a752700 1 -- --> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] -- mon_getmap magic: 0 v1 -- 0x7f82040a0a30 con 0x7f82040a3380 2023-09-30T01:37:32.303 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:32.284+0000 7f820a752700 10 monclient(hunting): start opening mon connection 2023-09-30T01:37:32.303 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:32.284+0000 7f820a752700 1 -- --> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] -- mon_getmap magic: 0 v1 -- 0x7f82040a64d0 con 0x7f8204138670 2023-09-30T01:37:32.303 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:32.284+0000 7f820a752700 10 monclient(hunting): _renew_subs 2023-09-30T01:37:32.303 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:32.284+0000 7f820a752700 10 monclient(hunting): authenticate will time out at 1124.326146s 2023-09-30T01:37:32.303 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:32.284+0000 7f8209750700 1 --2- >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f8204097a40 0x7f8204142cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:37:32.303 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:32.284+0000 7f8208f4f700 1 --2- >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7f82040a3380 0x7f82041431f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:37:32.303 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:32.284+0000 7f8209750700 1 --2- >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f8204097a40 0x7f8204142cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3302/0 says I am v2:172.21.15.88:46296/0 (socket says 172.21.15.88:46296) 2023-09-30T01:37:32.304 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:32.284+0000 7f8209750700 1 -- 172.21.15.88:0/3102900074 learned_addr learned my addr 172.21.15.88:0/3102900074 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-09-30T01:37:32.304 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:32.284+0000 7f8209750700 10 monclient(hunting): get_auth_request con 0x7f8204097a40 auth_method 0 2023-09-30T01:37:32.304 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:32.284+0000 7f8209750700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:37:32.304 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:32.284+0000 7f8209750700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:37:32.304 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:32.284+0000 7f8209750700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:37:32.304 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:32.284+0000 7f8208f4f700 10 monclient(hunting): get_auth_request con 0x7f82040a3380 auth_method 0 2023-09-30T01:37:32.304 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:32.284+0000 7f8208f4f700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:37:32.304 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:32.284+0000 7f8208f4f700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:37:32.304 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:32.284+0000 7f8208f4f700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:37:32.304 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:32.284+0000 7f8209f51700 1 --2- 172.21.15.88:0/3102900074 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7f8204138670 0x7f820409bb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:37:32.304 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:32.284+0000 7f8209f51700 10 monclient(hunting): get_auth_request con 0x7f8204138670 auth_method 0 2023-09-30T01:37:32.304 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:32.284+0000 7f8209f51700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:37:32.304 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:32.284+0000 7f8209f51700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:37:32.305 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:32.284+0000 7f8209f51700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:37:32.305 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:32.284+0000 7f8208f4f700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:37:32.305 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:32.284+0000 7f8208f4f700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:37:32.305 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:32.284+0000 7f8208f4f700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:37:32.305 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:32.284+0000 7f8209750700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:37:32.305 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:32.284+0000 7f8209750700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:37:32.305 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:32.284+0000 7f8209750700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:37:32.305 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:32.284+0000 7f8209f51700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:37:32.305 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:32.284+0000 7f8209f51700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:37:32.305 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:32.284+0000 7f8209f51700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:37:32.305 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:32.284+0000 7f8208f4f700 10 monclient(hunting): handle_auth_done global_id 4793 payload 824 2023-09-30T01:37:32.305 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:32.284+0000 7f8208f4f700 1 -- 172.21.15.88:0/3102900074 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7f8204138670 msgr2=0x7f820409bb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:37:32.306 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:32.284+0000 7f8208f4f700 1 --2- 172.21.15.88:0/3102900074 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7f8204138670 0x7f820409bb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:37:32.306 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:32.284+0000 7f8208f4f700 1 -- 172.21.15.88:0/3102900074 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f8204097a40 msgr2=0x7f8204142cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:37:32.306 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:32.284+0000 7f8208f4f700 1 --2- 172.21.15.88:0/3102900074 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f8204097a40 0x7f8204142cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:37:32.306 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:32.284+0000 7f8208f4f700 10 monclient: _finish_hunting 0 2023-09-30T01:37:32.306 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:32.284+0000 7f8208f4f700 1 monclient: found mon.noname-h 2023-09-30T01:37:32.306 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:32.284+0000 7f8208f4f700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:37:32.306 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:32.284+0000 7f8208f4f700 10 monclient: _send_mon_message to mon.noname-h at v2:172.21.15.107:3302/0 2023-09-30T01:37:32.306 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:32.284+0000 7f8208f4f700 1 -- 172.21.15.88:0/3102900074 --> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8200009ae0 con 0x7f82040a3380 2023-09-30T01:37:32.306 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:32.284+0000 7f8209750700 1 --2- 172.21.15.88:0/3102900074 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f8204097a40 0x7f8204142cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:37:32.306 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:32.284+0000 7f8208f4f700 1 --2- 172.21.15.88:0/3102900074 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7f82040a3380 0x7f82041431f0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f82040a7830 tx=0x7f8200018000 comp rx=0 tx=0).ready entity=mon.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:37:32.306 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:32.284+0000 7f81f97fa700 1 -- 172.21.15.88:0/3102900074 <== mon.5 v2:172.21.15.107:3302/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f8200003300 con 0x7f82040a3380 2023-09-30T01:37:32.306 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:32.284+0000 7f81f97fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:37:32.307 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:32.284+0000 7f81f97fa700 10 monclient: got monmap 1 from mon.noname-h (according to old e1) 2023-09-30T01:37:32.307 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:32.284+0000 7f81f97fa700 10 monclient: dump: 2023-09-30T01:37:32.307 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:37:32.307 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:37:32.307 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:37:32.307 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:37:32.307 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:37:32.307 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:37:32.307 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:37:32.307 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:37:32.307 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:37:32.307 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:37:32.307 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:37:32.308 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:37:32.308 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:37:32.308 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:37:32.308 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:37:32.308 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:37:32.308 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:32.284+0000 7f81f97fa700 10 monclient: _finish_auth 0 2023-09-30T01:37:32.308 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:32.284+0000 7f81f97fa700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:37:32.308 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:32.284+0000 7f81f97fa700 1 -- 172.21.15.88:0/3102900074 <== mon.5 v2:172.21.15.107:3302/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8200007000 con 0x7f82040a3380 2023-09-30T01:37:32.308 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:32.284+0000 7f820a752700 5 monclient: authenticate success, global_id 4793 2023-09-30T01:37:32.308 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:32.284+0000 7f820a752700 10 monclient: _renew_subs 2023-09-30T01:37:32.308 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:32.284+0000 7f820a752700 10 monclient: _send_mon_message to mon.h at v2:172.21.15.107:3302/0 2023-09-30T01:37:32.308 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:32.284+0000 7f820a752700 1 -- 172.21.15.88:0/3102900074 --> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f820409dd10 con 0x7f82040a3380 2023-09-30T01:37:32.308 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:32.288+0000 7f81f97fa700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:37:32.309 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:32.288+0000 7f81f97fa700 1 -- 172.21.15.88:0/3102900074 <== mon.5 v2:172.21.15.107:3302/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f820001d8a0 con 0x7f82040a3380 2023-09-30T01:37:32.309 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:32.288+0000 7f81f97fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:37:32.309 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:32.288+0000 7f81f97fa700 10 monclient: got monmap 1 from mon.h (according to old e1) 2023-09-30T01:37:32.309 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:32.288+0000 7f81f97fa700 10 monclient: dump: 2023-09-30T01:37:32.309 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:37:32.309 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:37:32.309 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:37:32.309 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:37:32.309 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:37:32.309 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:37:32.309 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:37:32.309 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:37:32.310 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:37:32.310 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:37:32.310 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:37:32.310 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:37:32.310 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:37:32.310 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:37:32.310 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:37:32.310 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:37:32.310 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:32.288+0000 7f81fb7fe700 4 set_mon_vals no callback set 2023-09-30T01:37:32.310 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:32.288+0000 7f820a752700 10 monclient: _renew_subs 2023-09-30T01:37:32.310 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:32.288+0000 7f820a752700 10 monclient: _send_mon_message to mon.h at v2:172.21.15.107:3302/0 2023-09-30T01:37:32.310 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:32.288+0000 7f820a752700 1 -- 172.21.15.88:0/3102900074 --> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f820409f2c0 con 0x7f82040a3380 2023-09-30T01:37:32.310 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:32.288+0000 7f820a752700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:37:32.311 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:32.288+0000 7f820a752700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:37:32.311 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:32.288+0000 7f820a752700 10 monclient: _send_mon_message to mon.h at v2:172.21.15.107:3302/0 2023-09-30T01:37:32.311 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:32.288+0000 7f820a752700 1 -- 172.21.15.88:0/3102900074 --> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f82040710d0 con 0x7f82040a3380 2023-09-30T01:37:32.311 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:32.288+0000 7f81f97fa700 1 -- 172.21.15.88:0/3102900074 <== mon.5 v2:172.21.15.107:3302/0 4 ==== mgrmap(e 5) v1 ==== 40144+0+0 (secure 0 0 0) 0x7f8200003050 con 0x7f82040a3380 2023-09-30T01:37:32.311 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:32.288+0000 7f81f97fa700 1 --2- 172.21.15.88:0/3102900074 >> v2:172.21.15.107:6812/29926 conn(0x7f81e8033390 0x7f81e8035840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:37:32.311 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:32.288+0000 7f81f97fa700 1 -- 172.21.15.88:0/3102900074 <== mon.5 v2:172.21.15.107:3302/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 3502+0+0 (secure 0 0 0) 0x7f8200044c60 con 0x7f82040a3380 2023-09-30T01:37:32.311 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:32.288+0000 7f8209750700 1 --2- 172.21.15.88:0/3102900074 >> v2:172.21.15.107:6812/29926 conn(0x7f81e8033390 0x7f81e8035840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:37:32.311 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:32.288+0000 7f8209750700 10 monclient: get_auth_request con 0x7f81e8033390 auth_method 0 2023-09-30T01:37:32.311 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:32.288+0000 7f8209750700 1 --2- 172.21.15.88:0/3102900074 >> v2:172.21.15.107:6812/29926 conn(0x7f81e8033390 0x7f81e8035840 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f81f4007fd0 tx=0x7f81f4009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:37:32.312 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:32.300+0000 7f81f97fa700 1 -- 172.21.15.88:0/3102900074 <== mon.5 v2:172.21.15.107:3302/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f820001d600 con 0x7f82040a3380 2023-09-30T01:37:32.312 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:32.300+0000 7f81f97fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:37:32.312 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:32.300+0000 7f81f97fa700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:37:32.504 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:32.500+0000 7f820a752700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T01:37:32.505 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:32.500+0000 7f820a752700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:37:32.505 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:32.500+0000 7f820a752700 10 monclient: _send_mon_message to mon.h at v2:172.21.15.107:3302/0 2023-09-30T01:37:32.505 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:32.500+0000 7f820a752700 1 -- 172.21.15.88:0/3102900074 --> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f8204062c70 con 0x7f82040a3380 2023-09-30T01:37:32.505 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:32.500+0000 7f81f97fa700 1 -- 172.21.15.88:0/3102900074 <== mon.5 v2:172.21.15.107:3302/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f820001bbc0 con 0x7f82040a3380 2023-09-30T01:37:32.505 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:32.500+0000 7f81f97fa700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:37:32.505 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:32.500+0000 7f81f97fa700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:37:32.506 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-09-30T01:37:32.506 INFO:teuthology.orchestra.run.smithi088.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:37:32.508 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:32.504+0000 7f820a752700 1 -- 172.21.15.88:0/3102900074 >> v2:172.21.15.107:6812/29926 conn(0x7f81e8033390 msgr2=0x7f81e8035840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:37:32.508 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:32.504+0000 7f820a752700 1 --2- 172.21.15.88:0/3102900074 >> v2:172.21.15.107:6812/29926 conn(0x7f81e8033390 0x7f81e8035840 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f81f4007fd0 tx=0x7f81f4009040 comp rx=0 tx=0).stop 2023-09-30T01:37:32.508 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:32.504+0000 7f820a752700 10 monclient: shutdown 2023-09-30T01:37:32.509 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:32.504+0000 7f820a752700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:37:32.509 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:32.504+0000 7f820a752700 1 -- 172.21.15.88:0/3102900074 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7f82040a3380 msgr2=0x7f82041431f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:37:32.509 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:32.504+0000 7f820a752700 1 --2- 172.21.15.88:0/3102900074 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7f82040a3380 0x7f82041431f0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f82040a7830 tx=0x7f8200018000 comp rx=0 tx=0).stop 2023-09-30T01:37:32.509 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:32.504+0000 7f820a752700 1 -- 172.21.15.88:0/3102900074 shutdown_connections 2023-09-30T01:37:32.509 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:32.504+0000 7f820a752700 1 --2- 172.21.15.88:0/3102900074 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7f8204138670 0x7f820409bb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:37:32.509 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:32.504+0000 7f820a752700 1 --2- 172.21.15.88:0/3102900074 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7f82040a3380 0x7f82041431f0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:37:32.509 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:32.504+0000 7f820a752700 1 --2- 172.21.15.88:0/3102900074 >> v2:172.21.15.107:6812/29926 conn(0x7f81e8033390 0x7f81e8035840 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:37:32.509 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:32.504+0000 7f820a752700 1 --2- 172.21.15.88:0/3102900074 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f8204097a40 0x7f8204142cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:37:32.509 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:32.504+0000 7f820a752700 1 -- 172.21.15.88:0/3102900074 >> 172.21.15.88:0/3102900074 conn(0x7f8204069310 msgr2=0x7f82041313f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:37:32.509 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:32.504+0000 7f820a752700 1 -- 172.21.15.88:0/3102900074 shutdown_connections 2023-09-30T01:37:32.509 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:32.504+0000 7f820a752700 1 -- 172.21.15.88:0/3102900074 wait complete. 2023-09-30T01:37:35.523 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:37:35.638 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:35.631+0000 7f617adff700 10 monclient: get_monmap_and_config 2023-09-30T01:37:35.639 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:35.631+0000 7f617adff700 10 monclient: build_initial_monmap 2023-09-30T01:37:35.639 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:35.631+0000 7f617adff700 10 monclient: monmap: 2023-09-30T01:37:35.639 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2023-09-30T01:37:35.639 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:37:35.639 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:37:35.638638+0000 2023-09-30T01:37:35.639 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:37:35.638638+0000 2023-09-30T01:37:35.639 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2023-09-30T01:37:35.639 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:37:35.640 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a-legacy 2023-09-30T01:37:35.640 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.107:6789/0 mon.noname-f-legacy 2023-09-30T01:37:35.640 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.noname-b 2023-09-30T01:37:35.640 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.noname-g 2023-09-30T01:37:35.640 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.noname-c 2023-09-30T01:37:35.640 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.noname-h 2023-09-30T01:37:35.640 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.noname-d 2023-09-30T01:37:35.640 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.noname-i 2023-09-30T01:37:35.640 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.noname-e 2023-09-30T01:37:35.640 INFO:teuthology.orchestra.run.smithi088.stderr:9: v2:172.21.15.88:3300/0 mon.noname-a 2023-09-30T01:37:35.641 INFO:teuthology.orchestra.run.smithi088.stderr:10: v2:172.21.15.107:3300/0 mon.noname-f 2023-09-30T01:37:35.641 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:37:35.649 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:35.631+0000 7f617adff700 1 Processor -- start 2023-09-30T01:37:35.649 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:35.631+0000 7f617adff700 1 -- start start 2023-09-30T01:37:35.649 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:35.631+0000 7f617adff700 10 monclient: init 2023-09-30T01:37:35.649 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:35.631+0000 7f617adff700 10 monclient: _reopen_session rank -1 2023-09-30T01:37:35.649 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:35.635+0000 7f617adff700 10 monclient: _add_conns ranks=[2,5,1,6,8,0,9,7,4,3,10] 2023-09-30T01:37:35.649 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:35.635+0000 7f617adff700 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f61740a3380 0x7f61740a48b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:37:35.649 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:35.635+0000 7f617adff700 10 monclient(hunting): picked mon.noname-b con 0x7f61740a3380 addr [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] 2023-09-30T01:37:35.649 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:35.635+0000 7f617adff700 1 --2- >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7f617409ffd0 0x7f61740a03a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:37:35.650 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:35.635+0000 7f617adff700 10 monclient(hunting): picked mon.noname-h con 0x7f617409ffd0 addr [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] 2023-09-30T01:37:35.650 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:35.635+0000 7f617adff700 10 monclient(hunting): picked mon.noname-f-legacy con 0x7f61741333b0 addr v1:172.21.15.107:6789/0 2023-09-30T01:37:35.650 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:35.635+0000 7f617adff700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f617404bb20 con 0x7f61741333b0 2023-09-30T01:37:35.650 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:35.635+0000 7f617adff700 10 monclient(hunting): start opening mon connection 2023-09-30T01:37:35.650 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:35.635+0000 7f617adff700 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6174097ab0 con 0x7f61740a3380 2023-09-30T01:37:35.650 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:35.635+0000 7f617adff700 10 monclient(hunting): start opening mon connection 2023-09-30T01:37:35.650 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:35.635+0000 7f617adff700 1 -- --> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] -- mon_getmap magic: 0 v1 -- 0x7f61740a08e0 con 0x7f617409ffd0 2023-09-30T01:37:35.650 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:35.635+0000 7f617adff700 10 monclient(hunting): _renew_subs 2023-09-30T01:37:35.650 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:35.635+0000 7f617adff700 10 monclient(hunting): authenticate will time out at 1127.676125s 2023-09-30T01:37:35.651 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:35.635+0000 7f6179dfd700 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f61740a3380 0x7f61740a48b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:37:35.651 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:35.635+0000 7f6179dfd700 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f61740a3380 0x7f61740a48b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3301/0 says I am v2:172.21.15.88:38628/0 (socket says 172.21.15.88:38628) 2023-09-30T01:37:35.651 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:35.635+0000 7f6179dfd700 1 -- 172.21.15.88:0/1484637135 learned_addr learned my addr 172.21.15.88:0/1484637135 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-09-30T01:37:35.651 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:35.635+0000 7f6179dfd700 10 monclient(hunting): get_auth_request con 0x7f61740a3380 auth_method 0 2023-09-30T01:37:35.651 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:35.635+0000 7f6179dfd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:37:35.651 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:35.635+0000 7f6179dfd700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:37:35.651 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:35.635+0000 7f6179dfd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:37:35.651 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:35.635+0000 7f6179dfd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:37:35.651 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:35.635+0000 7f6179dfd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:37:35.651 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:35.635+0000 7f6179dfd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:37:35.652 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:35.635+0000 7f61795fc700 1 --2- 172.21.15.88:0/1484637135 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7f617409ffd0 0x7f61740a03a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:37:35.652 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:35.635+0000 7f6179dfd700 10 monclient(hunting): handle_auth_done global_id 4736 payload 274 2023-09-30T01:37:35.652 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:35.635+0000 7f6179dfd700 1 -- 172.21.15.88:0/1484637135 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7f617409ffd0 msgr2=0x7f61740a03a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:37:35.652 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:35.635+0000 7f6179dfd700 1 --2- 172.21.15.88:0/1484637135 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7f617409ffd0 0x7f61740a03a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:37:35.652 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:35.635+0000 7f6179dfd700 1 -- 172.21.15.88:0/1484637135 >> v1:172.21.15.107:6789/0 conn(0x7f61741333b0 legacy=0x7f6174133800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:37:35.652 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:35.635+0000 7f6179dfd700 10 monclient: _finish_hunting 0 2023-09-30T01:37:35.652 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:35.635+0000 7f6179dfd700 1 monclient: found mon.noname-b 2023-09-30T01:37:35.652 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:35.635+0000 7f6179dfd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:37:35.652 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:35.635+0000 7f6179dfd700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.88:3301/0 2023-09-30T01:37:35.652 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:35.635+0000 7f6179dfd700 1 -- 172.21.15.88:0/1484637135 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6174098480 con 0x7f61740a3380 2023-09-30T01:37:35.653 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:35.635+0000 7f61795fc700 10 monclient: get_auth_request con 0x7f617409ffd0 auth_method 0 2023-09-30T01:37:35.653 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:35.635+0000 7f61795fc700 1 --2- 172.21.15.88:0/1484637135 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7f617409ffd0 0x7f61740a03a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-09-30T01:37:35.653 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:35.635+0000 7f6179dfd700 1 --2- 172.21.15.88:0/1484637135 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f61740a3380 0x7f61740a48b0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f616c009f50 tx=0x7f616c009840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3dda8b1636597ac0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:37:35.653 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:35.635+0000 7f6178dfb700 1 -- 172.21.15.88:0/1484637135 <== mon.2 v2:172.21.15.88:3301/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f616c00e070 con 0x7f61740a3380 2023-09-30T01:37:35.653 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:35.635+0000 7f6178dfb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:37:35.653 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:35.635+0000 7f6178dfb700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T01:37:35.653 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:35.635+0000 7f6178dfb700 10 monclient: dump: 2023-09-30T01:37:35.653 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:37:35.653 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:37:35.653 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:37:35.654 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:37:35.654 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:37:35.654 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:37:35.654 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:37:35.654 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:37:35.654 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:37:35.654 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:37:35.654 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:37:35.654 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:37:35.654 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:37:35.654 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:37:35.655 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:37:35.655 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:37:35.655 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:35.635+0000 7f6178dfb700 10 monclient: _finish_auth 0 2023-09-30T01:37:35.655 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:35.635+0000 7f6178dfb700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:37:35.655 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:35.635+0000 7f617adff700 5 monclient: authenticate success, global_id 4736 2023-09-30T01:37:35.655 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:35.635+0000 7f617adff700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:37:35.655 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:35.635+0000 7f6178dfb700 1 -- 172.21.15.88:0/1484637135 <== mon.2 v2:172.21.15.88:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f616c00be90 con 0x7f61740a3380 2023-09-30T01:37:35.655 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:35.635+0000 7f6178dfb700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:37:35.655 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:35.635+0000 7f617adff700 10 monclient: get_monmap_and_config success 2023-09-30T01:37:35.655 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:35.635+0000 7f617adff700 4 set_mon_vals no callback set 2023-09-30T01:37:35.656 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:35.635+0000 7f6178dfb700 1 -- 172.21.15.88:0/1484637135 <== mon.2 v2:172.21.15.88:3301/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f616c0092e0 con 0x7f61740a3380 2023-09-30T01:37:35.656 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:35.635+0000 7f6178dfb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:37:35.656 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:35.635+0000 7f6178dfb700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T01:37:35.656 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:35.635+0000 7f6178dfb700 10 monclient: dump: 2023-09-30T01:37:35.656 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:37:35.656 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:37:35.656 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:37:35.656 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:37:35.656 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:37:35.656 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:37:35.657 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:37:35.657 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:37:35.657 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:37:35.657 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:37:35.657 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:37:35.657 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:37:35.657 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:37:35.657 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:37:35.657 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:37:35.657 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:37:35.658 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:35.635+0000 7f617adff700 10 monclient: shutdown 2023-09-30T01:37:35.658 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:35.635+0000 7f617adff700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:37:35.658 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:35.635+0000 7f617adff700 1 -- 172.21.15.88:0/1484637135 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f61740a3380 msgr2=0x7f61740a48b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:37:35.658 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:35.635+0000 7f617adff700 1 --2- 172.21.15.88:0/1484637135 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f61740a3380 0x7f61740a48b0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f616c009f50 tx=0x7f616c009840 comp rx=0 tx=0).stop 2023-09-30T01:37:35.658 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:35.635+0000 7f617adff700 1 -- 172.21.15.88:0/1484637135 shutdown_connections 2023-09-30T01:37:35.658 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:35.635+0000 7f617adff700 1 --2- 172.21.15.88:0/1484637135 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7f617409ffd0 0x7f61740a03a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:37:35.658 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:35.635+0000 7f617adff700 1 --2- 172.21.15.88:0/1484637135 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f61740a3380 0x7f61740a48b0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:37:35.658 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:35.635+0000 7f617adff700 1 -- 172.21.15.88:0/1484637135 >> 172.21.15.88:0/1484637135 conn(0x7f617406e5e0 msgr2=0x7f617406e9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:37:35.658 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:35.635+0000 7f617adff700 1 -- 172.21.15.88:0/1484637135 shutdown_connections 2023-09-30T01:37:35.658 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:35.635+0000 7f617adff700 1 -- 172.21.15.88:0/1484637135 wait complete. 2023-09-30T01:37:35.658 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:35.635+0000 7f617adff700 10 monclient: build_initial_monmap 2023-09-30T01:37:35.659 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:35.635+0000 7f617adff700 10 monclient: monmap: 2023-09-30T01:37:35.659 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2023-09-30T01:37:35.659 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:37:35.659 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 0.000000 2023-09-30T01:37:35.659 INFO:teuthology.orchestra.run.smithi088.stderr:created 0.000000 2023-09-30T01:37:35.659 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2023-09-30T01:37:35.659 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 1 2023-09-30T01:37:35.659 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.noname-a 2023-09-30T01:37:35.659 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.noname-f 2023-09-30T01:37:35.659 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.noname-b 2023-09-30T01:37:35.660 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.noname-g 2023-09-30T01:37:35.660 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.noname-c 2023-09-30T01:37:35.660 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.noname-h 2023-09-30T01:37:35.660 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.noname-d 2023-09-30T01:37:35.660 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.noname-i 2023-09-30T01:37:35.660 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.noname-e 2023-09-30T01:37:35.660 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:37:35.660 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:35.635+0000 7f617adff700 1 Processor -- start 2023-09-30T01:37:35.660 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:35.635+0000 7f617adff700 1 -- start start 2023-09-30T01:37:35.660 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:35.635+0000 7f617adff700 10 monclient: init 2023-09-30T01:37:35.661 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:35.639+0000 7f617adff700 10 monclient: _reopen_session rank -1 2023-09-30T01:37:35.661 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:35.639+0000 7f617adff700 10 monclient: _add_conns ranks=[1,6,8,2,5,0,4,3,7] 2023-09-30T01:37:35.661 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:35.639+0000 7f617adff700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f617409ffd0 0x7f617411d6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:37:35.661 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:35.639+0000 7f617adff700 10 monclient(hunting): picked mon.noname-f con 0x7f617409ffd0 addr [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] 2023-09-30T01:37:35.661 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:35.639+0000 7f617adff700 1 --2- >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7f61740a3380 0x7f617411dc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:37:35.661 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:35.639+0000 7f617adff700 10 monclient(hunting): picked mon.noname-d con 0x7f61740a3380 addr [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] 2023-09-30T01:37:35.661 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:35.639+0000 7f617adff700 1 --2- >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f61741333b0 0x7f617411e150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:37:35.661 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:35.639+0000 7f617adff700 10 monclient(hunting): picked mon.noname-e con 0x7f61741333b0 addr [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] 2023-09-30T01:37:35.661 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:35.639+0000 7f617adff700 10 monclient(hunting): start opening mon connection 2023-09-30T01:37:35.661 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:35.639+0000 7f617adff700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61740a0ac0 con 0x7f617409ffd0 2023-09-30T01:37:35.661 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:35.639+0000 7f617adff700 10 monclient(hunting): start opening mon connection 2023-09-30T01:37:35.662 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:35.639+0000 7f617adff700 1 -- --> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] -- mon_getmap magic: 0 v1 -- 0x7f6174097ce0 con 0x7f61740a3380 2023-09-30T01:37:35.662 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:35.639+0000 7f617adff700 10 monclient(hunting): start opening mon connection 2023-09-30T01:37:35.662 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:35.639+0000 7f617adff700 1 -- --> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] -- mon_getmap magic: 0 v1 -- 0x7f6174121110 con 0x7f61741333b0 2023-09-30T01:37:35.662 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:35.639+0000 7f617adff700 10 monclient(hunting): _renew_subs 2023-09-30T01:37:35.662 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:35.639+0000 7f617adff700 10 monclient(hunting): authenticate will time out at 1127.680401s 2023-09-30T01:37:35.662 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:35.639+0000 7f61795fc700 1 --2- >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7f61740a3380 0x7f617411dc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:37:35.662 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:35.639+0000 7f617a5fe700 1 --2- >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f61741333b0 0x7f617411e150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:37:35.662 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:35.639+0000 7f6179dfd700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f617409ffd0 0x7f617411d6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:37:35.662 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:35.639+0000 7f617a5fe700 1 --2- >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f61741333b0 0x7f617411e150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3304/0 says I am v2:172.21.15.88:45156/0 (socket says 172.21.15.88:45156) 2023-09-30T01:37:35.663 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:35.639+0000 7f61795fc700 1 --2- >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7f61740a3380 0x7f617411dc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3303/0 says I am v2:172.21.15.88:38184/0 (socket says 172.21.15.88:38184) 2023-09-30T01:37:35.663 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:35.639+0000 7f617a5fe700 1 -- 172.21.15.88:0/382966819 learned_addr learned my addr 172.21.15.88:0/382966819 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-09-30T01:37:35.663 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:35.639+0000 7f617a5fe700 10 monclient(hunting): get_auth_request con 0x7f61741333b0 auth_method 0 2023-09-30T01:37:35.663 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:35.639+0000 7f617a5fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:37:35.663 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:35.639+0000 7f617a5fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:37:35.663 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:35.639+0000 7f617a5fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:37:35.663 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:35.639+0000 7f61795fc700 10 monclient(hunting): get_auth_request con 0x7f61740a3380 auth_method 0 2023-09-30T01:37:35.663 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:35.639+0000 7f61795fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:37:35.663 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:35.639+0000 7f61795fc700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:37:35.664 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:35.639+0000 7f61795fc700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:37:35.664 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:35.639+0000 7f6179dfd700 10 monclient(hunting): get_auth_request con 0x7f617409ffd0 auth_method 0 2023-09-30T01:37:35.664 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:35.639+0000 7f6179dfd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:37:35.664 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:35.639+0000 7f6179dfd700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:37:35.664 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:35.639+0000 7f6179dfd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:37:35.664 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:35.639+0000 7f617a5fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:37:35.664 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:35.639+0000 7f617a5fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:37:35.664 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:35.639+0000 7f617a5fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:37:35.664 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:35.639+0000 7f61795fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:37:35.664 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:35.639+0000 7f61795fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:37:35.664 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:35.639+0000 7f61795fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:37:35.665 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:35.639+0000 7f6179dfd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:37:35.665 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:35.639+0000 7f6179dfd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:37:35.665 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:35.639+0000 7f6179dfd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:37:35.665 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:35.639+0000 7f61795fc700 10 monclient(hunting): handle_auth_done global_id 4704 payload 824 2023-09-30T01:37:35.665 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:35.639+0000 7f61795fc700 1 -- 172.21.15.88:0/382966819 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f61741333b0 msgr2=0x7f617411e150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:37:35.665 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:35.639+0000 7f61795fc700 1 --2- 172.21.15.88:0/382966819 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f61741333b0 0x7f617411e150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:37:35.665 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:35.639+0000 7f61795fc700 1 -- 172.21.15.88:0/382966819 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f617409ffd0 msgr2=0x7f617411d6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:37:35.665 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:35.639+0000 7f61795fc700 1 --2- 172.21.15.88:0/382966819 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f617409ffd0 0x7f617411d6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:37:35.665 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:35.639+0000 7f61795fc700 10 monclient: _finish_hunting 0 2023-09-30T01:37:35.665 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:35.639+0000 7f61795fc700 1 monclient: found mon.noname-d 2023-09-30T01:37:35.666 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:35.639+0000 7f61795fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:37:35.666 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:35.639+0000 7f61795fc700 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.88:3303/0 2023-09-30T01:37:35.666 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:35.639+0000 7f61795fc700 1 -- 172.21.15.88:0/382966819 --> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f616c009ae0 con 0x7f61740a3380 2023-09-30T01:37:35.666 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:35.639+0000 7f6179dfd700 1 --2- 172.21.15.88:0/382966819 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f617409ffd0 0x7f617411d6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:37:35.666 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:35.639+0000 7f61795fc700 1 --2- 172.21.15.88:0/382966819 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7f61740a3380 0x7f617411dc10 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f616800c880 tx=0x7f6168008040 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:37:35.666 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:35.639+0000 7f6171ffb700 1 -- 172.21.15.88:0/382966819 <== mon.6 v2:172.21.15.88:3303/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f6168011a80 con 0x7f61740a3380 2023-09-30T01:37:35.667 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:35.639+0000 7f6171ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:37:35.667 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:35.639+0000 7f6171ffb700 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2023-09-30T01:37:35.667 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:35.639+0000 7f6171ffb700 10 monclient: dump: 2023-09-30T01:37:35.667 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:37:35.667 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:37:35.667 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:37:35.667 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:37:35.667 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:37:35.667 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:37:35.667 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:37:35.668 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:37:35.668 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:37:35.668 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:37:35.668 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:37:35.668 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:37:35.668 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:37:35.668 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:37:35.668 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:37:35.668 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:37:35.668 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:35.639+0000 7f6171ffb700 10 monclient: _finish_auth 0 2023-09-30T01:37:35.669 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:35.639+0000 7f6171ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:37:35.669 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:35.639+0000 7f6171ffb700 1 -- 172.21.15.88:0/382966819 <== mon.6 v2:172.21.15.88:3303/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6168011c20 con 0x7f61740a3380 2023-09-30T01:37:35.669 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:35.639+0000 7f617adff700 5 monclient: authenticate success, global_id 4704 2023-09-30T01:37:35.669 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:35.639+0000 7f617adff700 10 monclient: _renew_subs 2023-09-30T01:37:35.669 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:35.639+0000 7f617adff700 10 monclient: _send_mon_message to mon.d at v2:172.21.15.88:3303/0 2023-09-30T01:37:35.669 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:35.639+0000 7f617adff700 1 -- 172.21.15.88:0/382966819 --> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6174120e30 con 0x7f61740a3380 2023-09-30T01:37:35.669 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:35.639+0000 7f6171ffb700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:37:35.669 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:35.639+0000 7f6171ffb700 1 -- 172.21.15.88:0/382966819 <== mon.6 v2:172.21.15.88:3303/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f6168011d90 con 0x7f61740a3380 2023-09-30T01:37:35.669 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:35.639+0000 7f6171ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:37:35.669 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:35.639+0000 7f6171ffb700 10 monclient: got monmap 1 from mon.d (according to old e1) 2023-09-30T01:37:35.670 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:35.639+0000 7f6171ffb700 10 monclient: dump: 2023-09-30T01:37:35.670 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:37:35.670 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:37:35.670 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:37:35.670 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:37:35.670 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:37:35.670 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:37:35.670 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:37:35.670 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:37:35.670 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:37:35.671 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:37:35.671 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:37:35.671 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:37:35.671 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:37:35.671 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:37:35.671 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:37:35.671 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:37:35.671 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:35.639+0000 7f617adff700 10 monclient: _renew_subs 2023-09-30T01:37:35.671 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:35.639+0000 7f617adff700 10 monclient: _send_mon_message to mon.d at v2:172.21.15.88:3303/0 2023-09-30T01:37:35.671 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:35.639+0000 7f6173fff700 4 set_mon_vals no callback set 2023-09-30T01:37:35.672 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:35.639+0000 7f617adff700 1 -- 172.21.15.88:0/382966819 --> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6174130ec0 con 0x7f61740a3380 2023-09-30T01:37:35.672 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:35.639+0000 7f6171ffb700 1 -- 172.21.15.88:0/382966819 <== mon.6 v2:172.21.15.88:3303/0 4 ==== mgrmap(e 5) v1 ==== 40144+0+0 (secure 0 0 0) 0x7f6168011600 con 0x7f61740a3380 2023-09-30T01:37:35.672 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:35.639+0000 7f6171ffb700 1 --2- 172.21.15.88:0/382966819 >> v2:172.21.15.107:6812/29926 conn(0x7f6160033360 0x7f6160035810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:37:35.672 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:35.639+0000 7f6171ffb700 1 -- 172.21.15.88:0/382966819 <== mon.6 v2:172.21.15.88:3303/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 3502+0+0 (secure 0 0 0) 0x7f6168043230 con 0x7f61740a3380 2023-09-30T01:37:35.672 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:35.639+0000 7f6179dfd700 1 --2- 172.21.15.88:0/382966819 >> v2:172.21.15.107:6812/29926 conn(0x7f6160033360 0x7f6160035810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:37:35.672 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:35.639+0000 7f6179dfd700 10 monclient: get_auth_request con 0x7f6160033360 auth_method 0 2023-09-30T01:37:35.672 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:35.643+0000 7f617adff700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:37:35.672 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:35.643+0000 7f617adff700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:37:35.672 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:35.643+0000 7f617adff700 10 monclient: _send_mon_message to mon.d at v2:172.21.15.88:3303/0 2023-09-30T01:37:35.672 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:35.643+0000 7f617adff700 1 -- 172.21.15.88:0/382966819 --> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f617409c690 con 0x7f61740a3380 2023-09-30T01:37:35.673 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:35.643+0000 7f6179dfd700 1 --2- 172.21.15.88:0/382966819 >> v2:172.21.15.107:6812/29926 conn(0x7f6160033360 0x7f6160035810 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f616c009f50 tx=0x7f616c017000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:37:35.673 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:35.647+0000 7f6171ffb700 1 -- 172.21.15.88:0/382966819 <== mon.6 v2:172.21.15.88:3303/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f6168022350 con 0x7f61740a3380 2023-09-30T01:37:35.673 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:35.647+0000 7f6171ffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:37:35.673 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:35.647+0000 7f6171ffb700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:37:35.848 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:35.839+0000 7f617adff700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T01:37:35.848 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:35.843+0000 7f617adff700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:37:35.848 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:35.843+0000 7f617adff700 10 monclient: _send_mon_message to mon.d at v2:172.21.15.88:3303/0 2023-09-30T01:37:35.848 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:35.843+0000 7f617adff700 1 -- 172.21.15.88:0/382966819 --> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f61740710d0 con 0x7f61740a3380 2023-09-30T01:37:35.849 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:35.843+0000 7f6171ffb700 1 -- 172.21.15.88:0/382966819 <== mon.6 v2:172.21.15.88:3303/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f6168028080 con 0x7f61740a3380 2023-09-30T01:37:35.849 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:35.843+0000 7f6171ffb700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:37:35.849 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:35.843+0000 7f6171ffb700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:37:35.849 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-09-30T01:37:35.849 INFO:teuthology.orchestra.run.smithi088.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:37:35.851 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:35.843+0000 7f617adff700 1 -- 172.21.15.88:0/382966819 >> v2:172.21.15.107:6812/29926 conn(0x7f6160033360 msgr2=0x7f6160035810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:37:35.851 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:35.843+0000 7f617adff700 1 --2- 172.21.15.88:0/382966819 >> v2:172.21.15.107:6812/29926 conn(0x7f6160033360 0x7f6160035810 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f616c009f50 tx=0x7f616c017000 comp rx=0 tx=0).stop 2023-09-30T01:37:35.851 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:35.843+0000 7f617adff700 10 monclient: shutdown 2023-09-30T01:37:35.851 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:35.843+0000 7f617adff700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:37:35.851 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:35.843+0000 7f617adff700 1 -- 172.21.15.88:0/382966819 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7f61740a3380 msgr2=0x7f617411dc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:37:35.851 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:35.843+0000 7f617adff700 1 --2- 172.21.15.88:0/382966819 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7f61740a3380 0x7f617411dc10 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f616800c880 tx=0x7f6168008040 comp rx=0 tx=0).stop 2023-09-30T01:37:35.851 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:35.843+0000 7f617adff700 1 -- 172.21.15.88:0/382966819 shutdown_connections 2023-09-30T01:37:35.851 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:35.843+0000 7f617adff700 1 --2- 172.21.15.88:0/382966819 >> v2:172.21.15.107:6812/29926 conn(0x7f6160033360 0x7f6160035810 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:37:35.852 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:35.843+0000 7f617adff700 1 --2- 172.21.15.88:0/382966819 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f61741333b0 0x7f617411e150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:37:35.852 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:35.843+0000 7f617adff700 1 --2- 172.21.15.88:0/382966819 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7f61740a3380 0x7f617411dc10 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:37:35.852 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:35.843+0000 7f617adff700 1 --2- 172.21.15.88:0/382966819 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f617409ffd0 0x7f617411d6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:37:35.852 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:35.843+0000 7f617adff700 1 -- 172.21.15.88:0/382966819 >> 172.21.15.88:0/382966819 conn(0x7f617406e5e0 msgr2=0x7f6174132450 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:37:35.852 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:35.843+0000 7f617adff700 1 -- 172.21.15.88:0/382966819 shutdown_connections 2023-09-30T01:37:35.852 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:35.843+0000 7f617adff700 1 -- 172.21.15.88:0/382966819 wait complete. 2023-09-30T01:37:38.863 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:37:38.974 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:38.967+0000 7f4c7515f700 10 monclient: get_monmap_and_config 2023-09-30T01:37:38.974 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:38.967+0000 7f4c7515f700 10 monclient: build_initial_monmap 2023-09-30T01:37:38.974 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:38.967+0000 7f4c7515f700 10 monclient: monmap: 2023-09-30T01:37:38.974 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2023-09-30T01:37:38.975 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:37:38.975 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:37:38.974171+0000 2023-09-30T01:37:38.975 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:37:38.974171+0000 2023-09-30T01:37:38.975 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2023-09-30T01:37:38.975 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:37:38.975 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a-legacy 2023-09-30T01:37:38.975 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.107:6789/0 mon.noname-f-legacy 2023-09-30T01:37:38.975 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.noname-b 2023-09-30T01:37:38.975 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.noname-g 2023-09-30T01:37:38.975 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.noname-c 2023-09-30T01:37:38.975 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.noname-h 2023-09-30T01:37:38.975 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.noname-d 2023-09-30T01:37:38.976 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.noname-i 2023-09-30T01:37:38.976 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.noname-e 2023-09-30T01:37:38.976 INFO:teuthology.orchestra.run.smithi088.stderr:9: v2:172.21.15.88:3300/0 mon.noname-a 2023-09-30T01:37:38.976 INFO:teuthology.orchestra.run.smithi088.stderr:10: v2:172.21.15.107:3300/0 mon.noname-f 2023-09-30T01:37:38.976 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:37:38.986 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:38.967+0000 7f4c7515f700 1 Processor -- start 2023-09-30T01:37:38.987 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:38.967+0000 7f4c7515f700 1 -- start start 2023-09-30T01:37:38.987 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:38.967+0000 7f4c7515f700 10 monclient: init 2023-09-30T01:37:38.987 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:38.967+0000 7f4c7515f700 10 monclient: _reopen_session rank -1 2023-09-30T01:37:38.987 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:38.967+0000 7f4c7515f700 10 monclient: _add_conns ranks=[2,0,4,7,1,8,6,9,10,3,5] 2023-09-30T01:37:38.987 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:38.967+0000 7f4c7515f700 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4c700a35f0 0x7f4c70133330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:37:38.987 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:38.967+0000 7f4c7515f700 10 monclient(hunting): picked mon.noname-b con 0x7f4c700a35f0 addr [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] 2023-09-30T01:37:38.987 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:38.967+0000 7f4c7515f700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f4c70133870 addr v1:172.21.15.88:6789/0 2023-09-30T01:37:38.987 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:38.967+0000 7f4c7515f700 1 --2- >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f4c70133c80 0x7f4c700a4e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:37:38.987 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:38.967+0000 7f4c7515f700 10 monclient(hunting): picked mon.noname-c con 0x7f4c70133c80 addr [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] 2023-09-30T01:37:38.987 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:38.967+0000 7f4c7515f700 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c7005ecb0 con 0x7f4c70133870 2023-09-30T01:37:38.988 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:38.967+0000 7f4c7515f700 10 monclient(hunting): start opening mon connection 2023-09-30T01:37:38.988 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:38.967+0000 7f4c7515f700 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c70097ad0 con 0x7f4c700a35f0 2023-09-30T01:37:38.988 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:38.967+0000 7f4c7515f700 10 monclient(hunting): start opening mon connection 2023-09-30T01:37:38.988 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:38.967+0000 7f4c7515f700 1 -- --> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] -- mon_getmap magic: 0 v1 -- 0x7f4c700a0940 con 0x7f4c70133c80 2023-09-30T01:37:38.988 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:38.967+0000 7f4c7515f700 10 monclient(hunting): _renew_subs 2023-09-30T01:37:38.988 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:38.967+0000 7f4c7515f700 10 monclient(hunting): authenticate will time out at 1131.011840s 2023-09-30T01:37:38.988 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:38.971+0000 7f4c6f59e700 1 --2- >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f4c70133c80 0x7f4c700a4e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:37:38.988 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:38.971+0000 7f4c6ed9d700 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4c700a35f0 0x7f4c70133330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:37:38.988 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:38.971+0000 7f4c6e59c700 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f4c70133870 0x7f4c700a46f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:40648/0 (socket says 172.21.15.88:40648) 2023-09-30T01:37:38.988 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:38.971+0000 7f4c6e59c700 1 -- 172.21.15.88:0/1001042393 learned_addr learned my addr 172.21.15.88:0/1001042393 (peer_addr_for_me v1:172.21.15.88:0/0) 2023-09-30T01:37:38.988 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:38.971+0000 7f4c6f59e700 10 monclient(hunting): get_auth_request con 0x7f4c70133c80 auth_method 0 2023-09-30T01:37:38.988 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:38.971+0000 7f4c6f59e700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:37:38.989 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:38.971+0000 7f4c6f59e700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:37:38.989 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:38.971+0000 7f4c6f59e700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:37:38.989 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:38.971+0000 7f4c6ed9d700 10 monclient(hunting): get_auth_request con 0x7f4c700a35f0 auth_method 0 2023-09-30T01:37:38.989 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:38.971+0000 7f4c6ed9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:37:38.989 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:38.971+0000 7f4c6ed9d700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:37:38.989 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:38.971+0000 7f4c6ed9d700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:37:38.989 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:38.971+0000 7f4c6f59e700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:37:38.989 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:38.971+0000 7f4c6f59e700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:37:38.989 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:38.971+0000 7f4c6f59e700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:37:38.989 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:38.971+0000 7f4c6ed9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:37:38.989 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:38.971+0000 7f4c6ed9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:37:38.990 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:38.971+0000 7f4c6ed9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:37:38.990 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:38.971+0000 7f4c6f59e700 10 monclient(hunting): handle_auth_done global_id 4828 payload 274 2023-09-30T01:37:38.990 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:38.971+0000 7f4c6f59e700 1 -- 172.21.15.88:0/1001042393 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4c700a35f0 msgr2=0x7f4c70133330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:37:38.990 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:38.971+0000 7f4c6f59e700 1 --2- 172.21.15.88:0/1001042393 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4c700a35f0 0x7f4c70133330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:37:38.990 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:38.971+0000 7f4c6f59e700 1 -- 172.21.15.88:0/1001042393 >> v1:172.21.15.88:6789/0 conn(0x7f4c70133870 legacy=0x7f4c700a46f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:37:38.990 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:38.971+0000 7f4c6f59e700 10 monclient: _finish_hunting 0 2023-09-30T01:37:38.990 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:38.971+0000 7f4c6f59e700 1 monclient: found mon.noname-c 2023-09-30T01:37:38.990 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:38.971+0000 7f4c6f59e700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:37:38.990 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:38.971+0000 7f4c6f59e700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.88:3302/0 2023-09-30T01:37:38.990 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:38.971+0000 7f4c6f59e700 1 -- 172.21.15.88:0/1001042393 --> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c700a39c0 con 0x7f4c70133c80 2023-09-30T01:37:38.990 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:38.971+0000 7f4c6f59e700 1 --2- 172.21.15.88:0/1001042393 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f4c70133c80 0x7f4c700a4e00 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f4c68009910 tx=0x7f4c680091a0 comp rx=0 tx=0).ready entity=mon.4 client_cookie=85cab22865c44c57 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:37:38.991 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:38.971+0000 7f4c6dd9b700 1 -- 172.21.15.88:0/1001042393 <== mon.4 v2:172.21.15.88:3302/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f4c6800bb00 con 0x7f4c70133c80 2023-09-30T01:37:38.991 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:38.971+0000 7f4c6dd9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:37:38.991 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:38.971+0000 7f4c6dd9b700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T01:37:38.991 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:38.971+0000 7f4c6dd9b700 10 monclient: dump: 2023-09-30T01:37:38.991 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:37:38.991 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:37:38.991 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:37:38.991 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:37:38.991 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:37:38.991 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:37:38.991 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:37:38.991 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:37:38.992 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:37:38.992 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:37:38.992 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:37:38.992 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:37:38.992 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:37:38.992 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:37:38.992 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:37:38.992 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:37:38.992 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:38.971+0000 7f4c6dd9b700 10 monclient: _finish_auth 0 2023-09-30T01:37:38.992 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:38.971+0000 7f4c6dd9b700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:37:38.992 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:38.971+0000 7f4c6dd9b700 1 -- 172.21.15.88:0/1001042393 <== mon.4 v2:172.21.15.88:3302/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c6800bc80 con 0x7f4c70133c80 2023-09-30T01:37:38.992 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:38.971+0000 7f4c7515f700 5 monclient: authenticate success, global_id 4828 2023-09-30T01:37:38.993 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:38.971+0000 7f4c7515f700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:37:38.993 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:38.971+0000 7f4c6dd9b700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:37:38.993 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:38.971+0000 7f4c7515f700 10 monclient: get_monmap_and_config success 2023-09-30T01:37:38.993 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:38.971+0000 7f4c6dd9b700 1 -- 172.21.15.88:0/1001042393 <== mon.4 v2:172.21.15.88:3302/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f4c6800bdf0 con 0x7f4c70133c80 2023-09-30T01:37:38.993 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:38.971+0000 7f4c7515f700 4 set_mon_vals no callback set 2023-09-30T01:37:38.993 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:38.971+0000 7f4c6dd9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:37:38.993 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:38.971+0000 7f4c6dd9b700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T01:37:38.993 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:38.971+0000 7f4c6dd9b700 10 monclient: dump: 2023-09-30T01:37:38.993 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:37:38.993 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:37:38.993 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:37:38.994 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:37:38.994 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:37:38.994 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:37:38.994 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:37:38.994 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:37:38.994 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:37:38.994 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:37:38.994 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:37:38.994 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:37:38.994 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:37:38.994 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:37:38.994 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:37:38.995 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:37:38.995 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:38.971+0000 7f4c7515f700 10 monclient: shutdown 2023-09-30T01:37:38.995 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:38.971+0000 7f4c7515f700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:37:38.995 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:38.971+0000 7f4c7515f700 1 -- 172.21.15.88:0/1001042393 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f4c70133c80 msgr2=0x7f4c700a4e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:37:38.995 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:38.971+0000 7f4c7515f700 1 --2- 172.21.15.88:0/1001042393 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f4c70133c80 0x7f4c700a4e00 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f4c68009910 tx=0x7f4c680091a0 comp rx=0 tx=0).stop 2023-09-30T01:37:38.995 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:38.971+0000 7f4c7515f700 1 -- 172.21.15.88:0/1001042393 shutdown_connections 2023-09-30T01:37:38.995 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:38.971+0000 7f4c7515f700 1 --2- 172.21.15.88:0/1001042393 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f4c70133c80 0x7f4c700a4e00 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:37:38.995 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:38.971+0000 7f4c7515f700 1 --2- 172.21.15.88:0/1001042393 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4c700a35f0 0x7f4c70133330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:37:38.995 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:38.971+0000 7f4c7515f700 1 -- 172.21.15.88:0/1001042393 >> 172.21.15.88:0/1001042393 conn(0x7f4c7006e5e0 msgr2=0x7f4c7006e9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:37:38.995 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:38.971+0000 7f4c7515f700 1 -- 172.21.15.88:0/1001042393 shutdown_connections 2023-09-30T01:37:38.995 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:38.971+0000 7f4c7515f700 1 -- 172.21.15.88:0/1001042393 wait complete. 2023-09-30T01:37:38.995 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:38.971+0000 7f4c7515f700 10 monclient: build_initial_monmap 2023-09-30T01:37:38.996 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:38.971+0000 7f4c7515f700 10 monclient: monmap: 2023-09-30T01:37:38.996 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2023-09-30T01:37:38.996 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:37:38.996 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 0.000000 2023-09-30T01:37:38.996 INFO:teuthology.orchestra.run.smithi088.stderr:created 0.000000 2023-09-30T01:37:38.996 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2023-09-30T01:37:38.996 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 1 2023-09-30T01:37:38.996 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.noname-a 2023-09-30T01:37:38.996 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.noname-f 2023-09-30T01:37:38.996 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.noname-b 2023-09-30T01:37:38.996 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.noname-g 2023-09-30T01:37:38.996 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.noname-c 2023-09-30T01:37:38.997 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.noname-h 2023-09-30T01:37:38.997 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.noname-d 2023-09-30T01:37:38.997 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.noname-i 2023-09-30T01:37:38.997 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.noname-e 2023-09-30T01:37:38.997 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:37:38.997 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:38.971+0000 7f4c7515f700 1 Processor -- start 2023-09-30T01:37:38.997 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:38.971+0000 7f4c7515f700 1 -- start start 2023-09-30T01:37:38.997 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:38.971+0000 7f4c7515f700 10 monclient: init 2023-09-30T01:37:38.997 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:38.975+0000 7f4c7515f700 10 monclient: _reopen_session rank -1 2023-09-30T01:37:38.997 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:38.975+0000 7f4c7515f700 10 monclient: _add_conns ranks=[6,3,8,4,1,5,2,7,0] 2023-09-30T01:37:38.997 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:38.975+0000 7f4c7515f700 1 --2- >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7f4c700a35f0 0x7f4c7011d510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:37:38.997 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:38.975+0000 7f4c7515f700 10 monclient(hunting): picked mon.noname-d con 0x7f4c700a35f0 addr [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] 2023-09-30T01:37:38.997 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:38.975+0000 7f4c7515f700 1 --2- >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7f4c70133870 0x7f4c7011da50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:37:38.998 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:38.975+0000 7f4c7515f700 10 monclient(hunting): picked mon.noname-g con 0x7f4c70133870 addr [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] 2023-09-30T01:37:38.998 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:38.975+0000 7f4c7515f700 1 --2- >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f4c70133c80 0x7f4c7011df90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:37:38.998 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:38.975+0000 7f4c7515f700 10 monclient(hunting): picked mon.noname-e con 0x7f4c70133c80 addr [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] 2023-09-30T01:37:38.998 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:38.975+0000 7f4c7515f700 10 monclient(hunting): start opening mon connection 2023-09-30T01:37:38.998 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:38.975+0000 7f4c7515f700 1 -- --> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c70120ed0 con 0x7f4c70133870 2023-09-30T01:37:38.998 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:38.975+0000 7f4c7515f700 10 monclient(hunting): start opening mon connection 2023-09-30T01:37:38.998 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:38.975+0000 7f4c7515f700 1 -- --> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] -- mon_getmap magic: 0 v1 -- 0x7f4c700a0b70 con 0x7f4c700a35f0 2023-09-30T01:37:38.998 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:38.975+0000 7f4c7515f700 10 monclient(hunting): start opening mon connection 2023-09-30T01:37:38.998 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:38.975+0000 7f4c7515f700 1 -- --> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] -- mon_getmap magic: 0 v1 -- 0x7f4c70097d90 con 0x7f4c70133c80 2023-09-30T01:37:38.998 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:38.975+0000 7f4c7515f700 10 monclient(hunting): _renew_subs 2023-09-30T01:37:38.998 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:38.975+0000 7f4c7515f700 10 monclient(hunting): authenticate will time out at 1131.016516s 2023-09-30T01:37:38.999 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:38.975+0000 7f4c6f59e700 1 --2- >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f4c70133c80 0x7f4c7011df90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:37:38.999 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:38.975+0000 7f4c6f59e700 1 --2- >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f4c70133c80 0x7f4c7011df90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3304/0 says I am v2:172.21.15.88:48972/0 (socket says 172.21.15.88:48972) 2023-09-30T01:37:38.999 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:38.975+0000 7f4c6ed9d700 1 --2- >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7f4c700a35f0 0x7f4c7011d510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:37:38.999 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:38.975+0000 7f4c6f59e700 1 -- 172.21.15.88:0/773137368 learned_addr learned my addr 172.21.15.88:0/773137368 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-09-30T01:37:38.999 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:38.975+0000 7f4c6f59e700 10 monclient(hunting): get_auth_request con 0x7f4c70133c80 auth_method 0 2023-09-30T01:37:38.999 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:38.975+0000 7f4c6f59e700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:37:38.999 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:38.975+0000 7f4c6f59e700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:37:38.999 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:38.975+0000 7f4c6f59e700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:37:38.999 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:38.975+0000 7f4c6ed9d700 10 monclient(hunting): get_auth_request con 0x7f4c700a35f0 auth_method 0 2023-09-30T01:37:38.999 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:38.975+0000 7f4c6ed9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:37:38.999 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:38.975+0000 7f4c6ed9d700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:37:38.999 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:38.975+0000 7f4c6ed9d700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:37:39.000 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:38.975+0000 7f4c6f59e700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:37:39.000 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:38.975+0000 7f4c6f59e700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:37:39.000 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:38.975+0000 7f4c6e59c700 1 --2- 172.21.15.88:0/773137368 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7f4c70133870 0x7f4c7011da50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:37:39.000 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:38.975+0000 7f4c6f59e700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:37:39.000 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:38.975+0000 7f4c6ed9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:37:39.000 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:38.975+0000 7f4c6ed9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:37:39.000 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:38.975+0000 7f4c6ed9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:37:39.000 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:38.975+0000 7f4c6e59c700 10 monclient(hunting): get_auth_request con 0x7f4c70133870 auth_method 0 2023-09-30T01:37:39.000 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:38.975+0000 7f4c6e59c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:37:39.000 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:38.975+0000 7f4c6e59c700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:37:39.000 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:38.975+0000 7f4c6e59c700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:37:39.001 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:38.975+0000 7f4c6e59c700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:37:39.001 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:38.975+0000 7f4c6e59c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:37:39.001 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:38.975+0000 7f4c6e59c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:37:39.001 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:38.975+0000 7f4c6f59e700 10 monclient(hunting): handle_auth_done global_id 4823 payload 824 2023-09-30T01:37:39.001 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:38.975+0000 7f4c6f59e700 1 -- 172.21.15.88:0/773137368 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7f4c700a35f0 msgr2=0x7f4c7011d510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:37:39.001 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:38.975+0000 7f4c6f59e700 1 --2- 172.21.15.88:0/773137368 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7f4c700a35f0 0x7f4c7011d510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:37:39.001 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:38.975+0000 7f4c6f59e700 1 -- 172.21.15.88:0/773137368 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7f4c70133870 msgr2=0x7f4c7011da50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:37:39.001 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:38.975+0000 7f4c6f59e700 1 --2- 172.21.15.88:0/773137368 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7f4c70133870 0x7f4c7011da50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:37:39.001 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:38.975+0000 7f4c6f59e700 10 monclient: _finish_hunting 0 2023-09-30T01:37:39.001 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:38.975+0000 7f4c6f59e700 1 monclient: found mon.noname-e 2023-09-30T01:37:39.001 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:38.975+0000 7f4c6f59e700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:37:39.001 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:38.975+0000 7f4c6f59e700 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.88:3304/0 2023-09-30T01:37:39.002 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:38.975+0000 7f4c6f59e700 1 -- 172.21.15.88:0/773137368 --> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c68009420 con 0x7f4c70133c80 2023-09-30T01:37:39.002 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:38.975+0000 7f4c6ed9d700 1 --2- 172.21.15.88:0/773137368 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7f4c700a35f0 0x7f4c7011d510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:37:39.002 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:38.975+0000 7f4c6f59e700 1 --2- 172.21.15.88:0/773137368 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f4c70133c80 0x7f4c7011df90 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f4c68009fb0 tx=0x7f4c68017000 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:37:39.002 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:38.975+0000 7f4c56ffd700 1 -- 172.21.15.88:0/773137368 <== mon.8 v2:172.21.15.88:3304/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f4c6801a450 con 0x7f4c70133c80 2023-09-30T01:37:39.002 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:38.975+0000 7f4c56ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:37:39.002 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:38.975+0000 7f4c56ffd700 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2023-09-30T01:37:39.002 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:38.975+0000 7f4c56ffd700 10 monclient: dump: 2023-09-30T01:37:39.002 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:37:39.002 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:37:39.002 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:37:39.002 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:37:39.003 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:37:39.003 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:37:39.003 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:37:39.003 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:37:39.003 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:37:39.003 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:37:39.003 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:37:39.003 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:37:39.003 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:37:39.003 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:37:39.003 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:37:39.003 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:37:39.004 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:38.975+0000 7f4c56ffd700 10 monclient: _finish_auth 0 2023-09-30T01:37:39.004 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:38.975+0000 7f4c56ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:37:39.004 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:38.975+0000 7f4c56ffd700 1 -- 172.21.15.88:0/773137368 <== mon.8 v2:172.21.15.88:3304/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c6801a5d0 con 0x7f4c70133c80 2023-09-30T01:37:39.004 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:38.975+0000 7f4c7515f700 5 monclient: authenticate success, global_id 4823 2023-09-30T01:37:39.004 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:38.975+0000 7f4c56ffd700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:37:39.004 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:38.975+0000 7f4c56ffd700 1 -- 172.21.15.88:0/773137368 <== mon.8 v2:172.21.15.88:3304/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f4c6801abc0 con 0x7f4c70133c80 2023-09-30T01:37:39.004 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:38.975+0000 7f4c7515f700 10 monclient: _renew_subs 2023-09-30T01:37:39.004 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:38.975+0000 7f4c7515f700 10 monclient: _send_mon_message to mon.e at v2:172.21.15.88:3304/0 2023-09-30T01:37:39.004 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:38.975+0000 7f4c7515f700 1 -- 172.21.15.88:0/773137368 --> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c7011f2d0 con 0x7f4c70133c80 2023-09-30T01:37:39.004 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:38.975+0000 7f4c6d59a700 4 set_mon_vals no callback set 2023-09-30T01:37:39.004 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:38.975+0000 7f4c56ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:37:39.004 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:38.975+0000 7f4c56ffd700 10 monclient: got monmap 1 from mon.e (according to old e1) 2023-09-30T01:37:39.004 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:38.975+0000 7f4c56ffd700 10 monclient: dump: 2023-09-30T01:37:39.005 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:37:39.005 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:37:39.005 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:37:39.005 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:37:39.005 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:37:39.005 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:37:39.005 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:37:39.005 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:37:39.005 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:37:39.005 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:37:39.005 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:37:39.005 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:37:39.006 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:37:39.006 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:37:39.006 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:37:39.006 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:37:39.006 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:38.975+0000 7f4c7515f700 10 monclient: _renew_subs 2023-09-30T01:37:39.006 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:38.975+0000 7f4c7515f700 10 monclient: _send_mon_message to mon.e at v2:172.21.15.88:3304/0 2023-09-30T01:37:39.006 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:38.975+0000 7f4c7515f700 1 -- 172.21.15.88:0/773137368 --> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c70120880 con 0x7f4c70133c80 2023-09-30T01:37:39.006 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:38.975+0000 7f4c7515f700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:37:39.006 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:38.975+0000 7f4c7515f700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:37:39.006 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:38.975+0000 7f4c7515f700 10 monclient: _send_mon_message to mon.e at v2:172.21.15.88:3304/0 2023-09-30T01:37:39.006 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:38.975+0000 7f4c7515f700 1 -- 172.21.15.88:0/773137368 --> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4c7011ee30 con 0x7f4c70133c80 2023-09-30T01:37:39.006 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:38.975+0000 7f4c56ffd700 1 -- 172.21.15.88:0/773137368 <== mon.8 v2:172.21.15.88:3304/0 4 ==== mgrmap(e 5) v1 ==== 40144+0+0 (secure 0 0 0) 0x7f4c6801ad40 con 0x7f4c70133c80 2023-09-30T01:37:39.007 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:38.979+0000 7f4c56ffd700 1 --2- 172.21.15.88:0/773137368 >> v2:172.21.15.107:6812/29926 conn(0x7f4c5c033360 0x7f4c5c035810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:37:39.007 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:38.979+0000 7f4c56ffd700 1 -- 172.21.15.88:0/773137368 <== mon.8 v2:172.21.15.88:3304/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 3502+0+0 (secure 0 0 0) 0x7f4c68044f90 con 0x7f4c70133c80 2023-09-30T01:37:39.007 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:38.979+0000 7f4c6ed9d700 1 --2- 172.21.15.88:0/773137368 >> v2:172.21.15.107:6812/29926 conn(0x7f4c5c033360 0x7f4c5c035810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:37:39.007 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:38.979+0000 7f4c6ed9d700 10 monclient: get_auth_request con 0x7f4c5c033360 auth_method 0 2023-09-30T01:37:39.007 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:38.979+0000 7f4c6ed9d700 1 --2- 172.21.15.88:0/773137368 >> v2:172.21.15.107:6812/29926 conn(0x7f4c5c033360 0x7f4c5c035810 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f4c64011480 tx=0x7f4c640091a0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:37:39.007 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:38.983+0000 7f4c56ffd700 1 -- 172.21.15.88:0/773137368 <== mon.8 v2:172.21.15.88:3304/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f4c7011ee30 con 0x7f4c70133c80 2023-09-30T01:37:39.008 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:38.983+0000 7f4c56ffd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:37:39.008 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:38.983+0000 7f4c56ffd700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:37:39.184 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:39.179+0000 7f4c7515f700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T01:37:39.184 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:39.179+0000 7f4c7515f700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:37:39.185 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:39.179+0000 7f4c7515f700 10 monclient: _send_mon_message to mon.e at v2:172.21.15.88:3304/0 2023-09-30T01:37:39.185 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:39.179+0000 7f4c7515f700 1 -- 172.21.15.88:0/773137368 --> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f4c7011ee30 con 0x7f4c70133c80 2023-09-30T01:37:39.185 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:39.179+0000 7f4c56ffd700 1 -- 172.21.15.88:0/773137368 <== mon.8 v2:172.21.15.88:3304/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f4c7011ee30 con 0x7f4c70133c80 2023-09-30T01:37:39.185 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:39.179+0000 7f4c56ffd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:37:39.185 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:39.179+0000 7f4c56ffd700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:37:39.186 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-09-30T01:37:39.186 INFO:teuthology.orchestra.run.smithi088.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:37:39.190 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:39.183+0000 7f4c7515f700 1 -- 172.21.15.88:0/773137368 >> v2:172.21.15.107:6812/29926 conn(0x7f4c5c033360 msgr2=0x7f4c5c035810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:37:39.190 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:39.183+0000 7f4c7515f700 1 --2- 172.21.15.88:0/773137368 >> v2:172.21.15.107:6812/29926 conn(0x7f4c5c033360 0x7f4c5c035810 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f4c64011480 tx=0x7f4c640091a0 comp rx=0 tx=0).stop 2023-09-30T01:37:39.190 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:39.183+0000 7f4c7515f700 10 monclient: shutdown 2023-09-30T01:37:39.190 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:39.183+0000 7f4c7515f700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:37:39.190 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:39.183+0000 7f4c7515f700 1 -- 172.21.15.88:0/773137368 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f4c70133c80 msgr2=0x7f4c7011df90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:37:39.190 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:39.183+0000 7f4c7515f700 1 --2- 172.21.15.88:0/773137368 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f4c70133c80 0x7f4c7011df90 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f4c68009fb0 tx=0x7f4c68017000 comp rx=0 tx=0).stop 2023-09-30T01:37:39.191 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:39.183+0000 7f4c7515f700 1 -- 172.21.15.88:0/773137368 shutdown_connections 2023-09-30T01:37:39.193 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:39.183+0000 7f4c7515f700 1 --2- 172.21.15.88:0/773137368 >> v2:172.21.15.107:6812/29926 conn(0x7f4c5c033360 0x7f4c5c035810 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:37:39.193 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:39.183+0000 7f4c7515f700 1 --2- 172.21.15.88:0/773137368 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f4c70133c80 0x7f4c7011df90 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:37:39.194 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:39.183+0000 7f4c7515f700 1 --2- 172.21.15.88:0/773137368 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7f4c70133870 0x7f4c7011da50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:37:39.194 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:39.183+0000 7f4c7515f700 1 --2- 172.21.15.88:0/773137368 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7f4c700a35f0 0x7f4c7011d510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:37:39.194 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:39.183+0000 7f4c7515f700 1 -- 172.21.15.88:0/773137368 >> 172.21.15.88:0/773137368 conn(0x7f4c7006e5e0 msgr2=0x7f4c700a1150 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:37:39.195 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:39.183+0000 7f4c7515f700 1 -- 172.21.15.88:0/773137368 shutdown_connections 2023-09-30T01:37:39.195 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:39.183+0000 7f4c7515f700 1 -- 172.21.15.88:0/773137368 wait complete. 2023-09-30T01:37:42.207 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:37:42.315 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:42.307+0000 7fc2db531700 10 monclient: get_monmap_and_config 2023-09-30T01:37:42.316 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:42.307+0000 7fc2db531700 10 monclient: build_initial_monmap 2023-09-30T01:37:42.316 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:42.307+0000 7fc2db531700 10 monclient: monmap: 2023-09-30T01:37:42.316 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2023-09-30T01:37:42.316 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:37:42.316 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:37:42.315028+0000 2023-09-30T01:37:42.316 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:37:42.315028+0000 2023-09-30T01:37:42.316 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2023-09-30T01:37:42.316 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:37:42.316 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a-legacy 2023-09-30T01:37:42.316 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.107:6789/0 mon.noname-f-legacy 2023-09-30T01:37:42.317 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.noname-b 2023-09-30T01:37:42.317 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.noname-g 2023-09-30T01:37:42.317 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.noname-c 2023-09-30T01:37:42.317 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.noname-h 2023-09-30T01:37:42.317 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.noname-d 2023-09-30T01:37:42.317 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.noname-i 2023-09-30T01:37:42.317 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.noname-e 2023-09-30T01:37:42.317 INFO:teuthology.orchestra.run.smithi088.stderr:9: v2:172.21.15.88:3300/0 mon.noname-a 2023-09-30T01:37:42.317 INFO:teuthology.orchestra.run.smithi088.stderr:10: v2:172.21.15.107:3300/0 mon.noname-f 2023-09-30T01:37:42.317 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:37:42.325 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:42.311+0000 7fc2db531700 1 Processor -- start 2023-09-30T01:37:42.325 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:42.311+0000 7fc2db531700 1 -- start start 2023-09-30T01:37:42.325 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:42.311+0000 7fc2db531700 10 monclient: init 2023-09-30T01:37:42.325 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:42.311+0000 7fc2db531700 10 monclient: _reopen_session rank -1 2023-09-30T01:37:42.325 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:42.311+0000 7fc2db531700 10 monclient: _add_conns ranks=[8,3,1,4,2,5,10,6,0,9,7] 2023-09-30T01:37:42.325 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:42.311+0000 7fc2db531700 1 --2- >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7fc2d40a3380 0x7fc2d40a48b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:37:42.325 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:42.311+0000 7fc2db531700 10 monclient(hunting): picked mon.noname-e con 0x7fc2d40a3380 addr [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] 2023-09-30T01:37:42.325 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:42.311+0000 7fc2db531700 1 --2- >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7fc2d409ffd0 0x7fc2d40a03a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:37:42.325 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:42.311+0000 7fc2db531700 10 monclient(hunting): picked mon.noname-g con 0x7fc2d409ffd0 addr [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] 2023-09-30T01:37:42.325 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:42.311+0000 7fc2db531700 10 monclient(hunting): picked mon.noname-f-legacy con 0x7fc2d41333b0 addr v1:172.21.15.107:6789/0 2023-09-30T01:37:42.325 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:42.311+0000 7fc2db531700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2d404bb20 con 0x7fc2d41333b0 2023-09-30T01:37:42.326 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:42.311+0000 7fc2db531700 10 monclient(hunting): start opening mon connection 2023-09-30T01:37:42.326 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:42.311+0000 7fc2db531700 1 -- --> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2d4097ab0 con 0x7fc2d409ffd0 2023-09-30T01:37:42.326 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:42.311+0000 7fc2db531700 10 monclient(hunting): start opening mon connection 2023-09-30T01:37:42.326 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:42.311+0000 7fc2db531700 1 -- --> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] -- mon_getmap magic: 0 v1 -- 0x7fc2d40a08e0 con 0x7fc2d40a3380 2023-09-30T01:37:42.326 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:42.311+0000 7fc2db531700 10 monclient(hunting): _renew_subs 2023-09-30T01:37:42.326 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:42.311+0000 7fc2db531700 10 monclient(hunting): authenticate will time out at 1134.353139s 2023-09-30T01:37:42.326 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:42.311+0000 7fc2da52f700 1 --2- >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7fc2d40a3380 0x7fc2d40a48b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:37:42.326 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:42.311+0000 7fc2d9d2e700 1 --2- >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7fc2d409ffd0 0x7fc2d40a03a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:37:42.326 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:42.311+0000 7fc2da52f700 1 --2- >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7fc2d40a3380 0x7fc2d40a48b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3304/0 says I am v2:172.21.15.88:48978/0 (socket says 172.21.15.88:48978) 2023-09-30T01:37:42.326 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:42.311+0000 7fc2da52f700 1 -- 172.21.15.88:0/3275952259 learned_addr learned my addr 172.21.15.88:0/3275952259 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-09-30T01:37:42.326 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:42.311+0000 7fc2da52f700 10 monclient(hunting): get_auth_request con 0x7fc2d40a3380 auth_method 0 2023-09-30T01:37:42.326 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:42.311+0000 7fc2da52f700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:37:42.327 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:42.311+0000 7fc2da52f700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:37:42.327 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:42.311+0000 7fc2da52f700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:37:42.327 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:42.311+0000 7fc2d9d2e700 10 monclient(hunting): get_auth_request con 0x7fc2d409ffd0 auth_method 0 2023-09-30T01:37:42.327 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:42.311+0000 7fc2d9d2e700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:37:42.327 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:42.311+0000 7fc2d9d2e700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:37:42.327 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:42.311+0000 7fc2d9d2e700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:37:42.327 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:42.311+0000 7fc2da52f700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:37:42.327 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:42.311+0000 7fc2da52f700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:37:42.327 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:42.311+0000 7fc2da52f700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:37:42.328 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:42.311+0000 7fc2d9d2e700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:37:42.328 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:42.311+0000 7fc2d9d2e700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:37:42.328 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:42.311+0000 7fc2d9d2e700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:37:42.328 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:42.311+0000 7fc2da52f700 10 monclient(hunting): handle_auth_done global_id 4832 payload 274 2023-09-30T01:37:42.328 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:42.311+0000 7fc2da52f700 1 -- 172.21.15.88:0/3275952259 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7fc2d409ffd0 msgr2=0x7fc2d40a03a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:37:42.328 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:42.311+0000 7fc2da52f700 1 --2- 172.21.15.88:0/3275952259 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7fc2d409ffd0 0x7fc2d40a03a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:37:42.328 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:42.311+0000 7fc2da52f700 1 -- 172.21.15.88:0/3275952259 >> v1:172.21.15.107:6789/0 conn(0x7fc2d41333b0 legacy=0x7fc2d4133800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:37:42.328 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:42.311+0000 7fc2da52f700 10 monclient: _finish_hunting 0 2023-09-30T01:37:42.328 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:42.311+0000 7fc2da52f700 1 monclient: found mon.noname-e 2023-09-30T01:37:42.328 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:42.311+0000 7fc2da52f700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:37:42.329 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:42.311+0000 7fc2da52f700 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.88:3304/0 2023-09-30T01:37:42.329 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:42.311+0000 7fc2da52f700 1 -- 172.21.15.88:0/3275952259 --> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2d4098480 con 0x7fc2d40a3380 2023-09-30T01:37:42.329 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:42.311+0000 7fc2da52f700 1 --2- 172.21.15.88:0/3275952259 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7fc2d40a3380 0x7fc2d40a48b0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fc2c80099e0 tx=0x7fc2c8009250 comp rx=0 tx=0).ready entity=mon.8 client_cookie=446591903eda1061 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:37:42.329 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:42.311+0000 7fc2d952d700 1 -- 172.21.15.88:0/3275952259 <== mon.8 v2:172.21.15.88:3304/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7fc2c800bce0 con 0x7fc2d40a3380 2023-09-30T01:37:42.329 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:42.311+0000 7fc2d952d700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:37:42.329 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:42.311+0000 7fc2d952d700 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2023-09-30T01:37:42.329 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:42.311+0000 7fc2d952d700 10 monclient: dump: 2023-09-30T01:37:42.329 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:37:42.329 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:37:42.329 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:37:42.329 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:37:42.329 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:37:42.330 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:37:42.330 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:37:42.330 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:37:42.330 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:37:42.330 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:37:42.330 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:37:42.330 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:37:42.330 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:37:42.330 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:37:42.330 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:37:42.331 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:37:42.331 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:42.311+0000 7fc2d952d700 10 monclient: _finish_auth 0 2023-09-30T01:37:42.331 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:42.311+0000 7fc2d952d700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:37:42.331 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:42.311+0000 7fc2d952d700 1 -- 172.21.15.88:0/3275952259 <== mon.8 v2:172.21.15.88:3304/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc2c800be60 con 0x7fc2d40a3380 2023-09-30T01:37:42.331 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:42.311+0000 7fc2db531700 5 monclient: authenticate success, global_id 4832 2023-09-30T01:37:42.331 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:42.311+0000 7fc2db531700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:37:42.331 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:42.311+0000 7fc2d952d700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:37:42.331 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:42.311+0000 7fc2db531700 10 monclient: get_monmap_and_config success 2023-09-30T01:37:42.331 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:42.311+0000 7fc2d952d700 1 -- 172.21.15.88:0/3275952259 <== mon.8 v2:172.21.15.88:3304/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7fc2c8011420 con 0x7fc2d40a3380 2023-09-30T01:37:42.332 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:42.311+0000 7fc2d952d700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:37:42.332 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:42.311+0000 7fc2db531700 4 set_mon_vals no callback set 2023-09-30T01:37:42.332 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:42.311+0000 7fc2d952d700 10 monclient: got monmap 1 from mon.e (according to old e1) 2023-09-30T01:37:42.332 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:42.311+0000 7fc2d952d700 10 monclient: dump: 2023-09-30T01:37:42.332 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:37:42.332 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:37:42.332 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:37:42.332 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:37:42.332 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:37:42.332 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:37:42.332 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:37:42.332 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:37:42.333 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:37:42.333 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:37:42.333 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:37:42.333 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:37:42.333 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:37:42.333 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:37:42.333 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:37:42.334 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:37:42.334 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:42.311+0000 7fc2db531700 10 monclient: shutdown 2023-09-30T01:37:42.334 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:42.311+0000 7fc2db531700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:37:42.334 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:42.311+0000 7fc2db531700 1 -- 172.21.15.88:0/3275952259 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7fc2d40a3380 msgr2=0x7fc2d40a48b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:37:42.334 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:42.311+0000 7fc2db531700 1 --2- 172.21.15.88:0/3275952259 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7fc2d40a3380 0x7fc2d40a48b0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fc2c80099e0 tx=0x7fc2c8009250 comp rx=0 tx=0).stop 2023-09-30T01:37:42.335 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:42.311+0000 7fc2db531700 1 -- 172.21.15.88:0/3275952259 shutdown_connections 2023-09-30T01:37:42.335 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:42.311+0000 7fc2db531700 1 --2- 172.21.15.88:0/3275952259 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7fc2d409ffd0 0x7fc2d40a03a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:37:42.335 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:42.311+0000 7fc2db531700 1 --2- 172.21.15.88:0/3275952259 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7fc2d40a3380 0x7fc2d40a48b0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:37:42.335 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:42.311+0000 7fc2db531700 1 -- 172.21.15.88:0/3275952259 >> 172.21.15.88:0/3275952259 conn(0x7fc2d406e5e0 msgr2=0x7fc2d406e9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:37:42.335 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:42.311+0000 7fc2db531700 1 -- 172.21.15.88:0/3275952259 shutdown_connections 2023-09-30T01:37:42.335 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:42.311+0000 7fc2db531700 1 -- 172.21.15.88:0/3275952259 wait complete. 2023-09-30T01:37:42.335 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:42.315+0000 7fc2db531700 10 monclient: build_initial_monmap 2023-09-30T01:37:42.335 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:42.315+0000 7fc2db531700 10 monclient: monmap: 2023-09-30T01:37:42.335 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2023-09-30T01:37:42.335 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:37:42.335 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 0.000000 2023-09-30T01:37:42.336 INFO:teuthology.orchestra.run.smithi088.stderr:created 0.000000 2023-09-30T01:37:42.336 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2023-09-30T01:37:42.336 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 1 2023-09-30T01:37:42.336 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.noname-a 2023-09-30T01:37:42.336 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.noname-f 2023-09-30T01:37:42.336 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.noname-b 2023-09-30T01:37:42.336 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.noname-g 2023-09-30T01:37:42.336 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.noname-c 2023-09-30T01:37:42.336 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.noname-h 2023-09-30T01:37:42.336 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.noname-d 2023-09-30T01:37:42.336 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.noname-i 2023-09-30T01:37:42.336 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.noname-e 2023-09-30T01:37:42.337 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:37:42.337 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:42.315+0000 7fc2db531700 1 Processor -- start 2023-09-30T01:37:42.337 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:42.315+0000 7fc2db531700 1 -- start start 2023-09-30T01:37:42.337 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:42.315+0000 7fc2db531700 10 monclient: init 2023-09-30T01:37:42.337 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:42.315+0000 7fc2db531700 10 monclient: _reopen_session rank -1 2023-09-30T01:37:42.337 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:42.315+0000 7fc2db531700 10 monclient: _add_conns ranks=[7,2,8,5,3,1,4,0,6] 2023-09-30T01:37:42.338 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:42.315+0000 7fc2db531700 1 --2- >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7fc2d409ffd0 0x7fc2d411f230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:37:42.338 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:42.315+0000 7fc2db531700 10 monclient(hunting): picked mon.noname-i con 0x7fc2d409ffd0 addr [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] 2023-09-30T01:37:42.338 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:42.315+0000 7fc2db531700 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc2d40a3380 0x7fc2d411f770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:37:42.338 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:42.315+0000 7fc2db531700 10 monclient(hunting): picked mon.noname-b con 0x7fc2d40a3380 addr [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] 2023-09-30T01:37:42.338 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:42.315+0000 7fc2db531700 1 --2- >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7fc2d41333b0 0x7fc2d411fcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:37:42.338 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:42.315+0000 7fc2db531700 10 monclient(hunting): picked mon.noname-e con 0x7fc2d41333b0 addr [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] 2023-09-30T01:37:42.338 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:42.315+0000 7fc2db531700 10 monclient(hunting): start opening mon connection 2023-09-30T01:37:42.339 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:42.315+0000 7fc2db531700 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2d40a0ac0 con 0x7fc2d40a3380 2023-09-30T01:37:42.339 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:42.315+0000 7fc2db531700 10 monclient(hunting): start opening mon connection 2023-09-30T01:37:42.339 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:42.315+0000 7fc2db531700 1 -- --> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] -- mon_getmap magic: 0 v1 -- 0x7fc2d4097ce0 con 0x7fc2d409ffd0 2023-09-30T01:37:42.339 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:42.315+0000 7fc2db531700 10 monclient(hunting): start opening mon connection 2023-09-30T01:37:42.339 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:42.315+0000 7fc2db531700 1 -- --> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] -- mon_getmap magic: 0 v1 -- 0x7fc2d4122980 con 0x7fc2d41333b0 2023-09-30T01:37:42.339 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:42.315+0000 7fc2db531700 10 monclient(hunting): _renew_subs 2023-09-30T01:37:42.339 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:42.315+0000 7fc2db531700 10 monclient(hunting): authenticate will time out at 1134.357881s 2023-09-30T01:37:42.339 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:42.315+0000 7fc2d9d2e700 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc2d40a3380 0x7fc2d411f770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:37:42.339 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:42.315+0000 7fc2d9d2e700 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc2d40a3380 0x7fc2d411f770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3301/0 says I am v2:172.21.15.88:49278/0 (socket says 172.21.15.88:49278) 2023-09-30T01:37:42.339 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:42.315+0000 7fc2dad30700 1 --2- 172.21.15.88:0/289525147 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7fc2d41333b0 0x7fc2d411fcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:37:42.339 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:42.315+0000 7fc2d9d2e700 1 -- 172.21.15.88:0/289525147 learned_addr learned my addr 172.21.15.88:0/289525147 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-09-30T01:37:42.339 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:42.315+0000 7fc2d9d2e700 10 monclient(hunting): get_auth_request con 0x7fc2d40a3380 auth_method 0 2023-09-30T01:37:42.340 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:42.315+0000 7fc2d9d2e700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:37:42.340 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:42.315+0000 7fc2d9d2e700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:37:42.340 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:42.315+0000 7fc2d9d2e700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:37:42.340 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:42.315+0000 7fc2dad30700 10 monclient(hunting): get_auth_request con 0x7fc2d41333b0 auth_method 0 2023-09-30T01:37:42.340 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:42.315+0000 7fc2dad30700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:37:42.340 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:42.315+0000 7fc2dad30700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:37:42.340 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:42.315+0000 7fc2dad30700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:37:42.340 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:42.315+0000 7fc2da52f700 1 --2- 172.21.15.88:0/289525147 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7fc2d409ffd0 0x7fc2d411f230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:37:42.340 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:42.315+0000 7fc2da52f700 10 monclient(hunting): get_auth_request con 0x7fc2d409ffd0 auth_method 0 2023-09-30T01:37:42.340 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:42.315+0000 7fc2da52f700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:37:42.340 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:42.315+0000 7fc2da52f700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:37:42.340 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:42.315+0000 7fc2da52f700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:37:42.341 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:42.315+0000 7fc2dad30700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:37:42.341 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:42.315+0000 7fc2dad30700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:37:42.341 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:42.315+0000 7fc2dad30700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:37:42.341 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:42.315+0000 7fc2d9d2e700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:37:42.341 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:42.315+0000 7fc2d9d2e700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:37:42.341 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:42.315+0000 7fc2d9d2e700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:37:42.341 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:42.315+0000 7fc2da52f700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:37:42.341 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:42.315+0000 7fc2da52f700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:37:42.341 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:42.315+0000 7fc2da52f700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:37:42.341 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:42.315+0000 7fc2dad30700 10 monclient(hunting): handle_auth_done global_id 4841 payload 824 2023-09-30T01:37:42.341 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:42.315+0000 7fc2dad30700 1 -- 172.21.15.88:0/289525147 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7fc2d409ffd0 msgr2=0x7fc2d411f230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:37:42.341 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:42.315+0000 7fc2dad30700 1 --2- 172.21.15.88:0/289525147 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7fc2d409ffd0 0x7fc2d411f230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:37:42.341 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:42.315+0000 7fc2dad30700 1 -- 172.21.15.88:0/289525147 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc2d40a3380 msgr2=0x7fc2d411f770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:37:42.342 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:42.315+0000 7fc2dad30700 1 --2- 172.21.15.88:0/289525147 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc2d40a3380 0x7fc2d411f770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:37:42.342 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:42.315+0000 7fc2dad30700 10 monclient: _finish_hunting 0 2023-09-30T01:37:42.342 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:42.315+0000 7fc2dad30700 1 monclient: found mon.noname-e 2023-09-30T01:37:42.342 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:42.315+0000 7fc2dad30700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:37:42.342 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:42.315+0000 7fc2dad30700 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.88:3304/0 2023-09-30T01:37:42.342 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:42.315+0000 7fc2dad30700 1 -- 172.21.15.88:0/289525147 --> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2c80094f0 con 0x7fc2d41333b0 2023-09-30T01:37:42.342 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:42.315+0000 7fc2dad30700 1 --2- 172.21.15.88:0/289525147 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7fc2d41333b0 0x7fc2d411fcb0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fc2d000c4b0 tx=0x7fc2d0007e60 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:37:42.342 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:42.315+0000 7fc2c67fc700 1 -- 172.21.15.88:0/289525147 <== mon.8 v2:172.21.15.88:3304/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7fc2d000ce60 con 0x7fc2d41333b0 2023-09-30T01:37:42.342 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:42.315+0000 7fc2c67fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:37:42.342 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:42.315+0000 7fc2c67fc700 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2023-09-30T01:37:42.342 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:42.315+0000 7fc2c67fc700 10 monclient: dump: 2023-09-30T01:37:42.342 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:37:42.343 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:37:42.343 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:37:42.343 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:37:42.343 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:37:42.343 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:37:42.343 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:37:42.343 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:37:42.343 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:37:42.343 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:37:42.343 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:37:42.343 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:37:42.343 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:37:42.343 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:37:42.344 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:37:42.344 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:37:42.344 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:42.315+0000 7fc2c67fc700 10 monclient: _finish_auth 0 2023-09-30T01:37:42.344 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:42.315+0000 7fc2c67fc700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:37:42.344 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:42.315+0000 7fc2db531700 5 monclient: authenticate success, global_id 4841 2023-09-30T01:37:42.344 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:42.315+0000 7fc2c67fc700 1 -- 172.21.15.88:0/289525147 <== mon.8 v2:172.21.15.88:3304/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc2d0011a10 con 0x7fc2d41333b0 2023-09-30T01:37:42.344 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:42.315+0000 7fc2db531700 10 monclient: _renew_subs 2023-09-30T01:37:42.344 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:42.315+0000 7fc2db531700 10 monclient: _send_mon_message to mon.e at v2:172.21.15.88:3304/0 2023-09-30T01:37:42.344 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:42.315+0000 7fc2db531700 1 -- 172.21.15.88:0/289525147 --> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc2d41226a0 con 0x7fc2d41333b0 2023-09-30T01:37:42.344 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:42.315+0000 7fc2c67fc700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:37:42.344 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:42.315+0000 7fc2c67fc700 1 -- 172.21.15.88:0/289525147 <== mon.8 v2:172.21.15.88:3304/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7fc2d00102e0 con 0x7fc2d41333b0 2023-09-30T01:37:42.344 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:42.315+0000 7fc2c67fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:37:42.345 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:42.315+0000 7fc2c67fc700 10 monclient: got monmap 1 from mon.e (according to old e1) 2023-09-30T01:37:42.345 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:42.315+0000 7fc2c67fc700 10 monclient: dump: 2023-09-30T01:37:42.345 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:37:42.345 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:37:42.345 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:37:42.345 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:37:42.345 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:37:42.345 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:37:42.345 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:37:42.345 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:37:42.345 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:37:42.345 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:37:42.345 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:37:42.346 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:37:42.346 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:37:42.346 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:37:42.346 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:37:42.346 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:37:42.346 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:42.315+0000 7fc2db531700 10 monclient: _renew_subs 2023-09-30T01:37:42.346 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:42.315+0000 7fc2db531700 10 monclient: _send_mon_message to mon.e at v2:172.21.15.88:3304/0 2023-09-30T01:37:42.346 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:42.315+0000 7fc2db531700 1 -- 172.21.15.88:0/289525147 --> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2d4132730 con 0x7fc2d41333b0 2023-09-30T01:37:42.346 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:42.315+0000 7fc2d8d2c700 4 set_mon_vals no callback set 2023-09-30T01:37:42.346 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:42.319+0000 7fc2db531700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:37:42.346 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:42.319+0000 7fc2c67fc700 1 -- 172.21.15.88:0/289525147 <== mon.8 v2:172.21.15.88:3304/0 4 ==== mgrmap(e 5) v1 ==== 40144+0+0 (secure 0 0 0) 0x7fc2d0011b80 con 0x7fc2d41333b0 2023-09-30T01:37:42.346 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:42.319+0000 7fc2db531700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:37:42.347 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:42.319+0000 7fc2db531700 10 monclient: _send_mon_message to mon.e at v2:172.21.15.88:3304/0 2023-09-30T01:37:42.347 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:42.319+0000 7fc2db531700 1 -- 172.21.15.88:0/289525147 --> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc2d409d6a0 con 0x7fc2d41333b0 2023-09-30T01:37:42.347 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:42.319+0000 7fc2c67fc700 1 --2- 172.21.15.88:0/289525147 >> v2:172.21.15.107:6812/29926 conn(0x7fc2b8033290 0x7fc2b8035740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:37:42.347 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:42.319+0000 7fc2c67fc700 1 -- 172.21.15.88:0/289525147 <== mon.8 v2:172.21.15.88:3304/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 3502+0+0 (secure 0 0 0) 0x7fc2d00423d0 con 0x7fc2d41333b0 2023-09-30T01:37:42.348 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:42.319+0000 7fc2da52f700 1 --2- 172.21.15.88:0/289525147 >> v2:172.21.15.107:6812/29926 conn(0x7fc2b8033290 0x7fc2b8035740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:37:42.348 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:42.319+0000 7fc2da52f700 10 monclient: get_auth_request con 0x7fc2b8033290 auth_method 0 2023-09-30T01:37:42.348 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:42.319+0000 7fc2da52f700 1 --2- 172.21.15.88:0/289525147 >> v2:172.21.15.107:6812/29926 conn(0x7fc2b8033290 0x7fc2b8035740 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fc2c800bfd0 tx=0x7fc2c8017000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:37:42.348 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:42.327+0000 7fc2c67fc700 1 -- 172.21.15.88:0/289525147 <== mon.8 v2:172.21.15.88:3304/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fc2d0018680 con 0x7fc2d41333b0 2023-09-30T01:37:42.348 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:42.327+0000 7fc2c67fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:37:42.348 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:42.327+0000 7fc2c67fc700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:37:42.527 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:42.519+0000 7fc2db531700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T01:37:42.527 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:42.519+0000 7fc2db531700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:37:42.528 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:42.519+0000 7fc2db531700 10 monclient: _send_mon_message to mon.e at v2:172.21.15.88:3304/0 2023-09-30T01:37:42.528 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:42.519+0000 7fc2db531700 1 -- 172.21.15.88:0/289525147 --> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fc2d40710d0 con 0x7fc2d41333b0 2023-09-30T01:37:42.528 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:42.523+0000 7fc2c67fc700 1 -- 172.21.15.88:0/289525147 <== mon.8 v2:172.21.15.88:3304/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7fc2d0017020 con 0x7fc2d41333b0 2023-09-30T01:37:42.528 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:42.523+0000 7fc2c67fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:37:42.528 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:42.523+0000 7fc2c67fc700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:37:42.529 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-09-30T01:37:42.529 INFO:teuthology.orchestra.run.smithi088.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:37:42.530 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:42.523+0000 7fc2db531700 1 -- 172.21.15.88:0/289525147 >> v2:172.21.15.107:6812/29926 conn(0x7fc2b8033290 msgr2=0x7fc2b8035740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:37:42.530 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:42.523+0000 7fc2db531700 1 --2- 172.21.15.88:0/289525147 >> v2:172.21.15.107:6812/29926 conn(0x7fc2b8033290 0x7fc2b8035740 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fc2c800bfd0 tx=0x7fc2c8017000 comp rx=0 tx=0).stop 2023-09-30T01:37:42.530 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:42.523+0000 7fc2db531700 10 monclient: shutdown 2023-09-30T01:37:42.531 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:42.523+0000 7fc2db531700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:37:42.531 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:42.523+0000 7fc2db531700 1 -- 172.21.15.88:0/289525147 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7fc2d41333b0 msgr2=0x7fc2d411fcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:37:42.531 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:42.523+0000 7fc2db531700 1 --2- 172.21.15.88:0/289525147 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7fc2d41333b0 0x7fc2d411fcb0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fc2d000c4b0 tx=0x7fc2d0007e60 comp rx=0 tx=0).stop 2023-09-30T01:37:42.531 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:42.523+0000 7fc2db531700 1 -- 172.21.15.88:0/289525147 shutdown_connections 2023-09-30T01:37:42.531 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:42.523+0000 7fc2db531700 1 --2- 172.21.15.88:0/289525147 >> v2:172.21.15.107:6812/29926 conn(0x7fc2b8033290 0x7fc2b8035740 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:37:42.531 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:42.523+0000 7fc2db531700 1 --2- 172.21.15.88:0/289525147 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7fc2d41333b0 0x7fc2d411fcb0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:37:42.531 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:42.523+0000 7fc2db531700 1 --2- 172.21.15.88:0/289525147 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc2d40a3380 0x7fc2d411f770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:37:42.531 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:42.523+0000 7fc2db531700 1 --2- 172.21.15.88:0/289525147 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7fc2d409ffd0 0x7fc2d411f230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:37:42.531 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:42.523+0000 7fc2db531700 1 -- 172.21.15.88:0/289525147 >> 172.21.15.88:0/289525147 conn(0x7fc2d406e5e0 msgr2=0x7fc2d40a1cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:37:42.531 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:42.523+0000 7fc2db531700 1 -- 172.21.15.88:0/289525147 shutdown_connections 2023-09-30T01:37:42.531 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:42.523+0000 7fc2db531700 1 -- 172.21.15.88:0/289525147 wait complete. 2023-09-30T01:37:45.542 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:37:45.655 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:45.647+0000 7f3441438700 10 monclient: get_monmap_and_config 2023-09-30T01:37:45.655 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:45.647+0000 7f3441438700 10 monclient: build_initial_monmap 2023-09-30T01:37:45.655 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:45.647+0000 7f3441438700 10 monclient: monmap: 2023-09-30T01:37:45.655 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2023-09-30T01:37:45.655 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:37:45.656 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:37:45.655176+0000 2023-09-30T01:37:45.656 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:37:45.655176+0000 2023-09-30T01:37:45.656 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2023-09-30T01:37:45.656 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:37:45.656 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a-legacy 2023-09-30T01:37:45.656 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.107:6789/0 mon.noname-f-legacy 2023-09-30T01:37:45.656 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.noname-b 2023-09-30T01:37:45.656 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.noname-g 2023-09-30T01:37:45.656 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.noname-c 2023-09-30T01:37:45.656 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.noname-h 2023-09-30T01:37:45.656 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.noname-d 2023-09-30T01:37:45.656 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.noname-i 2023-09-30T01:37:45.656 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.noname-e 2023-09-30T01:37:45.657 INFO:teuthology.orchestra.run.smithi088.stderr:9: v2:172.21.15.88:3300/0 mon.noname-a 2023-09-30T01:37:45.657 INFO:teuthology.orchestra.run.smithi088.stderr:10: v2:172.21.15.107:3300/0 mon.noname-f 2023-09-30T01:37:45.657 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:37:45.663 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:45.651+0000 7f3441438700 1 Processor -- start 2023-09-30T01:37:45.663 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:45.651+0000 7f3441438700 1 -- start start 2023-09-30T01:37:45.664 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:45.651+0000 7f3441438700 10 monclient: init 2023-09-30T01:37:45.664 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:45.651+0000 7f3441438700 10 monclient: _reopen_session rank -1 2023-09-30T01:37:45.664 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:45.651+0000 7f3441438700 10 monclient: _add_conns ranks=[4,0,7,6,10,3,1,2,9,5,8] 2023-09-30T01:37:45.664 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:45.651+0000 7f3441438700 1 --2- >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f343c0a3270 0x7f343c1333a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:37:45.664 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:45.651+0000 7f3441438700 10 monclient(hunting): picked mon.noname-c con 0x7f343c0a3270 addr [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] 2023-09-30T01:37:45.664 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:45.651+0000 7f3441438700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f343c1338e0 addr v1:172.21.15.88:6789/0 2023-09-30T01:37:45.664 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:45.651+0000 7f3441438700 1 --2- >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7f343c133cf0 0x7f343c0a4d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:37:45.664 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:45.651+0000 7f3441438700 10 monclient(hunting): picked mon.noname-i con 0x7f343c133cf0 addr [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] 2023-09-30T01:37:45.664 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:45.651+0000 7f3441438700 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f343c05ecb0 con 0x7f343c1338e0 2023-09-30T01:37:45.665 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:45.651+0000 7f3441438700 10 monclient(hunting): start opening mon connection 2023-09-30T01:37:45.665 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:45.651+0000 7f3441438700 1 -- --> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] -- mon_getmap magic: 0 v1 -- 0x7f343c0a3640 con 0x7f343c0a3270 2023-09-30T01:37:45.665 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:45.651+0000 7f3441438700 10 monclient(hunting): start opening mon connection 2023-09-30T01:37:45.665 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:45.651+0000 7f3441438700 1 -- --> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] -- mon_getmap magic: 0 v1 -- 0x7f343c0981f0 con 0x7f343c133cf0 2023-09-30T01:37:45.665 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:45.651+0000 7f3441438700 10 monclient(hunting): _renew_subs 2023-09-30T01:37:45.665 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:45.651+0000 7f3441438700 10 monclient(hunting): authenticate will time out at 1137.693109s 2023-09-30T01:37:45.665 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:45.651+0000 7f343affd700 1 --2- >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f343c0a3270 0x7f343c1333a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:37:45.665 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:45.651+0000 7f343affd700 1 --2- >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f343c0a3270 0x7f343c1333a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3302/0 says I am v2:172.21.15.88:36448/0 (socket says 172.21.15.88:36448) 2023-09-30T01:37:45.665 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:45.651+0000 7f34327fc700 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f343c1338e0 0x7f343c0a45f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:40664/0 (socket says 172.21.15.88:40664) 2023-09-30T01:37:45.665 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:45.651+0000 7f343affd700 1 -- 172.21.15.88:0/4254135589 learned_addr learned my addr 172.21.15.88:0/4254135589 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-09-30T01:37:45.665 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:45.651+0000 7f343affd700 10 monclient(hunting): get_auth_request con 0x7f343c0a3270 auth_method 0 2023-09-30T01:37:45.665 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:45.651+0000 7f343affd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:37:45.665 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:45.651+0000 7f343affd700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:37:45.666 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:45.651+0000 7f343affd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:37:45.666 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:45.651+0000 7f343affd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:37:45.666 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:45.651+0000 7f343affd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:37:45.666 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:45.651+0000 7f343affd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:37:45.666 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:45.651+0000 7f343b7fe700 1 --2- 172.21.15.88:0/4254135589 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7f343c133cf0 0x7f343c0a4d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:37:45.666 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:45.651+0000 7f343b7fe700 10 monclient(hunting): get_auth_request con 0x7f343c133cf0 auth_method 0 2023-09-30T01:37:45.666 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:45.651+0000 7f343b7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:37:45.666 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:45.651+0000 7f343b7fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:37:45.666 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:45.651+0000 7f343b7fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:37:45.666 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:45.651+0000 7f343b7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:37:45.666 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:45.651+0000 7f343b7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:37:45.666 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:45.651+0000 7f343b7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:37:45.667 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:45.651+0000 7f343affd700 10 monclient(hunting): handle_auth_done global_id 4837 payload 274 2023-09-30T01:37:45.667 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:45.651+0000 7f343affd700 1 -- 172.21.15.88:0/4254135589 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7f343c133cf0 msgr2=0x7f343c0a4d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:37:45.667 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:45.651+0000 7f343affd700 1 --2- 172.21.15.88:0/4254135589 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7f343c133cf0 0x7f343c0a4d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:37:45.667 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:45.651+0000 7f343affd700 1 -- 172.21.15.88:0/4254135589 >> v1:172.21.15.88:6789/0 conn(0x7f343c1338e0 legacy=0x7f343c0a45f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:37:45.667 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:45.651+0000 7f343affd700 10 monclient: _finish_hunting 0 2023-09-30T01:37:45.667 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:45.651+0000 7f343affd700 1 monclient: found mon.noname-c 2023-09-30T01:37:45.667 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:45.651+0000 7f343affd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:37:45.667 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:45.651+0000 7f343affd700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.88:3302/0 2023-09-30T01:37:45.667 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:45.651+0000 7f343affd700 1 -- 172.21.15.88:0/4254135589 --> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f343c0984d0 con 0x7f343c0a3270 2023-09-30T01:37:45.667 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:45.651+0000 7f343a7fc700 1 -- 172.21.15.88:0/4254135589 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2209351040 0 0) 0x7f343c05ecb0 con 0x7f343c1338e0 2023-09-30T01:37:45.667 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:45.651+0000 7f343a7fc700 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2023-09-30T01:37:45.667 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:45.651+0000 7f343affd700 1 --2- 172.21.15.88:0/4254135589 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f343c0a3270 0x7f343c1333a0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f342c0099e0 tx=0x7f342c009250 comp rx=0 tx=0).ready entity=mon.4 client_cookie=8ff9e26d7d1bdb61 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:37:45.667 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:45.651+0000 7f343a7fc700 1 -- 172.21.15.88:0/4254135589 <== mon.4 v2:172.21.15.88:3302/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f342c00bce0 con 0x7f343c0a3270 2023-09-30T01:37:45.668 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:45.651+0000 7f343a7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:37:45.668 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:45.651+0000 7f343a7fc700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T01:37:45.668 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:45.651+0000 7f343a7fc700 10 monclient: dump: 2023-09-30T01:37:45.668 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:37:45.668 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:37:45.668 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:37:45.668 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:37:45.668 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:37:45.668 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:37:45.668 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:37:45.668 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:37:45.668 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:37:45.668 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:37:45.669 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:37:45.669 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:37:45.669 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:37:45.669 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:37:45.669 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:37:45.669 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:37:45.669 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:45.651+0000 7f343a7fc700 10 monclient: _finish_auth 0 2023-09-30T01:37:45.669 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:45.651+0000 7f343a7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:37:45.669 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:45.651+0000 7f3441438700 5 monclient: authenticate success, global_id 4837 2023-09-30T01:37:45.669 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:45.651+0000 7f343a7fc700 1 -- 172.21.15.88:0/4254135589 <== mon.4 v2:172.21.15.88:3302/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f342c00be60 con 0x7f343c0a3270 2023-09-30T01:37:45.669 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:45.651+0000 7f3441438700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:37:45.669 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:45.651+0000 7f343a7fc700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:37:45.670 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:45.651+0000 7f3441438700 10 monclient: get_monmap_and_config success 2023-09-30T01:37:45.670 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:45.651+0000 7f3441438700 4 set_mon_vals no callback set 2023-09-30T01:37:45.670 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:45.651+0000 7f343a7fc700 1 -- 172.21.15.88:0/4254135589 <== mon.4 v2:172.21.15.88:3302/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f342c0118a0 con 0x7f343c0a3270 2023-09-30T01:37:45.670 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:45.651+0000 7f343a7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:37:45.670 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:45.651+0000 7f343a7fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T01:37:45.670 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:45.651+0000 7f343a7fc700 10 monclient: dump: 2023-09-30T01:37:45.670 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:37:45.670 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:37:45.670 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:37:45.670 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:37:45.670 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:37:45.670 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:37:45.671 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:37:45.671 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:37:45.671 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:37:45.671 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:37:45.671 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:37:45.671 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:37:45.671 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:37:45.671 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:37:45.671 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:37:45.671 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:37:45.671 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:45.651+0000 7f3441438700 10 monclient: shutdown 2023-09-30T01:37:45.671 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:45.651+0000 7f3441438700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:37:45.671 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:45.651+0000 7f3441438700 1 -- 172.21.15.88:0/4254135589 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f343c0a3270 msgr2=0x7f343c1333a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:37:45.672 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:45.651+0000 7f3441438700 1 --2- 172.21.15.88:0/4254135589 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f343c0a3270 0x7f343c1333a0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f342c0099e0 tx=0x7f342c009250 comp rx=0 tx=0).stop 2023-09-30T01:37:45.672 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:45.651+0000 7f3441438700 1 -- 172.21.15.88:0/4254135589 shutdown_connections 2023-09-30T01:37:45.672 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:45.651+0000 7f3441438700 1 --2- 172.21.15.88:0/4254135589 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7f343c133cf0 0x7f343c0a4d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:37:45.672 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:45.651+0000 7f3441438700 1 --2- 172.21.15.88:0/4254135589 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f343c0a3270 0x7f343c1333a0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:37:45.672 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:45.651+0000 7f3441438700 1 -- 172.21.15.88:0/4254135589 >> 172.21.15.88:0/4254135589 conn(0x7f343c06e5e0 msgr2=0x7f343c06e9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:37:45.672 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:45.651+0000 7f3441438700 1 -- 172.21.15.88:0/4254135589 shutdown_connections 2023-09-30T01:37:45.672 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:45.651+0000 7f3441438700 1 -- 172.21.15.88:0/4254135589 wait complete. 2023-09-30T01:37:45.672 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:45.655+0000 7f3441438700 10 monclient: build_initial_monmap 2023-09-30T01:37:45.672 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:45.655+0000 7f3441438700 10 monclient: monmap: 2023-09-30T01:37:45.672 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2023-09-30T01:37:45.672 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:37:45.672 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 0.000000 2023-09-30T01:37:45.672 INFO:teuthology.orchestra.run.smithi088.stderr:created 0.000000 2023-09-30T01:37:45.673 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2023-09-30T01:37:45.673 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 1 2023-09-30T01:37:45.673 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.noname-a 2023-09-30T01:37:45.673 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.noname-f 2023-09-30T01:37:45.673 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.noname-b 2023-09-30T01:37:45.673 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.noname-g 2023-09-30T01:37:45.673 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.noname-c 2023-09-30T01:37:45.673 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.noname-h 2023-09-30T01:37:45.673 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.noname-d 2023-09-30T01:37:45.673 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.noname-i 2023-09-30T01:37:45.673 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.noname-e 2023-09-30T01:37:45.673 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:37:45.674 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:45.655+0000 7f3441438700 1 Processor -- start 2023-09-30T01:37:45.674 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:45.655+0000 7f3441438700 1 -- start start 2023-09-30T01:37:45.674 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:45.655+0000 7f3441438700 10 monclient: init 2023-09-30T01:37:45.674 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:45.655+0000 7f3441438700 10 monclient: _reopen_session rank -1 2023-09-30T01:37:45.674 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:45.655+0000 7f3441438700 10 monclient: _add_conns ranks=[6,8,0,3,5,2,1,4,7] 2023-09-30T01:37:45.674 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:45.655+0000 7f3441438700 1 --2- >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7f343c0a3270 0x7f343c11ce50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:37:45.674 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:45.655+0000 7f3441438700 10 monclient(hunting): picked mon.noname-d con 0x7f343c0a3270 addr [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] 2023-09-30T01:37:45.674 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:45.655+0000 7f3441438700 1 --2- >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f343c1338e0 0x7f343c11d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:37:45.675 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:45.655+0000 7f3441438700 10 monclient(hunting): picked mon.noname-e con 0x7f343c1338e0 addr [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] 2023-09-30T01:37:45.675 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:45.655+0000 7f3441438700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f343c133cf0 0x7f343c11d8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:37:45.675 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:45.655+0000 7f3441438700 10 monclient(hunting): picked mon.noname-a con 0x7f343c133cf0 addr [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] 2023-09-30T01:37:45.675 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:45.655+0000 7f3441438700 10 monclient(hunting): start opening mon connection 2023-09-30T01:37:45.675 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:45.655+0000 7f3441438700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f343c0a0bc0 con 0x7f343c133cf0 2023-09-30T01:37:45.675 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:45.655+0000 7f3441438700 10 monclient(hunting): start opening mon connection 2023-09-30T01:37:45.675 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:45.655+0000 7f3441438700 1 -- --> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] -- mon_getmap magic: 0 v1 -- 0x7f343c120200 con 0x7f343c0a3270 2023-09-30T01:37:45.675 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:45.655+0000 7f3441438700 10 monclient(hunting): start opening mon connection 2023-09-30T01:37:45.675 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:45.655+0000 7f3441438700 1 -- --> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] -- mon_getmap magic: 0 v1 -- 0x7f343c097d90 con 0x7f343c1338e0 2023-09-30T01:37:45.675 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:45.655+0000 7f3441438700 10 monclient(hunting): _renew_subs 2023-09-30T01:37:45.675 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:45.655+0000 7f3441438700 10 monclient(hunting): authenticate will time out at 1137.697441s 2023-09-30T01:37:45.675 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:45.655+0000 7f343affd700 1 --2- >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7f343c0a3270 0x7f343c11ce50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:37:45.676 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:45.655+0000 7f343b7fe700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f343c133cf0 0x7f343c11d8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:37:45.676 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:45.655+0000 7f34327fc700 1 --2- >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f343c1338e0 0x7f343c11d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:37:45.676 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:45.655+0000 7f343b7fe700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f343c133cf0 0x7f343c11d8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:47556/0 (socket says 172.21.15.88:47556) 2023-09-30T01:37:45.676 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:45.655+0000 7f343affd700 1 --2- >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7f343c0a3270 0x7f343c11ce50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3303/0 says I am v2:172.21.15.88:40176/0 (socket says 172.21.15.88:40176) 2023-09-30T01:37:45.676 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:45.655+0000 7f343b7fe700 1 -- 172.21.15.88:0/115153708 learned_addr learned my addr 172.21.15.88:0/115153708 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-09-30T01:37:45.676 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:45.655+0000 7f343b7fe700 10 monclient(hunting): get_auth_request con 0x7f343c133cf0 auth_method 0 2023-09-30T01:37:45.676 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:45.655+0000 7f343b7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:37:45.676 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:45.655+0000 7f343b7fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:37:45.676 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:45.655+0000 7f343b7fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:37:45.676 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:45.655+0000 7f343affd700 10 monclient(hunting): get_auth_request con 0x7f343c0a3270 auth_method 0 2023-09-30T01:37:45.676 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:45.655+0000 7f343affd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:37:45.676 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:45.655+0000 7f343affd700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:37:45.677 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:45.655+0000 7f343affd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:37:45.677 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:45.655+0000 7f34327fc700 10 monclient(hunting): get_auth_request con 0x7f343c1338e0 auth_method 0 2023-09-30T01:37:45.677 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:45.655+0000 7f34327fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:37:45.677 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:45.655+0000 7f34327fc700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:37:45.677 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:45.655+0000 7f34327fc700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:37:45.677 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:45.655+0000 7f343b7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:37:45.677 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:45.655+0000 7f343b7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:37:45.677 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:45.655+0000 7f343b7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:37:45.677 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:45.655+0000 7f343affd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:37:45.677 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:45.655+0000 7f343affd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:37:45.678 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:45.655+0000 7f343affd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:37:45.678 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:45.655+0000 7f34327fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:37:45.678 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:45.655+0000 7f34327fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:37:45.678 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:45.655+0000 7f34327fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:37:45.678 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:45.655+0000 7f343b7fe700 10 monclient(hunting): handle_auth_done global_id 5121 payload 824 2023-09-30T01:37:45.678 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:45.655+0000 7f343b7fe700 1 -- 172.21.15.88:0/115153708 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f343c1338e0 msgr2=0x7f343c11d390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:37:45.678 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:45.655+0000 7f343b7fe700 1 --2- 172.21.15.88:0/115153708 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f343c1338e0 0x7f343c11d390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:37:45.678 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:45.655+0000 7f343b7fe700 1 -- 172.21.15.88:0/115153708 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7f343c0a3270 msgr2=0x7f343c11ce50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:37:45.678 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:45.655+0000 7f343b7fe700 1 --2- 172.21.15.88:0/115153708 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7f343c0a3270 0x7f343c11ce50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:37:45.678 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:45.655+0000 7f343b7fe700 10 monclient: _finish_hunting 0 2023-09-30T01:37:45.678 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:45.655+0000 7f343b7fe700 1 monclient: found mon.noname-a 2023-09-30T01:37:45.678 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:45.655+0000 7f343b7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:37:45.679 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:45.655+0000 7f343b7fe700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.88:3300/0 2023-09-30T01:37:45.679 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:45.655+0000 7f343b7fe700 1 -- 172.21.15.88:0/115153708 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f342c0094f0 con 0x7f343c133cf0 2023-09-30T01:37:45.679 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:45.655+0000 7f343affd700 1 --2- 172.21.15.88:0/115153708 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7f343c0a3270 0x7f343c11ce50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:37:45.679 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:45.655+0000 7f343b7fe700 1 --2- 172.21.15.88:0/115153708 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f343c133cf0 0x7f343c11d8d0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f3434009e90 tx=0x7f343400b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:37:45.679 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:45.655+0000 7f34337fe700 1 -- 172.21.15.88:0/115153708 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f3434014480 con 0x7f343c133cf0 2023-09-30T01:37:45.679 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:45.655+0000 7f34337fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:37:45.679 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:45.655+0000 7f34337fe700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T01:37:45.679 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:45.655+0000 7f34337fe700 10 monclient: dump: 2023-09-30T01:37:45.679 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:37:45.679 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:37:45.680 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:37:45.680 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:37:45.680 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:37:45.680 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:37:45.680 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:37:45.680 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:37:45.680 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:37:45.680 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:37:45.680 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:37:45.680 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:37:45.680 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:37:45.680 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:37:45.680 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:37:45.681 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:37:45.681 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:45.655+0000 7f34337fe700 10 monclient: _finish_auth 0 2023-09-30T01:37:45.681 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:45.655+0000 7f34337fe700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:37:45.681 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:45.655+0000 7f34337fe700 1 -- 172.21.15.88:0/115153708 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f343400f040 con 0x7f343c133cf0 2023-09-30T01:37:45.681 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:45.655+0000 7f3441438700 5 monclient: authenticate success, global_id 5121 2023-09-30T01:37:45.681 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:45.655+0000 7f3441438700 10 monclient: _renew_subs 2023-09-30T01:37:45.681 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:45.655+0000 7f3441438700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.88:3300/0 2023-09-30T01:37:45.681 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:45.655+0000 7f3441438700 1 -- 172.21.15.88:0/115153708 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f343c09fe20 con 0x7f343c133cf0 2023-09-30T01:37:45.681 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:45.655+0000 7f34337fe700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:37:45.681 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:45.655+0000 7f34337fe700 1 -- 172.21.15.88:0/115153708 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f3434014aa0 con 0x7f343c133cf0 2023-09-30T01:37:45.681 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:45.655+0000 7f34337fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:37:45.681 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:45.655+0000 7f34337fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:37:45.682 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:45.655+0000 7f34337fe700 10 monclient: dump: 2023-09-30T01:37:45.682 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:37:45.682 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:37:45.682 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:37:45.682 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:37:45.682 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:37:45.682 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:37:45.682 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:37:45.682 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:37:45.682 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:37:45.682 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:37:45.682 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:37:45.682 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:37:45.683 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:37:45.683 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:37:45.683 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:37:45.683 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:37:45.683 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:45.655+0000 7f3439ffb700 4 set_mon_vals no callback set 2023-09-30T01:37:45.683 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:45.655+0000 7f3441438700 10 monclient: _renew_subs 2023-09-30T01:37:45.683 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:45.655+0000 7f3441438700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.88:3300/0 2023-09-30T01:37:45.683 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:45.655+0000 7f3441438700 1 -- 172.21.15.88:0/115153708 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f343c09f660 con 0x7f343c133cf0 2023-09-30T01:37:45.685 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:45.659+0000 7f3441438700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:37:45.685 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:45.659+0000 7f3441438700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:37:45.685 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:45.659+0000 7f3441438700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.88:3300/0 2023-09-30T01:37:45.685 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:45.659+0000 7f34337fe700 1 -- 172.21.15.88:0/115153708 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 40144+0+0 (secure 0 0 0) 0x7f3434014c20 con 0x7f343c133cf0 2023-09-30T01:37:45.685 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:45.659+0000 7f3441438700 1 -- 172.21.15.88:0/115153708 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f343c071960 con 0x7f343c133cf0 2023-09-30T01:37:45.685 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:45.659+0000 7f34337fe700 1 --2- 172.21.15.88:0/115153708 >> v2:172.21.15.107:6812/29926 conn(0x7f341c033360 0x7f341c035810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:37:45.685 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:45.659+0000 7f34337fe700 1 -- 172.21.15.88:0/115153708 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 3502+0+0 (secure 0 0 0) 0x7f3434043d50 con 0x7f343c133cf0 2023-09-30T01:37:45.685 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:45.659+0000 7f343affd700 1 --2- 172.21.15.88:0/115153708 >> v2:172.21.15.107:6812/29926 conn(0x7f341c033360 0x7f341c035810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:37:45.685 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:45.659+0000 7f343affd700 10 monclient: get_auth_request con 0x7f341c033360 auth_method 0 2023-09-30T01:37:45.686 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:45.659+0000 7f343affd700 1 --2- 172.21.15.88:0/115153708 >> v2:172.21.15.107:6812/29926 conn(0x7f341c033360 0x7f341c035810 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f342c00bfd0 tx=0x7f342c017000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:37:45.686 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:45.667+0000 7f34337fe700 1 -- 172.21.15.88:0/115153708 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f343400caa0 con 0x7f343c133cf0 2023-09-30T01:37:45.686 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:45.667+0000 7f34337fe700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:37:45.686 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:45.667+0000 7f34337fe700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:37:45.867 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:45.859+0000 7f3441438700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T01:37:45.867 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:45.859+0000 7f3441438700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:37:45.867 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:45.859+0000 7f3441438700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.88:3300/0 2023-09-30T01:37:45.867 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:45.859+0000 7f3441438700 1 -- 172.21.15.88:0/115153708 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f343c067b20 con 0x7f343c133cf0 2023-09-30T01:37:45.868 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:45.859+0000 7f34337fe700 1 -- 172.21.15.88:0/115153708 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f3434018020 con 0x7f343c133cf0 2023-09-30T01:37:45.868 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:45.859+0000 7f34337fe700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:37:45.868 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:45.859+0000 7f34337fe700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:37:45.868 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-09-30T01:37:45.868 INFO:teuthology.orchestra.run.smithi088.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:37:45.870 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:45.863+0000 7f3441438700 1 -- 172.21.15.88:0/115153708 >> v2:172.21.15.107:6812/29926 conn(0x7f341c033360 msgr2=0x7f341c035810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:37:45.870 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:45.863+0000 7f3441438700 1 --2- 172.21.15.88:0/115153708 >> v2:172.21.15.107:6812/29926 conn(0x7f341c033360 0x7f341c035810 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f342c00bfd0 tx=0x7f342c017000 comp rx=0 tx=0).stop 2023-09-30T01:37:45.870 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:45.863+0000 7f3441438700 10 monclient: shutdown 2023-09-30T01:37:45.870 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:45.863+0000 7f3441438700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:37:45.870 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:45.863+0000 7f3441438700 1 -- 172.21.15.88:0/115153708 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f343c133cf0 msgr2=0x7f343c11d8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:37:45.870 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:45.863+0000 7f3441438700 1 --2- 172.21.15.88:0/115153708 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f343c133cf0 0x7f343c11d8d0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f3434009e90 tx=0x7f343400b040 comp rx=0 tx=0).stop 2023-09-30T01:37:45.870 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:45.863+0000 7f3441438700 1 -- 172.21.15.88:0/115153708 shutdown_connections 2023-09-30T01:37:45.870 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:45.863+0000 7f3441438700 1 --2- 172.21.15.88:0/115153708 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f343c133cf0 0x7f343c11d8d0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:37:45.870 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:45.863+0000 7f3441438700 1 --2- 172.21.15.88:0/115153708 >> v2:172.21.15.107:6812/29926 conn(0x7f341c033360 0x7f341c035810 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:37:45.871 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:45.863+0000 7f3441438700 1 --2- 172.21.15.88:0/115153708 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f343c1338e0 0x7f343c11d390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:37:45.871 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:45.863+0000 7f3441438700 1 --2- 172.21.15.88:0/115153708 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7f343c0a3270 0x7f343c11ce50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:37:45.871 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:45.863+0000 7f3441438700 1 -- 172.21.15.88:0/115153708 >> 172.21.15.88:0/115153708 conn(0x7f343c06e5e0 msgr2=0x7f343c0a1150 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:37:45.871 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:45.863+0000 7f3441438700 1 -- 172.21.15.88:0/115153708 shutdown_connections 2023-09-30T01:37:45.871 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:45.863+0000 7f3441438700 1 -- 172.21.15.88:0/115153708 wait complete. 2023-09-30T01:37:48.882 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:37:48.994 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:48.991+0000 7f0ed29ef700 10 monclient: get_monmap_and_config 2023-09-30T01:37:48.994 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:48.991+0000 7f0ed29ef700 10 monclient: build_initial_monmap 2023-09-30T01:37:48.994 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:48.991+0000 7f0ed29ef700 10 monclient: monmap: 2023-09-30T01:37:48.994 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2023-09-30T01:37:48.994 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:37:48.994 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:37:48.993720+0000 2023-09-30T01:37:48.995 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:37:48.993720+0000 2023-09-30T01:37:48.995 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2023-09-30T01:37:48.995 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:37:48.995 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a-legacy 2023-09-30T01:37:48.995 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.107:6789/0 mon.noname-f-legacy 2023-09-30T01:37:48.995 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.noname-b 2023-09-30T01:37:48.995 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.noname-g 2023-09-30T01:37:48.995 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.noname-c 2023-09-30T01:37:48.996 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.noname-h 2023-09-30T01:37:48.996 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.noname-d 2023-09-30T01:37:48.996 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.noname-i 2023-09-30T01:37:48.996 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.noname-e 2023-09-30T01:37:48.996 INFO:teuthology.orchestra.run.smithi088.stderr:9: v2:172.21.15.88:3300/0 mon.noname-a 2023-09-30T01:37:48.996 INFO:teuthology.orchestra.run.smithi088.stderr:10: v2:172.21.15.107:3300/0 mon.noname-f 2023-09-30T01:37:48.996 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:37:49.005 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:48.991+0000 7f0ed29ef700 1 Processor -- start 2023-09-30T01:37:49.006 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:48.991+0000 7f0ed29ef700 1 -- start start 2023-09-30T01:37:49.006 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:48.991+0000 7f0ed29ef700 10 monclient: init 2023-09-30T01:37:49.006 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:48.991+0000 7f0ed29ef700 10 monclient: _reopen_session rank -1 2023-09-30T01:37:49.006 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:48.991+0000 7f0ed29ef700 10 monclient: _add_conns ranks=[4,10,2,0,6,5,1,3,7,8,9] 2023-09-30T01:37:49.006 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:48.991+0000 7f0ed29ef700 1 --2- >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f0ecc0a35f0 0x7f0ecc133330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:37:49.006 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:48.991+0000 7f0ed29ef700 10 monclient(hunting): picked mon.noname-c con 0x7f0ecc0a35f0 addr [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] 2023-09-30T01:37:49.006 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:48.991+0000 7f0ed29ef700 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f0ecc133870 0x7f0ecc133c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:37:49.006 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:48.991+0000 7f0ed29ef700 10 monclient(hunting): picked mon.noname-f con 0x7f0ecc133870 addr v2:172.21.15.107:3300/0 2023-09-30T01:37:49.007 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:48.991+0000 7f0ed29ef700 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0ecc0a46f0 0x7f0ecc0a4b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:37:49.007 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:48.991+0000 7f0ed29ef700 10 monclient(hunting): picked mon.noname-b con 0x7f0ecc0a46f0 addr [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] 2023-09-30T01:37:49.007 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:48.991+0000 7f0ed29ef700 10 monclient(hunting): start opening mon connection 2023-09-30T01:37:49.007 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:48.991+0000 7f0ed29ef700 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0ecc097ad0 con 0x7f0ecc133870 2023-09-30T01:37:49.007 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:48.991+0000 7f0ed29ef700 10 monclient(hunting): start opening mon connection 2023-09-30T01:37:49.007 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:48.991+0000 7f0ed29ef700 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ecc1341c0 con 0x7f0ecc0a46f0 2023-09-30T01:37:49.007 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:48.991+0000 7f0ed29ef700 10 monclient(hunting): start opening mon connection 2023-09-30T01:37:49.007 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:48.991+0000 7f0ed29ef700 1 -- --> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] -- mon_getmap magic: 0 v1 -- 0x7f0ecc0a0940 con 0x7f0ecc0a35f0 2023-09-30T01:37:49.007 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:48.991+0000 7f0ed29ef700 10 monclient(hunting): _renew_subs 2023-09-30T01:37:49.008 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:48.991+0000 7f0ed29ef700 10 monclient(hunting): authenticate will time out at 1141.031336s 2023-09-30T01:37:49.008 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:48.991+0000 7f0ed0f8c700 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0ecc0a46f0 0x7f0ecc0a4b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:37:49.008 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:48.991+0000 7f0ecbfff700 1 --2- >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f0ecc0a35f0 0x7f0ecc133330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:37:49.008 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:48.991+0000 7f0ecbfff700 1 --2- >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f0ecc0a35f0 0x7f0ecc133330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3302/0 says I am v2:172.21.15.88:58910/0 (socket says 172.21.15.88:58910) 2023-09-30T01:37:49.008 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:48.991+0000 7f0ecbfff700 1 -- 172.21.15.88:0/1945908714 learned_addr learned my addr 172.21.15.88:0/1945908714 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-09-30T01:37:49.008 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:48.991+0000 7f0ecbfff700 10 monclient(hunting): get_auth_request con 0x7f0ecc0a35f0 auth_method 0 2023-09-30T01:37:49.008 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:48.991+0000 7f0ecbfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:37:49.008 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:48.991+0000 7f0ecbfff700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:37:49.008 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:48.991+0000 7f0ecbfff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:37:49.008 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:48.991+0000 7f0ed0f8c700 10 monclient(hunting): get_auth_request con 0x7f0ecc0a46f0 auth_method 0 2023-09-30T01:37:49.008 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:48.991+0000 7f0ed0f8c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:37:49.009 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:48.991+0000 7f0ed0f8c700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:37:49.009 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:48.991+0000 7f0ed0f8c700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:37:49.009 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:48.991+0000 7f0ed0f8c700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:37:49.009 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:48.991+0000 7f0ed0f8c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:37:49.009 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:48.991+0000 7f0ed0f8c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:37:49.009 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:48.991+0000 7f0ecbfff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:37:49.009 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:48.991+0000 7f0ecbfff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:37:49.009 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:48.991+0000 7f0ec37fe700 1 --2- 172.21.15.88:0/1945908714 >> v2:172.21.15.107:3300/0 conn(0x7f0ecc133870 0x7f0ecc133c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:37:49.009 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:48.991+0000 7f0ecbfff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:37:49.009 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:48.991+0000 7f0ec37fe700 10 monclient(hunting): get_auth_request con 0x7f0ecc133870 auth_method 0 2023-09-30T01:37:49.010 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:48.995+0000 7f0ec37fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:37:49.010 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:48.995+0000 7f0ec37fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:37:49.010 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:48.995+0000 7f0ec37fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:37:49.010 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:48.995+0000 7f0ed0f8c700 10 monclient(hunting): handle_auth_done global_id 4763 payload 274 2023-09-30T01:37:49.010 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:48.995+0000 7f0ed0f8c700 1 -- 172.21.15.88:0/1945908714 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f0ecc0a35f0 msgr2=0x7f0ecc133330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:37:49.010 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:48.995+0000 7f0ed0f8c700 1 --2- 172.21.15.88:0/1945908714 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f0ecc0a35f0 0x7f0ecc133330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:37:49.010 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:48.995+0000 7f0ed0f8c700 1 -- 172.21.15.88:0/1945908714 >> v2:172.21.15.107:3300/0 conn(0x7f0ecc133870 msgr2=0x7f0ecc133c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:37:49.010 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:48.995+0000 7f0ed0f8c700 1 --2- 172.21.15.88:0/1945908714 >> v2:172.21.15.107:3300/0 conn(0x7f0ecc133870 0x7f0ecc133c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:37:49.010 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:48.995+0000 7f0ed0f8c700 10 monclient: _finish_hunting 0 2023-09-30T01:37:49.010 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:48.995+0000 7f0ed0f8c700 1 monclient: found mon.noname-b 2023-09-30T01:37:49.011 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:48.995+0000 7f0ed0f8c700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:37:49.011 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:48.995+0000 7f0ed0f8c700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.88:3301/0 2023-09-30T01:37:49.011 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:48.995+0000 7f0ed0f8c700 1 -- 172.21.15.88:0/1945908714 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ecc05ecb0 con 0x7f0ecc0a46f0 2023-09-30T01:37:49.011 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:48.995+0000 7f0ed0f8c700 1 --2- 172.21.15.88:0/1945908714 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0ecc0a46f0 0x7f0ecc0a4b60 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f0ec4009e60 tx=0x7f0ec4009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=66f19b77e4670a87 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:37:49.011 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:48.995+0000 7f0ecb7fe700 1 -- 172.21.15.88:0/1945908714 <== mon.2 v2:172.21.15.88:3301/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f0ec400e070 con 0x7f0ecc0a46f0 2023-09-30T01:37:49.011 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:48.995+0000 7f0ecb7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:37:49.011 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:48.995+0000 7f0ecb7fe700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T01:37:49.011 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:48.995+0000 7f0ecb7fe700 10 monclient: dump: 2023-09-30T01:37:49.011 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:37:49.011 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:37:49.011 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:37:49.012 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:37:49.012 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:37:49.012 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:37:49.012 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:37:49.012 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:37:49.012 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:37:49.012 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:37:49.012 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:37:49.012 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:37:49.012 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:37:49.013 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:37:49.013 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:37:49.013 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:37:49.013 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:48.995+0000 7f0ecb7fe700 10 monclient: _finish_auth 0 2023-09-30T01:37:49.013 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:48.995+0000 7f0ecb7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:37:49.013 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:48.995+0000 7f0ed29ef700 5 monclient: authenticate success, global_id 4763 2023-09-30T01:37:49.013 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:48.995+0000 7f0ed29ef700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:37:49.013 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:48.995+0000 7f0ecb7fe700 1 -- 172.21.15.88:0/1945908714 <== mon.2 v2:172.21.15.88:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ec4012430 con 0x7f0ecc0a46f0 2023-09-30T01:37:49.013 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:48.995+0000 7f0ecb7fe700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:37:49.013 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:48.995+0000 7f0ed29ef700 10 monclient: get_monmap_and_config success 2023-09-30T01:37:49.014 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:48.995+0000 7f0ed29ef700 4 set_mon_vals no callback set 2023-09-30T01:37:49.014 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:48.995+0000 7f0ecb7fe700 1 -- 172.21.15.88:0/1945908714 <== mon.2 v2:172.21.15.88:3301/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f0ec4012a20 con 0x7f0ecc0a46f0 2023-09-30T01:37:49.014 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:48.995+0000 7f0ecb7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:37:49.014 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:48.995+0000 7f0ecb7fe700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T01:37:49.014 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:48.995+0000 7f0ecb7fe700 10 monclient: dump: 2023-09-30T01:37:49.014 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:37:49.014 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:37:49.014 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:37:49.014 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:37:49.014 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:37:49.014 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:37:49.015 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:37:49.015 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:37:49.015 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:37:49.015 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:37:49.015 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:37:49.015 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:37:49.015 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:37:49.015 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:37:49.015 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:37:49.015 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:37:49.016 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:48.995+0000 7f0ed29ef700 10 monclient: shutdown 2023-09-30T01:37:49.016 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:48.995+0000 7f0ed29ef700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:37:49.016 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:48.995+0000 7f0ed29ef700 1 -- 172.21.15.88:0/1945908714 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0ecc0a46f0 msgr2=0x7f0ecc0a4b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:37:49.016 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:48.995+0000 7f0ed29ef700 1 --2- 172.21.15.88:0/1945908714 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0ecc0a46f0 0x7f0ecc0a4b60 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f0ec4009e60 tx=0x7f0ec4009780 comp rx=0 tx=0).stop 2023-09-30T01:37:49.016 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:48.995+0000 7f0ed29ef700 1 -- 172.21.15.88:0/1945908714 shutdown_connections 2023-09-30T01:37:49.016 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:48.995+0000 7f0ed29ef700 1 --2- 172.21.15.88:0/1945908714 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0ecc0a46f0 0x7f0ecc0a4b60 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:37:49.016 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:48.995+0000 7f0ed29ef700 1 --2- 172.21.15.88:0/1945908714 >> v2:172.21.15.107:3300/0 conn(0x7f0ecc133870 0x7f0ecc133c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:37:49.016 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:48.995+0000 7f0ed29ef700 1 --2- 172.21.15.88:0/1945908714 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f0ecc0a35f0 0x7f0ecc133330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:37:49.016 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:48.995+0000 7f0ed29ef700 1 -- 172.21.15.88:0/1945908714 >> 172.21.15.88:0/1945908714 conn(0x7f0ecc06e5e0 msgr2=0x7f0ecc06e9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:37:49.016 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:48.995+0000 7f0ed29ef700 1 -- 172.21.15.88:0/1945908714 shutdown_connections 2023-09-30T01:37:49.016 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:48.995+0000 7f0ed29ef700 1 -- 172.21.15.88:0/1945908714 wait complete. 2023-09-30T01:37:49.017 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:48.995+0000 7f0ed29ef700 10 monclient: build_initial_monmap 2023-09-30T01:37:49.017 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:48.995+0000 7f0ed29ef700 10 monclient: monmap: 2023-09-30T01:37:49.017 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2023-09-30T01:37:49.017 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:37:49.017 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 0.000000 2023-09-30T01:37:49.017 INFO:teuthology.orchestra.run.smithi088.stderr:created 0.000000 2023-09-30T01:37:49.017 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2023-09-30T01:37:49.017 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 1 2023-09-30T01:37:49.017 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.noname-a 2023-09-30T01:37:49.017 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.noname-f 2023-09-30T01:37:49.018 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.noname-b 2023-09-30T01:37:49.018 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.noname-g 2023-09-30T01:37:49.018 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.noname-c 2023-09-30T01:37:49.018 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.noname-h 2023-09-30T01:37:49.018 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.noname-d 2023-09-30T01:37:49.018 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.noname-i 2023-09-30T01:37:49.018 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.noname-e 2023-09-30T01:37:49.018 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:37:49.018 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:48.995+0000 7f0ed29ef700 1 Processor -- start 2023-09-30T01:37:49.018 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:48.995+0000 7f0ed29ef700 1 -- start start 2023-09-30T01:37:49.019 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:48.995+0000 7f0ed29ef700 10 monclient: init 2023-09-30T01:37:49.019 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:48.995+0000 7f0ed29ef700 10 monclient: _reopen_session rank -1 2023-09-30T01:37:49.019 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:48.995+0000 7f0ed29ef700 10 monclient: _add_conns ranks=[4,1,8,3,6,5,7,0,2] 2023-09-30T01:37:49.019 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:48.995+0000 7f0ed29ef700 1 --2- >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f0ecc0a35f0 0x7f0ecc142550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:37:49.019 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:48.995+0000 7f0ed29ef700 10 monclient(hunting): picked mon.noname-c con 0x7f0ecc0a35f0 addr [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] 2023-09-30T01:37:49.019 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:48.995+0000 7f0ed29ef700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0ecc0a46f0 0x7f0ecc142a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:37:49.019 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:48.995+0000 7f0ed29ef700 10 monclient(hunting): picked mon.noname-f con 0x7f0ecc0a46f0 addr [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] 2023-09-30T01:37:49.019 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:48.995+0000 7f0ed29ef700 1 --2- >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f0ecc133870 0x7f0ecc142fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:37:49.019 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:48.995+0000 7f0ed29ef700 10 monclient(hunting): picked mon.noname-e con 0x7f0ecc133870 addr [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] 2023-09-30T01:37:49.019 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:48.995+0000 7f0ed29ef700 10 monclient(hunting): start opening mon connection 2023-09-30T01:37:49.020 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:48.995+0000 7f0ed29ef700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ecc121e00 con 0x7f0ecc0a46f0 2023-09-30T01:37:49.020 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:48.995+0000 7f0ed29ef700 10 monclient(hunting): start opening mon connection 2023-09-30T01:37:49.020 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:48.995+0000 7f0ed29ef700 1 -- --> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] -- mon_getmap magic: 0 v1 -- 0x7f0ecc122aa0 con 0x7f0ecc0a35f0 2023-09-30T01:37:49.020 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:48.995+0000 7f0ed29ef700 10 monclient(hunting): start opening mon connection 2023-09-30T01:37:49.020 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:48.995+0000 7f0ed29ef700 1 -- --> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] -- mon_getmap magic: 0 v1 -- 0x7f0ecc0a0b70 con 0x7f0ecc133870 2023-09-30T01:37:49.020 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:48.995+0000 7f0ed29ef700 10 monclient(hunting): _renew_subs 2023-09-30T01:37:49.020 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:48.995+0000 7f0ed29ef700 10 monclient(hunting): authenticate will time out at 1141.035406s 2023-09-30T01:37:49.020 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:48.995+0000 7f0ed0f8c700 1 --2- >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f0ecc133870 0x7f0ecc142fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:37:49.020 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:48.995+0000 7f0ecbfff700 1 --2- >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f0ecc0a35f0 0x7f0ecc142550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:37:49.020 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:48.995+0000 7f0ed0f8c700 1 --2- >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f0ecc133870 0x7f0ecc142fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3304/0 says I am v2:172.21.15.88:59740/0 (socket says 172.21.15.88:59740) 2023-09-30T01:37:49.021 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:48.995+0000 7f0ed0f8c700 1 -- 172.21.15.88:0/1325949768 learned_addr learned my addr 172.21.15.88:0/1325949768 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-09-30T01:37:49.021 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:48.995+0000 7f0ed0f8c700 10 monclient(hunting): get_auth_request con 0x7f0ecc133870 auth_method 0 2023-09-30T01:37:49.021 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:48.995+0000 7f0ed0f8c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:37:49.021 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:48.995+0000 7f0ed0f8c700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:37:49.021 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:48.995+0000 7f0ed0f8c700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:37:49.021 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:48.995+0000 7f0ecbfff700 10 monclient(hunting): get_auth_request con 0x7f0ecc0a35f0 auth_method 0 2023-09-30T01:37:49.021 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:48.995+0000 7f0ecbfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:37:49.021 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:48.995+0000 7f0ecbfff700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:37:49.021 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:48.995+0000 7f0ecbfff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:37:49.021 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:48.995+0000 7f0ed0f8c700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:37:49.022 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:48.995+0000 7f0ed0f8c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:37:49.022 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:48.995+0000 7f0ed0f8c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:37:49.022 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:48.995+0000 7f0ecbfff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:37:49.022 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:48.995+0000 7f0ec37fe700 1 --2- 172.21.15.88:0/1325949768 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0ecc0a46f0 0x7f0ecc142a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:37:49.022 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:48.995+0000 7f0ecbfff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:37:49.022 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:48.995+0000 7f0ecbfff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:37:49.022 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:48.995+0000 7f0ec37fe700 10 monclient(hunting): get_auth_request con 0x7f0ecc0a46f0 auth_method 0 2023-09-30T01:37:49.022 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:48.995+0000 7f0ec37fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:37:49.022 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:48.995+0000 7f0ec37fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:37:49.022 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:48.995+0000 7f0ec37fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:37:49.023 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:48.999+0000 7f0ec37fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:37:49.023 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:48.999+0000 7f0ec37fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:37:49.023 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:48.999+0000 7f0ec37fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:37:49.023 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:48.999+0000 7f0ecbfff700 10 monclient(hunting): handle_auth_done global_id 4855 payload 824 2023-09-30T01:37:49.023 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:48.999+0000 7f0ecbfff700 1 -- 172.21.15.88:0/1325949768 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f0ecc133870 msgr2=0x7f0ecc142fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:37:49.023 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:48.999+0000 7f0ecbfff700 1 --2- 172.21.15.88:0/1325949768 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f0ecc133870 0x7f0ecc142fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:37:49.023 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:48.999+0000 7f0ecbfff700 1 -- 172.21.15.88:0/1325949768 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0ecc0a46f0 msgr2=0x7f0ecc142a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:37:49.023 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:48.999+0000 7f0ecbfff700 1 --2- 172.21.15.88:0/1325949768 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0ecc0a46f0 0x7f0ecc142a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:37:49.023 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:48.999+0000 7f0ecbfff700 10 monclient: _finish_hunting 0 2023-09-30T01:37:49.023 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:48.999+0000 7f0ecbfff700 1 monclient: found mon.noname-c 2023-09-30T01:37:49.024 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:48.999+0000 7f0ecbfff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:37:49.024 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:48.999+0000 7f0ecbfff700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.88:3302/0 2023-09-30T01:37:49.024 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:48.999+0000 7f0ecbfff700 1 -- 172.21.15.88:0/1325949768 --> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ec4009a20 con 0x7f0ecc0a35f0 2023-09-30T01:37:49.024 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:48.999+0000 7f0ed0f8c700 1 --2- 172.21.15.88:0/1325949768 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f0ecc133870 0x7f0ecc142fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:37:49.024 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:48.999+0000 7f0ecbfff700 1 --2- 172.21.15.88:0/1325949768 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f0ecc0a35f0 0x7f0ecc142550 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f0ebc00e970 tx=0x7f0ebc009250 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:37:49.024 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:48.999+0000 7f0ec8ff9700 1 -- 172.21.15.88:0/1325949768 <== mon.4 v2:172.21.15.88:3302/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f0ebc013b00 con 0x7f0ecc0a35f0 2023-09-30T01:37:49.024 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:48.999+0000 7f0ec8ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:37:49.024 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:48.999+0000 7f0ec8ff9700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T01:37:49.024 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:48.999+0000 7f0ec8ff9700 10 monclient: dump: 2023-09-30T01:37:49.024 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:37:49.025 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:37:49.025 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:37:49.025 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:37:49.025 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:37:49.025 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:37:49.025 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:37:49.025 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:37:49.025 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:37:49.025 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:37:49.025 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:37:49.026 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:37:49.026 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:37:49.026 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:37:49.026 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:37:49.026 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:37:49.026 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:48.999+0000 7f0ec8ff9700 10 monclient: _finish_auth 0 2023-09-30T01:37:49.026 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:48.999+0000 7f0ec8ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:37:49.026 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:48.999+0000 7f0ed29ef700 5 monclient: authenticate success, global_id 4855 2023-09-30T01:37:49.026 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:48.999+0000 7f0ed29ef700 10 monclient: _renew_subs 2023-09-30T01:37:49.026 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:48.999+0000 7f0ed29ef700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.88:3302/0 2023-09-30T01:37:49.026 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:48.999+0000 7f0ed29ef700 1 -- 172.21.15.88:0/1325949768 --> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ecc1208e0 con 0x7f0ecc0a35f0 2023-09-30T01:37:49.027 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:48.999+0000 7f0ed29ef700 10 monclient: _renew_subs 2023-09-30T01:37:49.027 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:48.999+0000 7f0ec8ff9700 1 -- 172.21.15.88:0/1325949768 <== mon.4 v2:172.21.15.88:3302/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ebc013c60 con 0x7f0ecc0a35f0 2023-09-30T01:37:49.027 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:48.999+0000 7f0ed29ef700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.88:3302/0 2023-09-30T01:37:49.027 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:48.999+0000 7f0ed29ef700 1 -- 172.21.15.88:0/1325949768 --> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ecc1217d0 con 0x7f0ecc0a35f0 2023-09-30T01:37:49.027 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:48.999+0000 7f0ec8ff9700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:37:49.027 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:48.999+0000 7f0ec8ff9700 1 -- 172.21.15.88:0/1325949768 <== mon.4 v2:172.21.15.88:3302/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f0ebc013dd0 con 0x7f0ecc0a35f0 2023-09-30T01:37:49.027 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:48.999+0000 7f0ec8ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:37:49.027 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:48.999+0000 7f0ec8ff9700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T01:37:49.027 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:48.999+0000 7f0ec8ff9700 10 monclient: dump: 2023-09-30T01:37:49.027 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:37:49.028 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:37:49.028 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:37:49.028 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:37:49.028 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:37:49.028 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:37:49.028 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:37:49.028 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:37:49.028 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:37:49.029 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:37:49.029 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:37:49.029 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:37:49.029 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:37:49.029 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:37:49.029 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:37:49.029 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:37:49.029 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:48.999+0000 7f0ecaffd700 4 set_mon_vals no callback set 2023-09-30T01:37:49.029 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:48.999+0000 7f0ed29ef700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:37:49.029 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:48.999+0000 7f0ec8ff9700 1 -- 172.21.15.88:0/1325949768 <== mon.4 v2:172.21.15.88:3302/0 4 ==== mgrmap(e 5) v1 ==== 40144+0+0 (secure 0 0 0) 0x7f0ebc013680 con 0x7f0ecc0a35f0 2023-09-30T01:37:49.030 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:48.999+0000 7f0ed29ef700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:37:49.030 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:48.999+0000 7f0ed29ef700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.88:3302/0 2023-09-30T01:37:49.030 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:48.999+0000 7f0ed29ef700 1 -- 172.21.15.88:0/1325949768 --> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0eb00050b0 con 0x7f0ecc0a35f0 2023-09-30T01:37:49.030 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:48.999+0000 7f0ec8ff9700 1 --2- 172.21.15.88:0/1325949768 >> v2:172.21.15.107:6812/29926 conn(0x7f0eac033360 0x7f0eac035810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:37:49.030 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:48.999+0000 7f0ec8ff9700 1 -- 172.21.15.88:0/1325949768 <== mon.4 v2:172.21.15.88:3302/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 3502+0+0 (secure 0 0 0) 0x7f0ebc043520 con 0x7f0ecc0a35f0 2023-09-30T01:37:49.030 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:48.999+0000 7f0ec37fe700 1 --2- 172.21.15.88:0/1325949768 >> v2:172.21.15.107:6812/29926 conn(0x7f0eac033360 0x7f0eac035810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:37:49.030 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:48.999+0000 7f0ec37fe700 10 monclient: get_auth_request con 0x7f0eac033360 auth_method 0 2023-09-30T01:37:49.030 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:49.003+0000 7f0ec37fe700 1 --2- 172.21.15.88:0/1325949768 >> v2:172.21.15.107:6812/29926 conn(0x7f0eac033360 0x7f0eac035810 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f0eb4006fd0 tx=0x7f0eb4008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:37:49.031 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:49.007+0000 7f0ec8ff9700 1 -- 172.21.15.88:0/1325949768 <== mon.4 v2:172.21.15.88:3302/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f0ebc013930 con 0x7f0ecc0a35f0 2023-09-30T01:37:49.031 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:49.007+0000 7f0ec8ff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:37:49.031 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:49.007+0000 7f0ec8ff9700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:37:49.206 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:49.203+0000 7f0ed29ef700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T01:37:49.206 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:49.203+0000 7f0ed29ef700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:37:49.206 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:49.203+0000 7f0ed29ef700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.88:3302/0 2023-09-30T01:37:49.207 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:49.203+0000 7f0ed29ef700 1 -- 172.21.15.88:0/1325949768 --> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f0eb00058e0 con 0x7f0ecc0a35f0 2023-09-30T01:37:49.207 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:49.203+0000 7f0ec8ff9700 1 -- 172.21.15.88:0/1325949768 <== mon.4 v2:172.21.15.88:3302/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f0ebc01b3b0 con 0x7f0ecc0a35f0 2023-09-30T01:37:49.207 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:49.203+0000 7f0ec8ff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:37:49.207 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:49.203+0000 7f0ec8ff9700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:37:49.207 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-09-30T01:37:49.208 INFO:teuthology.orchestra.run.smithi088.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:37:49.210 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:49.207+0000 7f0ed29ef700 1 -- 172.21.15.88:0/1325949768 >> v2:172.21.15.107:6812/29926 conn(0x7f0eac033360 msgr2=0x7f0eac035810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:37:49.210 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:49.207+0000 7f0ed29ef700 1 --2- 172.21.15.88:0/1325949768 >> v2:172.21.15.107:6812/29926 conn(0x7f0eac033360 0x7f0eac035810 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f0eb4006fd0 tx=0x7f0eb4008040 comp rx=0 tx=0).stop 2023-09-30T01:37:49.210 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:49.207+0000 7f0ed29ef700 10 monclient: shutdown 2023-09-30T01:37:49.210 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:49.207+0000 7f0ed29ef700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:37:49.210 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:49.207+0000 7f0ed29ef700 1 -- 172.21.15.88:0/1325949768 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f0ecc0a35f0 msgr2=0x7f0ecc142550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:37:49.210 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:49.207+0000 7f0ed29ef700 1 --2- 172.21.15.88:0/1325949768 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f0ecc0a35f0 0x7f0ecc142550 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f0ebc00e970 tx=0x7f0ebc009250 comp rx=0 tx=0).stop 2023-09-30T01:37:49.210 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:49.207+0000 7f0ed29ef700 1 -- 172.21.15.88:0/1325949768 shutdown_connections 2023-09-30T01:37:49.210 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:49.207+0000 7f0ed29ef700 1 --2- 172.21.15.88:0/1325949768 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0ecc0a46f0 0x7f0ecc142a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:37:49.210 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:49.207+0000 7f0ed29ef700 1 --2- 172.21.15.88:0/1325949768 >> v2:172.21.15.107:6812/29926 conn(0x7f0eac033360 0x7f0eac035810 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:37:49.210 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:49.207+0000 7f0ed29ef700 1 --2- 172.21.15.88:0/1325949768 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f0ecc133870 0x7f0ecc142fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:37:49.210 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:49.207+0000 7f0ed29ef700 1 --2- 172.21.15.88:0/1325949768 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f0ecc0a35f0 0x7f0ecc142550 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:37:49.211 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:49.207+0000 7f0ed29ef700 1 -- 172.21.15.88:0/1325949768 >> 172.21.15.88:0/1325949768 conn(0x7f0ecc06e5e0 msgr2=0x7f0ecc09b020 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:37:49.211 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:49.207+0000 7f0ed29ef700 1 -- 172.21.15.88:0/1325949768 shutdown_connections 2023-09-30T01:37:49.211 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:49.207+0000 7f0ed29ef700 1 -- 172.21.15.88:0/1325949768 wait complete. 2023-09-30T01:37:51.514 INFO:tasks.ceph.mon.d.smithi088.stderr:2023-09-30T01:37:51.511+0000 7fabfe1e4700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-09-30T01:37:51.514 INFO:tasks.ceph.mon.d.smithi088.stderr:2023-09-30T01:37:51.511+0000 7fabfe1e4700 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {auth=11,config=2,health=12,kv=8,logm=67} crc {auth=4163141698,config=3140514009,health=1307886691,kv=755892160,logm=403916405}) 2023-09-30T01:37:51.514 INFO:tasks.ceph.mon.d.smithi088.stderr:2023-09-30T01:37:51.511+0000 7fabfe1e4700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=11,config=2,health=12,kv=7,logm=68} crc {auth=4163141698,config=3140514009,health=1307886691,kv=3191572970,logm=4102798835}) 2023-09-30T01:37:51.514 INFO:tasks.ceph.mon.d.smithi088.stderr:2023-09-30T01:37:51.511+0000 7fabfe1e4700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-09-30T01:37:51.515 INFO:tasks.ceph.mon.d.smithi088.stderr:2023-09-30T01:37:51.511+0000 7fabfe1e4700 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {auth=11,config=2,health=12,kv=8,logm=67} crc {auth=4163141698,config=3140514009,health=1307886691,kv=755892160,logm=403916405}) 2023-09-30T01:37:51.515 INFO:tasks.ceph.mon.d.smithi088.stderr:2023-09-30T01:37:51.511+0000 7fabfe1e4700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=11,config=2,health=12,kv=8,logm=67} crc {auth=4163141698,config=3140514009,health=1307886691,kv=755892160,logm=301335254}) 2023-09-30T01:37:51.515 INFO:tasks.ceph.mon.d.smithi088.stderr:2023-09-30T01:37:51.511+0000 7fabfe1e4700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-09-30T01:37:51.515 INFO:tasks.ceph.mon.d.smithi088.stderr:2023-09-30T01:37:51.511+0000 7fabfe1e4700 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {auth=11,config=2,health=12,kv=8,logm=67} crc {auth=4163141698,config=3140514009,health=1307886691,kv=755892160,logm=403916405}) 2023-09-30T01:37:51.515 INFO:tasks.ceph.mon.d.smithi088.stderr:2023-09-30T01:37:51.511+0000 7fabfe1e4700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=10,config=2,health=12,kv=8,logm=68} crc {auth=3539731369,config=3140514009,health=1307886692,kv=755892160,logm=406216267}) 2023-09-30T01:37:51.515 INFO:tasks.ceph.mon.d.smithi088.stderr:2023-09-30T01:37:51.511+0000 7fabfe1e4700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-09-30T01:37:51.515 INFO:tasks.ceph.mon.d.smithi088.stderr:2023-09-30T01:37:51.511+0000 7fabfe1e4700 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {auth=11,config=2,health=12,kv=8,logm=67} crc {auth=4163141698,config=3140514009,health=1307886691,kv=755892160,logm=403916405}) 2023-09-30T01:37:51.515 INFO:tasks.ceph.mon.d.smithi088.stderr:2023-09-30T01:37:51.511+0000 7fabfe1e4700 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {auth=11,config=2,health=12,kv=8,logm=67} crc {auth=4163141698,config=3140514009,health=1307886691,kv=755892160,logm=2242653326}) 2023-09-30T01:37:51.515 INFO:tasks.ceph.mon.d.smithi088.stderr:2023-09-30T01:37:51.511+0000 7fabfe1e4700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-09-30T01:37:51.515 INFO:tasks.ceph.mon.d.smithi088.stderr:2023-09-30T01:37:51.511+0000 7fabfe1e4700 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {auth=11,config=2,health=12,kv=8,logm=67} crc {auth=4163141698,config=3140514009,health=1307886691,kv=755892160,logm=403916405}) 2023-09-30T01:37:51.516 INFO:tasks.ceph.mon.d.smithi088.stderr:2023-09-30T01:37:51.511+0000 7fabfe1e4700 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {auth=11,config=2,health=12,kv=8,logm=67} crc {auth=4163141698,config=3140514009,health=1307886691,kv=755892160,logm=2536657359}) 2023-09-30T01:37:51.516 INFO:tasks.ceph.mon.d.smithi088.stderr:2023-09-30T01:37:51.511+0000 7fabfe1e4700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-09-30T01:37:51.516 INFO:tasks.ceph.mon.d.smithi088.stderr:2023-09-30T01:37:51.511+0000 7fabfe1e4700 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {auth=11,config=2,health=12,kv=8,logm=67} crc {auth=4163141698,config=3140514009,health=1307886691,kv=755892160,logm=403916405}) 2023-09-30T01:37:51.516 INFO:tasks.ceph.mon.d.smithi088.stderr:2023-09-30T01:37:51.511+0000 7fabfe1e4700 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {auth=9,config=2,health=12,kv=8,logm=69} crc {auth=1943848712,config=3140514009,health=1307886691,kv=755892160,logm=2031349494}) 2023-09-30T01:37:51.516 INFO:tasks.ceph.mon.d.smithi088.stderr:2023-09-30T01:37:51.511+0000 7fabfe1e4700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-09-30T01:37:51.516 INFO:tasks.ceph.mon.d.smithi088.stderr:2023-09-30T01:37:51.511+0000 7fabfe1e4700 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {auth=11,config=2,health=12,kv=8,logm=67} crc {auth=4163141698,config=3140514009,health=1307886691,kv=755892160,logm=403916405}) 2023-09-30T01:37:51.516 INFO:tasks.ceph.mon.d.smithi088.stderr:2023-09-30T01:37:51.511+0000 7fabfe1e4700 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {auth=10,config=2,health=12,kv=7,logm=69} crc {auth=3070316726,config=3140514009,health=2460175639,kv=3191572970,logm=337980966}) 2023-09-30T01:37:51.516 INFO:tasks.ceph.mon.d.smithi088.stderr:2023-09-30T01:37:51.511+0000 7fabfe1e4700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-09-30T01:37:51.516 INFO:tasks.ceph.mon.d.smithi088.stderr:2023-09-30T01:37:51.511+0000 7fabfe1e4700 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {auth=11,config=2,health=12,kv=8,logm=67} crc {auth=4163141698,config=3140514009,health=1307886691,kv=755892160,logm=403916405}) 2023-09-30T01:37:51.516 INFO:tasks.ceph.mon.d.smithi088.stderr:2023-09-30T01:37:51.511+0000 7fabfe1e4700 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {auth=10,config=2,health=12,kv=8,logm=68} crc {auth=3070316726,config=3140514009,health=1307886691,kv=755892160,logm=4281587099}) 2023-09-30T01:37:51.520 INFO:tasks.ceph.mon.d.smithi088.stderr:2023-09-30T01:37:51.515+0000 7fabfe1e4700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-09-30T01:37:51.520 INFO:tasks.ceph.mon.d.smithi088.stderr:2023-09-30T01:37:51.515+0000 7fabfe1e4700 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3939551809}) 2023-09-30T01:37:51.520 INFO:tasks.ceph.mon.d.smithi088.stderr:2023-09-30T01:37:51.515+0000 7fabfe1e4700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2874757948}) 2023-09-30T01:37:51.520 INFO:tasks.ceph.mon.d.smithi088.stderr:2023-09-30T01:37:51.515+0000 7fabfe1e4700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-09-30T01:37:51.520 INFO:tasks.ceph.mon.d.smithi088.stderr:2023-09-30T01:37:51.515+0000 7fabfe1e4700 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3939551809}) 2023-09-30T01:37:51.520 INFO:tasks.ceph.mon.d.smithi088.stderr:2023-09-30T01:37:51.515+0000 7fabfe1e4700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3795669692}) 2023-09-30T01:37:51.520 INFO:tasks.ceph.mon.d.smithi088.stderr:2023-09-30T01:37:51.515+0000 7fabfe1e4700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-09-30T01:37:51.521 INFO:tasks.ceph.mon.d.smithi088.stderr:2023-09-30T01:37:51.515+0000 7fabfe1e4700 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3939551809}) 2023-09-30T01:37:51.521 INFO:tasks.ceph.mon.d.smithi088.stderr:2023-09-30T01:37:51.515+0000 7fabfe1e4700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=554925595}) 2023-09-30T01:37:51.521 INFO:tasks.ceph.mon.d.smithi088.stderr:2023-09-30T01:37:51.515+0000 7fabfe1e4700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-09-30T01:37:51.521 INFO:tasks.ceph.mon.d.smithi088.stderr:2023-09-30T01:37:51.515+0000 7fabfe1e4700 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3939551809}) 2023-09-30T01:37:51.521 INFO:tasks.ceph.mon.d.smithi088.stderr:2023-09-30T01:37:51.515+0000 7fabfe1e4700 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3641246665}) 2023-09-30T01:37:51.521 INFO:tasks.ceph.mon.d.smithi088.stderr:2023-09-30T01:37:51.515+0000 7fabfe1e4700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-09-30T01:37:51.521 INFO:tasks.ceph.mon.d.smithi088.stderr:2023-09-30T01:37:51.515+0000 7fabfe1e4700 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3939551809}) 2023-09-30T01:37:51.521 INFO:tasks.ceph.mon.d.smithi088.stderr:2023-09-30T01:37:51.515+0000 7fabfe1e4700 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1267990038}) 2023-09-30T01:37:51.521 INFO:tasks.ceph.mon.d.smithi088.stderr:2023-09-30T01:37:51.515+0000 7fabfe1e4700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-09-30T01:37:51.522 INFO:tasks.ceph.mon.d.smithi088.stderr:2023-09-30T01:37:51.515+0000 7fabfe1e4700 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3939551809}) 2023-09-30T01:37:51.522 INFO:tasks.ceph.mon.d.smithi088.stderr:2023-09-30T01:37:51.515+0000 7fabfe1e4700 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2052902385}) 2023-09-30T01:37:51.522 INFO:tasks.ceph.mon.d.smithi088.stderr:2023-09-30T01:37:51.515+0000 7fabfe1e4700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-09-30T01:37:51.522 INFO:tasks.ceph.mon.d.smithi088.stderr:2023-09-30T01:37:51.515+0000 7fabfe1e4700 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3939551809}) 2023-09-30T01:37:51.522 INFO:tasks.ceph.mon.d.smithi088.stderr:2023-09-30T01:37:51.515+0000 7fabfe1e4700 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4281590288}) 2023-09-30T01:37:51.522 INFO:tasks.ceph.mon.d.smithi088.stderr:2023-09-30T01:37:51.515+0000 7fabfe1e4700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-09-30T01:37:51.522 INFO:tasks.ceph.mon.d.smithi088.stderr:2023-09-30T01:37:51.515+0000 7fabfe1e4700 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3939551809}) 2023-09-30T01:37:51.522 INFO:tasks.ceph.mon.d.smithi088.stderr:2023-09-30T01:37:51.515+0000 7fabfe1e4700 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3851050285}) 2023-09-30T01:37:51.526 INFO:tasks.ceph.mon.d.smithi088.stderr:2023-09-30T01:37:51.523+0000 7fabfe1e4700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-09-30T01:37:51.526 INFO:tasks.ceph.mon.d.smithi088.stderr:2023-09-30T01:37:51.523+0000 7fabfe1e4700 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3635862343}) 2023-09-30T01:37:51.526 INFO:tasks.ceph.mon.d.smithi088.stderr:2023-09-30T01:37:51.523+0000 7fabfe1e4700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3016772345}) 2023-09-30T01:37:51.526 INFO:tasks.ceph.mon.d.smithi088.stderr:2023-09-30T01:37:51.523+0000 7fabfe1e4700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-09-30T01:37:51.526 INFO:tasks.ceph.mon.d.smithi088.stderr:2023-09-30T01:37:51.523+0000 7fabfe1e4700 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3635862343}) 2023-09-30T01:37:51.526 INFO:tasks.ceph.mon.d.smithi088.stderr:2023-09-30T01:37:51.523+0000 7fabfe1e4700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3020932006}) 2023-09-30T01:37:51.526 INFO:tasks.ceph.mon.d.smithi088.stderr:2023-09-30T01:37:51.523+0000 7fabfe1e4700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-09-30T01:37:51.527 INFO:tasks.ceph.mon.d.smithi088.stderr:2023-09-30T01:37:51.523+0000 7fabfe1e4700 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3635862343}) 2023-09-30T01:37:51.527 INFO:tasks.ceph.mon.d.smithi088.stderr:2023-09-30T01:37:51.523+0000 7fabfe1e4700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2551423128}) 2023-09-30T01:37:51.527 INFO:tasks.ceph.mon.d.smithi088.stderr:2023-09-30T01:37:51.523+0000 7fabfe1e4700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-09-30T01:37:51.527 INFO:tasks.ceph.mon.d.smithi088.stderr:2023-09-30T01:37:51.523+0000 7fabfe1e4700 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3635862343}) 2023-09-30T01:37:51.527 INFO:tasks.ceph.mon.d.smithi088.stderr:2023-09-30T01:37:51.523+0000 7fabfe1e4700 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=321372782}) 2023-09-30T01:37:51.527 INFO:tasks.ceph.mon.d.smithi088.stderr:2023-09-30T01:37:51.523+0000 7fabfe1e4700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-09-30T01:37:51.527 INFO:tasks.ceph.mon.d.smithi088.stderr:2023-09-30T01:37:51.523+0000 7fabfe1e4700 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3635862343}) 2023-09-30T01:37:51.527 INFO:tasks.ceph.mon.d.smithi088.stderr:2023-09-30T01:37:51.523+0000 7fabfe1e4700 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2154261992}) 2023-09-30T01:37:51.527 INFO:tasks.ceph.mon.d.smithi088.stderr:2023-09-30T01:37:51.523+0000 7fabfe1e4700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-09-30T01:37:51.527 INFO:tasks.ceph.mon.d.smithi088.stderr:2023-09-30T01:37:51.523+0000 7fabfe1e4700 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3635862343}) 2023-09-30T01:37:51.528 INFO:tasks.ceph.mon.d.smithi088.stderr:2023-09-30T01:37:51.523+0000 7fabfe1e4700 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1730776733}) 2023-09-30T01:37:51.528 INFO:tasks.ceph.mon.d.smithi088.stderr:2023-09-30T01:37:51.523+0000 7fabfe1e4700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-09-30T01:37:51.528 INFO:tasks.ceph.mon.d.smithi088.stderr:2023-09-30T01:37:51.523+0000 7fabfe1e4700 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3635862343}) 2023-09-30T01:37:51.528 INFO:tasks.ceph.mon.d.smithi088.stderr:2023-09-30T01:37:51.523+0000 7fabfe1e4700 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2806293055}) 2023-09-30T01:37:51.528 INFO:tasks.ceph.mon.d.smithi088.stderr:2023-09-30T01:37:51.523+0000 7fabfe1e4700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-09-30T01:37:51.528 INFO:tasks.ceph.mon.d.smithi088.stderr:2023-09-30T01:37:51.523+0000 7fabfe1e4700 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3635862343}) 2023-09-30T01:37:51.528 INFO:tasks.ceph.mon.d.smithi088.stderr:2023-09-30T01:37:51.523+0000 7fabfe1e4700 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=4153627306}) 2023-09-30T01:37:51.530 INFO:tasks.ceph.mon.d.smithi088.stderr:2023-09-30T01:37:51.527+0000 7fabfe1e4700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-09-30T01:37:51.530 INFO:tasks.ceph.mon.d.smithi088.stderr:2023-09-30T01:37:51.527+0000 7fabfe1e4700 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=352073832}) 2023-09-30T01:37:51.530 INFO:tasks.ceph.mon.d.smithi088.stderr:2023-09-30T01:37:51.527+0000 7fabfe1e4700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2709619887}) 2023-09-30T01:37:51.531 INFO:tasks.ceph.mon.d.smithi088.stderr:2023-09-30T01:37:51.527+0000 7fabfe1e4700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-09-30T01:37:51.531 INFO:tasks.ceph.mon.d.smithi088.stderr:2023-09-30T01:37:51.527+0000 7fabfe1e4700 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=352073832}) 2023-09-30T01:37:51.531 INFO:tasks.ceph.mon.d.smithi088.stderr:2023-09-30T01:37:51.527+0000 7fabfe1e4700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3286671957}) 2023-09-30T01:37:51.531 INFO:tasks.ceph.mon.d.smithi088.stderr:2023-09-30T01:37:51.527+0000 7fabfe1e4700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-09-30T01:37:51.531 INFO:tasks.ceph.mon.d.smithi088.stderr:2023-09-30T01:37:51.527+0000 7fabfe1e4700 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=352073832}) 2023-09-30T01:37:51.531 INFO:tasks.ceph.mon.d.smithi088.stderr:2023-09-30T01:37:51.527+0000 7fabfe1e4700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1855553958}) 2023-09-30T01:37:51.531 INFO:tasks.ceph.mon.d.smithi088.stderr:2023-09-30T01:37:51.527+0000 7fabfe1e4700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-09-30T01:37:51.531 INFO:tasks.ceph.mon.d.smithi088.stderr:2023-09-30T01:37:51.527+0000 7fabfe1e4700 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=352073832}) 2023-09-30T01:37:51.531 INFO:tasks.ceph.mon.d.smithi088.stderr:2023-09-30T01:37:51.527+0000 7fabfe1e4700 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3362130111}) 2023-09-30T01:37:51.532 INFO:tasks.ceph.mon.d.smithi088.stderr:2023-09-30T01:37:51.527+0000 7fabfe1e4700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-09-30T01:37:51.532 INFO:tasks.ceph.mon.d.smithi088.stderr:2023-09-30T01:37:51.527+0000 7fabfe1e4700 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=352073832}) 2023-09-30T01:37:51.532 INFO:tasks.ceph.mon.d.smithi088.stderr:2023-09-30T01:37:51.527+0000 7fabfe1e4700 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2952253191}) 2023-09-30T01:37:51.532 INFO:tasks.ceph.mon.d.smithi088.stderr:2023-09-30T01:37:51.527+0000 7fabfe1e4700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-09-30T01:37:51.532 INFO:tasks.ceph.mon.d.smithi088.stderr:2023-09-30T01:37:51.527+0000 7fabfe1e4700 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=352073832}) 2023-09-30T01:37:51.532 INFO:tasks.ceph.mon.d.smithi088.stderr:2023-09-30T01:37:51.527+0000 7fabfe1e4700 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3406459270}) 2023-09-30T01:37:51.532 INFO:tasks.ceph.mon.d.smithi088.stderr:2023-09-30T01:37:51.527+0000 7fabfe1e4700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-09-30T01:37:51.532 INFO:tasks.ceph.mon.d.smithi088.stderr:2023-09-30T01:37:51.527+0000 7fabfe1e4700 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=352073832}) 2023-09-30T01:37:51.532 INFO:tasks.ceph.mon.d.smithi088.stderr:2023-09-30T01:37:51.527+0000 7fabfe1e4700 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4160969609}) 2023-09-30T01:37:51.532 INFO:tasks.ceph.mon.d.smithi088.stderr:2023-09-30T01:37:51.527+0000 7fabfe1e4700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-09-30T01:37:51.533 INFO:tasks.ceph.mon.d.smithi088.stderr:2023-09-30T01:37:51.527+0000 7fabfe1e4700 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=352073832}) 2023-09-30T01:37:51.533 INFO:tasks.ceph.mon.d.smithi088.stderr:2023-09-30T01:37:51.527+0000 7fabfe1e4700 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2746027298}) 2023-09-30T01:37:51.536 INFO:tasks.ceph.mon.d.smithi088.stderr:2023-09-30T01:37:51.531+0000 7fabfe1e4700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-09-30T01:37:51.536 INFO:tasks.ceph.mon.d.smithi088.stderr:2023-09-30T01:37:51.531+0000 7fabfe1e4700 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=4052848133}) 2023-09-30T01:37:51.536 INFO:tasks.ceph.mon.d.smithi088.stderr:2023-09-30T01:37:51.531+0000 7fabfe1e4700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2438052945}) 2023-09-30T01:37:51.536 INFO:tasks.ceph.mon.d.smithi088.stderr:2023-09-30T01:37:51.531+0000 7fabfe1e4700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-09-30T01:37:51.537 INFO:tasks.ceph.mon.d.smithi088.stderr:2023-09-30T01:37:51.531+0000 7fabfe1e4700 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=4052848133}) 2023-09-30T01:37:51.537 INFO:tasks.ceph.mon.d.smithi088.stderr:2023-09-30T01:37:51.531+0000 7fabfe1e4700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4008120625}) 2023-09-30T01:37:51.537 INFO:tasks.ceph.mon.d.smithi088.stderr:2023-09-30T01:37:51.531+0000 7fabfe1e4700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-09-30T01:37:51.537 INFO:tasks.ceph.mon.d.smithi088.stderr:2023-09-30T01:37:51.531+0000 7fabfe1e4700 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=4052848133}) 2023-09-30T01:37:51.537 INFO:tasks.ceph.mon.d.smithi088.stderr:2023-09-30T01:37:51.531+0000 7fabfe1e4700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2767768755}) 2023-09-30T01:37:51.537 INFO:tasks.ceph.mon.d.smithi088.stderr:2023-09-30T01:37:51.531+0000 7fabfe1e4700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-09-30T01:37:51.537 INFO:tasks.ceph.mon.d.smithi088.stderr:2023-09-30T01:37:51.531+0000 7fabfe1e4700 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=4052848133}) 2023-09-30T01:37:51.538 INFO:tasks.ceph.mon.d.smithi088.stderr:2023-09-30T01:37:51.531+0000 7fabfe1e4700 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2086444585}) 2023-09-30T01:37:51.538 INFO:tasks.ceph.mon.d.smithi088.stderr:2023-09-30T01:37:51.531+0000 7fabfe1e4700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-09-30T01:37:51.538 INFO:tasks.ceph.mon.d.smithi088.stderr:2023-09-30T01:37:51.531+0000 7fabfe1e4700 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=4052848133}) 2023-09-30T01:37:51.538 INFO:tasks.ceph.mon.d.smithi088.stderr:2023-09-30T01:37:51.531+0000 7fabfe1e4700 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3550786119}) 2023-09-30T01:37:51.538 INFO:tasks.ceph.mon.d.smithi088.stderr:2023-09-30T01:37:51.531+0000 7fabfe1e4700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-09-30T01:37:51.538 INFO:tasks.ceph.mon.d.smithi088.stderr:2023-09-30T01:37:51.531+0000 7fabfe1e4700 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=4052848133}) 2023-09-30T01:37:51.538 INFO:tasks.ceph.mon.d.smithi088.stderr:2023-09-30T01:37:51.531+0000 7fabfe1e4700 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3415058479}) 2023-09-30T01:37:51.538 INFO:tasks.ceph.mon.d.smithi088.stderr:2023-09-30T01:37:51.531+0000 7fabfe1e4700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-09-30T01:37:51.538 INFO:tasks.ceph.mon.d.smithi088.stderr:2023-09-30T01:37:51.531+0000 7fabfe1e4700 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=4052848133}) 2023-09-30T01:37:51.538 INFO:tasks.ceph.mon.d.smithi088.stderr:2023-09-30T01:37:51.531+0000 7fabfe1e4700 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3423161446}) 2023-09-30T01:37:51.538 INFO:tasks.ceph.mon.d.smithi088.stderr:2023-09-30T01:37:51.531+0000 7fabfe1e4700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-09-30T01:37:51.539 INFO:tasks.ceph.mon.d.smithi088.stderr:2023-09-30T01:37:51.531+0000 7fabfe1e4700 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=4052848133}) 2023-09-30T01:37:51.539 INFO:tasks.ceph.mon.d.smithi088.stderr:2023-09-30T01:37:51.531+0000 7fabfe1e4700 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1754641625}) 2023-09-30T01:37:51.551 INFO:tasks.ceph.mon.d.smithi088.stderr:2023-09-30T01:37:51.547+0000 7fabfe1e4700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-09-30T01:37:51.551 INFO:tasks.ceph.mon.d.smithi088.stderr:2023-09-30T01:37:51.547+0000 7fabfe1e4700 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2318080887}) 2023-09-30T01:37:51.551 INFO:tasks.ceph.mon.d.smithi088.stderr:2023-09-30T01:37:51.547+0000 7fabfe1e4700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=843099680}) 2023-09-30T01:37:51.551 INFO:tasks.ceph.mon.d.smithi088.stderr:2023-09-30T01:37:51.547+0000 7fabfe1e4700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-09-30T01:37:51.551 INFO:tasks.ceph.mon.d.smithi088.stderr:2023-09-30T01:37:51.547+0000 7fabfe1e4700 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2318080887}) 2023-09-30T01:37:51.551 INFO:tasks.ceph.mon.d.smithi088.stderr:2023-09-30T01:37:51.547+0000 7fabfe1e4700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3019596842}) 2023-09-30T01:37:51.551 INFO:tasks.ceph.mon.d.smithi088.stderr:2023-09-30T01:37:51.547+0000 7fabfe1e4700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-09-30T01:37:51.551 INFO:tasks.ceph.mon.d.smithi088.stderr:2023-09-30T01:37:51.547+0000 7fabfe1e4700 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2318080887}) 2023-09-30T01:37:51.551 INFO:tasks.ceph.mon.d.smithi088.stderr:2023-09-30T01:37:51.547+0000 7fabfe1e4700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2084958429}) 2023-09-30T01:37:51.552 INFO:tasks.ceph.mon.d.smithi088.stderr:2023-09-30T01:37:51.547+0000 7fabfe1e4700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-09-30T01:37:51.552 INFO:tasks.ceph.mon.d.smithi088.stderr:2023-09-30T01:37:51.547+0000 7fabfe1e4700 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2318080887}) 2023-09-30T01:37:51.552 INFO:tasks.ceph.mon.d.smithi088.stderr:2023-09-30T01:37:51.547+0000 7fabfe1e4700 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2475898129}) 2023-09-30T01:37:51.552 INFO:tasks.ceph.mon.d.smithi088.stderr:2023-09-30T01:37:51.547+0000 7fabfe1e4700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-09-30T01:37:51.552 INFO:tasks.ceph.mon.d.smithi088.stderr:2023-09-30T01:37:51.547+0000 7fabfe1e4700 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2318080887}) 2023-09-30T01:37:51.552 INFO:tasks.ceph.mon.d.smithi088.stderr:2023-09-30T01:37:51.547+0000 7fabfe1e4700 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=398891747}) 2023-09-30T01:37:51.552 INFO:tasks.ceph.mon.d.smithi088.stderr:2023-09-30T01:37:51.547+0000 7fabfe1e4700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-09-30T01:37:51.552 INFO:tasks.ceph.mon.d.smithi088.stderr:2023-09-30T01:37:51.547+0000 7fabfe1e4700 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2318080887}) 2023-09-30T01:37:51.553 INFO:tasks.ceph.mon.d.smithi088.stderr:2023-09-30T01:37:51.547+0000 7fabfe1e4700 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1372372135}) 2023-09-30T01:37:51.553 INFO:tasks.ceph.mon.d.smithi088.stderr:2023-09-30T01:37:51.547+0000 7fabfe1e4700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-09-30T01:37:51.553 INFO:tasks.ceph.mon.d.smithi088.stderr:2023-09-30T01:37:51.547+0000 7fabfe1e4700 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2318080887}) 2023-09-30T01:37:51.553 INFO:tasks.ceph.mon.d.smithi088.stderr:2023-09-30T01:37:51.547+0000 7fabfe1e4700 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=347473243}) 2023-09-30T01:37:51.553 INFO:tasks.ceph.mon.d.smithi088.stderr:2023-09-30T01:37:51.547+0000 7fabfe1e4700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-09-30T01:37:51.553 INFO:tasks.ceph.mon.d.smithi088.stderr:2023-09-30T01:37:51.547+0000 7fabfe1e4700 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2318080887}) 2023-09-30T01:37:51.553 INFO:tasks.ceph.mon.d.smithi088.stderr:2023-09-30T01:37:51.547+0000 7fabfe1e4700 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3292176395}) 2023-09-30T01:37:51.558 INFO:tasks.ceph.mon.d.smithi088.stderr:2023-09-30T01:37:51.551+0000 7fabfe1e4700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-09-30T01:37:51.558 INFO:tasks.ceph.mon.d.smithi088.stderr:2023-09-30T01:37:51.551+0000 7fabfe1e4700 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1326298206}) 2023-09-30T01:37:51.558 INFO:tasks.ceph.mon.d.smithi088.stderr:2023-09-30T01:37:51.551+0000 7fabfe1e4700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2785011480}) 2023-09-30T01:37:51.559 INFO:tasks.ceph.mon.d.smithi088.stderr:2023-09-30T01:37:51.551+0000 7fabfe1e4700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-09-30T01:37:51.559 INFO:tasks.ceph.mon.d.smithi088.stderr:2023-09-30T01:37:51.551+0000 7fabfe1e4700 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1326298206}) 2023-09-30T01:37:51.559 INFO:tasks.ceph.mon.d.smithi088.stderr:2023-09-30T01:37:51.551+0000 7fabfe1e4700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1271058125}) 2023-09-30T01:37:51.559 INFO:tasks.ceph.mon.d.smithi088.stderr:2023-09-30T01:37:51.551+0000 7fabfe1e4700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-09-30T01:37:51.559 INFO:tasks.ceph.mon.d.smithi088.stderr:2023-09-30T01:37:51.555+0000 7fabfe1e4700 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1326298206}) 2023-09-30T01:37:51.559 INFO:tasks.ceph.mon.d.smithi088.stderr:2023-09-30T01:37:51.555+0000 7fabfe1e4700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3951385390}) 2023-09-30T01:37:51.560 INFO:tasks.ceph.mon.d.smithi088.stderr:2023-09-30T01:37:51.555+0000 7fabfe1e4700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-09-30T01:37:51.560 INFO:tasks.ceph.mon.d.smithi088.stderr:2023-09-30T01:37:51.555+0000 7fabfe1e4700 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1326298206}) 2023-09-30T01:37:51.560 INFO:tasks.ceph.mon.d.smithi088.stderr:2023-09-30T01:37:51.555+0000 7fabfe1e4700 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4163614486}) 2023-09-30T01:37:51.560 INFO:tasks.ceph.mon.d.smithi088.stderr:2023-09-30T01:37:51.555+0000 7fabfe1e4700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-09-30T01:37:51.560 INFO:tasks.ceph.mon.d.smithi088.stderr:2023-09-30T01:37:51.555+0000 7fabfe1e4700 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1326298206}) 2023-09-30T01:37:51.560 INFO:tasks.ceph.mon.d.smithi088.stderr:2023-09-30T01:37:51.555+0000 7fabfe1e4700 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3008580153}) 2023-09-30T01:37:51.560 INFO:tasks.ceph.mon.d.smithi088.stderr:2023-09-30T01:37:51.555+0000 7fabfe1e4700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-09-30T01:37:51.561 INFO:tasks.ceph.mon.d.smithi088.stderr:2023-09-30T01:37:51.555+0000 7fabfe1e4700 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1326298206}) 2023-09-30T01:37:51.561 INFO:tasks.ceph.mon.d.smithi088.stderr:2023-09-30T01:37:51.555+0000 7fabfe1e4700 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1184426196}) 2023-09-30T01:37:51.561 INFO:tasks.ceph.mon.d.smithi088.stderr:2023-09-30T01:37:51.555+0000 7fabfe1e4700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-09-30T01:37:51.561 INFO:tasks.ceph.mon.d.smithi088.stderr:2023-09-30T01:37:51.555+0000 7fabfe1e4700 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1326298206}) 2023-09-30T01:37:51.561 INFO:tasks.ceph.mon.d.smithi088.stderr:2023-09-30T01:37:51.555+0000 7fabfe1e4700 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3034918777}) 2023-09-30T01:37:51.561 INFO:tasks.ceph.mon.d.smithi088.stderr:2023-09-30T01:37:51.555+0000 7fabfe1e4700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-09-30T01:37:51.561 INFO:tasks.ceph.mon.d.smithi088.stderr:2023-09-30T01:37:51.555+0000 7fabfe1e4700 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1326298206}) 2023-09-30T01:37:51.561 INFO:tasks.ceph.mon.d.smithi088.stderr:2023-09-30T01:37:51.555+0000 7fabfe1e4700 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1527324281}) 2023-09-30T01:37:51.565 INFO:tasks.ceph.mon.d.smithi088.stderr:2023-09-30T01:37:51.555+0000 7fabfe1e4700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-09-30T01:37:51.565 INFO:tasks.ceph.mon.d.smithi088.stderr:2023-09-30T01:37:51.555+0000 7fabfe1e4700 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=719466296}) 2023-09-30T01:37:51.565 INFO:tasks.ceph.mon.d.smithi088.stderr:2023-09-30T01:37:51.555+0000 7fabfe1e4700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3794324540}) 2023-09-30T01:37:51.565 INFO:tasks.ceph.mon.d.smithi088.stderr:2023-09-30T01:37:51.555+0000 7fabfe1e4700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-09-30T01:37:51.565 INFO:tasks.ceph.mon.d.smithi088.stderr:2023-09-30T01:37:51.555+0000 7fabfe1e4700 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=719466296}) 2023-09-30T01:37:51.566 INFO:tasks.ceph.mon.d.smithi088.stderr:2023-09-30T01:37:51.555+0000 7fabfe1e4700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3165148554}) 2023-09-30T01:37:51.566 INFO:tasks.ceph.mon.d.smithi088.stderr:2023-09-30T01:37:51.555+0000 7fabfe1e4700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-09-30T01:37:51.566 INFO:tasks.ceph.mon.d.smithi088.stderr:2023-09-30T01:37:51.555+0000 7fabfe1e4700 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=719466296}) 2023-09-30T01:37:51.566 INFO:tasks.ceph.mon.d.smithi088.stderr:2023-09-30T01:37:51.555+0000 7fabfe1e4700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1663492112}) 2023-09-30T01:37:51.566 INFO:tasks.ceph.mon.d.smithi088.stderr:2023-09-30T01:37:51.555+0000 7fabfe1e4700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-09-30T01:37:51.566 INFO:tasks.ceph.mon.d.smithi088.stderr:2023-09-30T01:37:51.555+0000 7fabfe1e4700 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=719466296}) 2023-09-30T01:37:51.567 INFO:tasks.ceph.mon.d.smithi088.stderr:2023-09-30T01:37:51.555+0000 7fabfe1e4700 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=939909731}) 2023-09-30T01:37:51.567 INFO:tasks.ceph.mon.d.smithi088.stderr:2023-09-30T01:37:51.555+0000 7fabfe1e4700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-09-30T01:37:51.567 INFO:tasks.ceph.mon.d.smithi088.stderr:2023-09-30T01:37:51.555+0000 7fabfe1e4700 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=719466296}) 2023-09-30T01:37:51.567 INFO:tasks.ceph.mon.d.smithi088.stderr:2023-09-30T01:37:51.555+0000 7fabfe1e4700 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2324214118}) 2023-09-30T01:37:51.567 INFO:tasks.ceph.mon.d.smithi088.stderr:2023-09-30T01:37:51.555+0000 7fabfe1e4700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-09-30T01:37:51.567 INFO:tasks.ceph.mon.d.smithi088.stderr:2023-09-30T01:37:51.555+0000 7fabfe1e4700 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=719466296}) 2023-09-30T01:37:51.568 INFO:tasks.ceph.mon.d.smithi088.stderr:2023-09-30T01:37:51.555+0000 7fabfe1e4700 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1974603965}) 2023-09-30T01:37:51.568 INFO:tasks.ceph.mon.d.smithi088.stderr:2023-09-30T01:37:51.555+0000 7fabfe1e4700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-09-30T01:37:51.568 INFO:tasks.ceph.mon.d.smithi088.stderr:2023-09-30T01:37:51.555+0000 7fabfe1e4700 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=719466296}) 2023-09-30T01:37:51.568 INFO:tasks.ceph.mon.d.smithi088.stderr:2023-09-30T01:37:51.555+0000 7fabfe1e4700 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3923092444}) 2023-09-30T01:37:51.568 INFO:tasks.ceph.mon.d.smithi088.stderr:2023-09-30T01:37:51.555+0000 7fabfe1e4700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-09-30T01:37:51.568 INFO:tasks.ceph.mon.d.smithi088.stderr:2023-09-30T01:37:51.555+0000 7fabfe1e4700 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=719466296}) 2023-09-30T01:37:51.568 INFO:tasks.ceph.mon.d.smithi088.stderr:2023-09-30T01:37:51.555+0000 7fabfe1e4700 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2899496077}) 2023-09-30T01:37:51.571 INFO:tasks.ceph.mon.d.smithi088.stderr:2023-09-30T01:37:51.567+0000 7fabfe1e4700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-09-30T01:37:51.571 INFO:tasks.ceph.mon.d.smithi088.stderr:2023-09-30T01:37:51.567+0000 7fabfe1e4700 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=19,mdsmap=3,mgr=6,mgr_command_descs=1,mgrstat=8,mon_config_key=5,monmap=3,osd_metadata=6,osd_pg_creating=1,osdmap=28} crc {logm=3327669458,mdsmap=1531151339,mgr=4075273690,mgr_command_descs=3745887455,mgrstat=327167559,mon_config_key=3381236218,monmap=3171275568,osd_metadata=2082802069,osd_pg_creating=534015559,osdmap=2444968561}) 2023-09-30T01:37:51.572 INFO:tasks.ceph.mon.d.smithi088.stderr:2023-09-30T01:37:51.567+0000 7fabfe1e4700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=19,mdsmap=2,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=8,mon_config_key=5,monmap=3,osd_metadata=5,osd_pg_creating=1,osdmap=29} crc {logm=1866226316,mdsmap=3585761405,mgr=3995181390,mgr_command_descs=3745887455,mgr_metadata=137767681,mgrstat=327167559,mon_config_key=3381236218,monmap=3171275568,osd_metadata=4184883115,osd_pg_creating=534015559,osdmap=612455168}) 2023-09-30T01:37:51.572 INFO:tasks.ceph.mon.d.smithi088.stderr:2023-09-30T01:37:51.567+0000 7fabfe1e4700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-09-30T01:37:51.572 INFO:tasks.ceph.mon.d.smithi088.stderr:2023-09-30T01:37:51.567+0000 7fabfe1e4700 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=19,mdsmap=3,mgr=6,mgr_command_descs=1,mgrstat=8,mon_config_key=5,monmap=3,osd_metadata=6,osd_pg_creating=1,osdmap=28} crc {logm=3327669458,mdsmap=1531151339,mgr=4075273690,mgr_command_descs=3745887455,mgrstat=327167559,mon_config_key=3381236218,monmap=3171275568,osd_metadata=2082802069,osd_pg_creating=534015559,osdmap=2444968561}) 2023-09-30T01:37:51.572 INFO:tasks.ceph.mon.d.smithi088.stderr:2023-09-30T01:37:51.567+0000 7fabfe1e4700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=20,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=8,mon_config_key=4,monmap=3,osd_metadata=6,osd_pg_creating=1,osdmap=30} crc {logm=2833829423,mdsmap=1531151339,mgr=2475652484,mgr_command_descs=3745887455,mgr_metadata=137767681,mgrstat=327167559,mon_config_key=1954089980,monmap=3171275568,osd_metadata=2082802069,osd_pg_creating=534015559,osdmap=3655256246}) 2023-09-30T01:37:51.572 INFO:tasks.ceph.mon.d.smithi088.stderr:2023-09-30T01:37:51.567+0000 7fabfe1e4700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-09-30T01:37:51.572 INFO:tasks.ceph.mon.d.smithi088.stderr:2023-09-30T01:37:51.567+0000 7fabfe1e4700 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=19,mdsmap=3,mgr=6,mgr_command_descs=1,mgrstat=8,mon_config_key=5,monmap=3,osd_metadata=6,osd_pg_creating=1,osdmap=28} crc {logm=3327669458,mdsmap=1531151339,mgr=4075273690,mgr_command_descs=3745887455,mgrstat=327167559,mon_config_key=3381236218,monmap=3171275568,osd_metadata=2082802069,osd_pg_creating=534015559,osdmap=2444968561}) 2023-09-30T01:37:51.573 INFO:tasks.ceph.mon.d.smithi088.stderr:2023-09-30T01:37:51.567+0000 7fabfe1e4700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=19,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=7,mon_config_key=5,monmap=3,osd_metadata=6,osd_pg_creating=1,osdmap=31} crc {logm=4003063377,mdsmap=1531151339,mgr=3578812728,mgr_command_descs=3745887455,mgr_metadata=137767681,mgrstat=2143155252,mon_config_key=3381236218,monmap=3171275568,osd_metadata=2082802069,osd_pg_creating=534015559,osdmap=4279994431}) 2023-09-30T01:37:51.573 INFO:tasks.ceph.mon.d.smithi088.stderr:2023-09-30T01:37:51.567+0000 7fabfe1e4700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-09-30T01:37:51.573 INFO:tasks.ceph.mon.d.smithi088.stderr:2023-09-30T01:37:51.567+0000 7fabfe1e4700 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=19,mdsmap=3,mgr=6,mgr_command_descs=1,mgrstat=8,mon_config_key=5,monmap=3,osd_metadata=6,osd_pg_creating=1,osdmap=28} crc {logm=3327669458,mdsmap=1531151339,mgr=4075273690,mgr_command_descs=3745887455,mgrstat=327167559,mon_config_key=3381236218,monmap=3171275568,osd_metadata=2082802069,osd_pg_creating=534015559,osdmap=2444968561}) 2023-09-30T01:37:51.573 INFO:tasks.ceph.mon.d.smithi088.stderr:2023-09-30T01:37:51.567+0000 7fabfe1e4700 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=19,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=8,mon_config_key=5,monmap=3,osd_metadata=6,osd_pg_creating=1,osdmap=33} crc {logm=726223979,mdsmap=1531151339,mgr=3995181390,mgr_command_descs=3745887455,mgr_metadata=137767681,mgrstat=327167559,mon_config_key=3381236218,monmap=3171275568,osd_metadata=2082802069,osd_pg_creating=534015559,osdmap=424077960}) 2023-09-30T01:37:51.573 INFO:tasks.ceph.mon.d.smithi088.stderr:2023-09-30T01:37:51.567+0000 7fabfe1e4700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-09-30T01:37:51.573 INFO:tasks.ceph.mon.d.smithi088.stderr:2023-09-30T01:37:51.567+0000 7fabfe1e4700 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=19,mdsmap=3,mgr=6,mgr_command_descs=1,mgrstat=8,mon_config_key=5,monmap=3,osd_metadata=6,osd_pg_creating=1,osdmap=28} crc {logm=3327669458,mdsmap=1531151339,mgr=4075273690,mgr_command_descs=3745887455,mgrstat=327167559,mon_config_key=3381236218,monmap=3171275568,osd_metadata=2082802069,osd_pg_creating=534015559,osdmap=2444968561}) 2023-09-30T01:37:51.574 INFO:tasks.ceph.mon.d.smithi088.stderr:2023-09-30T01:37:51.567+0000 7fabfe1e4700 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=19,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=7,mon_config_key=5,monmap=2,osd_metadata=6,osd_pg_creating=1,osdmap=29} crc {logm=2210648667,mdsmap=1531151339,mgr=3995181390,mgr_command_descs=3745887455,mgr_metadata=137767681,mgrstat=675700468,mon_config_key=3381236218,monmap=119882974,osd_metadata=2082802069,osd_pg_creating=534015559,osdmap=3082195530}) 2023-09-30T01:37:51.574 INFO:tasks.ceph.mon.d.smithi088.stderr:2023-09-30T01:37:51.567+0000 7fabfe1e4700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-09-30T01:37:51.574 INFO:tasks.ceph.mon.d.smithi088.stderr:2023-09-30T01:37:51.567+0000 7fabfe1e4700 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=19,mdsmap=3,mgr=6,mgr_command_descs=1,mgrstat=8,mon_config_key=5,monmap=3,osd_metadata=6,osd_pg_creating=1,osdmap=28} crc {logm=3327669458,mdsmap=1531151339,mgr=4075273690,mgr_command_descs=3745887455,mgrstat=327167559,mon_config_key=3381236218,monmap=3171275568,osd_metadata=2082802069,osd_pg_creating=534015559,osdmap=2444968561}) 2023-09-30T01:37:51.574 INFO:tasks.ceph.mon.d.smithi088.stderr:2023-09-30T01:37:51.567+0000 7fabfe1e4700 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=18,mdsmap=2,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=8,mon_config_key=5,monmap=3,osd_metadata=6,osd_pg_creating=1,osdmap=32} crc {logm=4195362827,mdsmap=3585761405,mgr=3995181391,mgr_command_descs=3745887455,mgr_metadata=137767681,mgrstat=327167559,mon_config_key=3381236218,monmap=3171275568,osd_metadata=2082802069,osd_pg_creating=534015559,osdmap=3079615681}) 2023-09-30T01:37:51.574 INFO:tasks.ceph.mon.d.smithi088.stderr:2023-09-30T01:37:51.567+0000 7fabfe1e4700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-09-30T01:37:51.574 INFO:tasks.ceph.mon.d.smithi088.stderr:2023-09-30T01:37:51.567+0000 7fabfe1e4700 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=19,mdsmap=3,mgr=6,mgr_command_descs=1,mgrstat=8,mon_config_key=5,monmap=3,osd_metadata=6,osd_pg_creating=1,osdmap=28} crc {logm=3327669458,mdsmap=1531151339,mgr=4075273690,mgr_command_descs=3745887455,mgrstat=327167559,mon_config_key=3381236218,monmap=3171275568,osd_metadata=2082802069,osd_pg_creating=534015559,osdmap=2444968561}) 2023-09-30T01:37:51.575 INFO:tasks.ceph.mon.d.smithi088.stderr:2023-09-30T01:37:51.567+0000 7fabfe1e4700 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=18,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=8,mon_config_key=4,monmap=3,osd_metadata=6,osd_pg_creating=1,osdmap=29} crc {logm=2772192272,mdsmap=1531151339,mgr=3995181390,mgr_command_descs=3745887455,mgr_metadata=137767681,mgrstat=2905396382,mon_config_key=2093487035,monmap=3171275568,osd_metadata=2082802069,osd_pg_creating=534015559,osdmap=1299449624}) 2023-09-30T01:37:51.575 INFO:tasks.ceph.mon.d.smithi088.stderr:2023-09-30T01:37:51.567+0000 7fabfe1e4700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-09-30T01:37:51.575 INFO:tasks.ceph.mon.d.smithi088.stderr:2023-09-30T01:37:51.567+0000 7fabfe1e4700 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=19,mdsmap=3,mgr=6,mgr_command_descs=1,mgrstat=8,mon_config_key=5,monmap=3,osd_metadata=6,osd_pg_creating=1,osdmap=28} crc {logm=3327669458,mdsmap=1531151339,mgr=4075273690,mgr_command_descs=3745887455,mgrstat=327167559,mon_config_key=3381236218,monmap=3171275568,osd_metadata=2082802069,osd_pg_creating=534015559,osdmap=2444968561}) 2023-09-30T01:37:51.575 INFO:tasks.ceph.mon.d.smithi088.stderr:2023-09-30T01:37:51.567+0000 7fabfe1e4700 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=20,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=8,mon_config_key=5,monmap=3,osd_metadata=6,osd_pg_creating=1,osdmap=32} crc {logm=2833829423,mdsmap=1531151339,mgr=3995181390,mgr_command_descs=3745887455,mgr_metadata=137767681,mgrstat=327167559,mon_config_key=3381236218,monmap=3171275568,osd_metadata=2082802069,osd_pg_creating=534015559,osdmap=2390628490}) 2023-09-30T01:37:52.222 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:37:52.335 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:52.331+0000 7fc60b59e700 10 monclient: get_monmap_and_config 2023-09-30T01:37:52.335 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:52.331+0000 7fc60b59e700 10 monclient: build_initial_monmap 2023-09-30T01:37:52.335 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:52.331+0000 7fc60b59e700 10 monclient: monmap: 2023-09-30T01:37:52.335 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2023-09-30T01:37:52.336 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:37:52.336 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:37:52.335159+0000 2023-09-30T01:37:52.336 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:37:52.335159+0000 2023-09-30T01:37:52.336 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2023-09-30T01:37:52.336 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:37:52.336 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a-legacy 2023-09-30T01:37:52.336 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.107:6789/0 mon.noname-f-legacy 2023-09-30T01:37:52.337 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.noname-b 2023-09-30T01:37:52.337 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.noname-g 2023-09-30T01:37:52.337 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.noname-c 2023-09-30T01:37:52.337 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.noname-h 2023-09-30T01:37:52.337 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.noname-d 2023-09-30T01:37:52.337 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.noname-i 2023-09-30T01:37:52.337 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.noname-e 2023-09-30T01:37:52.338 INFO:teuthology.orchestra.run.smithi088.stderr:9: v2:172.21.15.88:3300/0 mon.noname-a 2023-09-30T01:37:52.338 INFO:teuthology.orchestra.run.smithi088.stderr:10: v2:172.21.15.107:3300/0 mon.noname-f 2023-09-30T01:37:52.338 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:37:52.346 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:52.335+0000 7fc60b59e700 1 Processor -- start 2023-09-30T01:37:52.346 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:52.335+0000 7fc60b59e700 1 -- start start 2023-09-30T01:37:52.346 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:52.335+0000 7fc60b59e700 10 monclient: init 2023-09-30T01:37:52.346 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:52.335+0000 7fc60b59e700 10 monclient: _reopen_session rank -1 2023-09-30T01:37:52.346 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:52.335+0000 7fc60b59e700 10 monclient: _add_conns ranks=[10,8,3,5,2,4,7,9,0,6,1] 2023-09-30T01:37:52.346 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:52.335+0000 7fc60b59e700 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fc60c0a3380 0x7fc60c0a48b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:37:52.347 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:52.335+0000 7fc60b59e700 10 monclient(hunting): picked mon.noname-f con 0x7fc60c0a3380 addr v2:172.21.15.107:3300/0 2023-09-30T01:37:52.347 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:52.335+0000 7fc60b59e700 1 --2- >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7fc60c09ffd0 0x7fc60c0a03a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:37:52.347 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:52.335+0000 7fc60b59e700 10 monclient(hunting): picked mon.noname-e con 0x7fc60c09ffd0 addr [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] 2023-09-30T01:37:52.347 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:52.335+0000 7fc60b59e700 1 --2- >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7fc60c1333b0 0x7fc60c133820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:37:52.347 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:52.335+0000 7fc60b59e700 10 monclient(hunting): picked mon.noname-g con 0x7fc60c1333b0 addr [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] 2023-09-30T01:37:52.347 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:52.335+0000 7fc60b59e700 10 monclient(hunting): start opening mon connection 2023-09-30T01:37:52.347 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:52.335+0000 7fc60b59e700 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc60c097ab0 con 0x7fc60c0a3380 2023-09-30T01:37:52.347 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:52.335+0000 7fc60b59e700 10 monclient(hunting): start opening mon connection 2023-09-30T01:37:52.347 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:52.335+0000 7fc60b59e700 1 -- --> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc60c0a08e0 con 0x7fc60c1333b0 2023-09-30T01:37:52.347 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:52.335+0000 7fc60b59e700 10 monclient(hunting): start opening mon connection 2023-09-30T01:37:52.348 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:52.335+0000 7fc60b59e700 1 -- --> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] -- mon_getmap magic: 0 v1 -- 0x7fc60c098480 con 0x7fc60c09ffd0 2023-09-30T01:37:52.348 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:52.335+0000 7fc60b59e700 10 monclient(hunting): _renew_subs 2023-09-30T01:37:52.348 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:52.335+0000 7fc60b59e700 10 monclient(hunting): authenticate will time out at 1144.372663s 2023-09-30T01:37:52.348 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:52.335+0000 7fc609d9b700 1 --2- >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7fc60c09ffd0 0x7fc60c0a03a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:37:52.348 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:52.335+0000 7fc609d9b700 1 --2- >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7fc60c09ffd0 0x7fc60c0a03a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3304/0 says I am v2:172.21.15.88:59752/0 (socket says 172.21.15.88:59752) 2023-09-30T01:37:52.348 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:52.335+0000 7fc609d9b700 1 -- 172.21.15.88:0/2852481013 learned_addr learned my addr 172.21.15.88:0/2852481013 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-09-30T01:37:52.348 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:52.335+0000 7fc609d9b700 10 monclient(hunting): get_auth_request con 0x7fc60c09ffd0 auth_method 0 2023-09-30T01:37:52.348 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:52.335+0000 7fc609d9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:37:52.348 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:52.335+0000 7fc609d9b700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:37:52.348 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:52.335+0000 7fc609d9b700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:37:52.348 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:52.335+0000 7fc609d9b700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:37:52.348 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:52.335+0000 7fc609d9b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:37:52.349 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:52.335+0000 7fc609d9b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:37:52.349 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:52.335+0000 7fc60ad9d700 1 --2- 172.21.15.88:0/2852481013 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7fc60c1333b0 0x7fc60c133820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:37:52.349 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:52.335+0000 7fc60a59c700 1 --2- 172.21.15.88:0/2852481013 >> v2:172.21.15.107:3300/0 conn(0x7fc60c0a3380 0x7fc60c0a48b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:37:52.349 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:52.335+0000 7fc60ad9d700 10 monclient(hunting): get_auth_request con 0x7fc60c1333b0 auth_method 0 2023-09-30T01:37:52.349 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:52.335+0000 7fc60ad9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:37:52.349 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:52.335+0000 7fc60ad9d700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:37:52.349 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:52.335+0000 7fc60ad9d700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:37:52.349 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:52.335+0000 7fc60a59c700 10 monclient(hunting): get_auth_request con 0x7fc60c0a3380 auth_method 0 2023-09-30T01:37:52.349 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:52.335+0000 7fc60a59c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:37:52.349 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:52.335+0000 7fc60a59c700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:37:52.349 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:52.335+0000 7fc60a59c700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:37:52.349 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:52.335+0000 7fc609d9b700 10 monclient(hunting): handle_auth_done global_id 4868 payload 274 2023-09-30T01:37:52.349 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:52.335+0000 7fc609d9b700 1 -- 172.21.15.88:0/2852481013 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7fc60c1333b0 msgr2=0x7fc60c133820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:37:52.350 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:52.335+0000 7fc609d9b700 1 --2- 172.21.15.88:0/2852481013 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7fc60c1333b0 0x7fc60c133820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:37:52.350 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:52.335+0000 7fc609d9b700 1 -- 172.21.15.88:0/2852481013 >> v2:172.21.15.107:3300/0 conn(0x7fc60c0a3380 msgr2=0x7fc60c0a48b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:37:52.350 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:52.335+0000 7fc609d9b700 1 --2- 172.21.15.88:0/2852481013 >> v2:172.21.15.107:3300/0 conn(0x7fc60c0a3380 0x7fc60c0a48b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:37:52.350 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:52.335+0000 7fc609d9b700 10 monclient: _finish_hunting 0 2023-09-30T01:37:52.350 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:52.335+0000 7fc609d9b700 1 monclient: found mon.noname-e 2023-09-30T01:37:52.350 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:52.335+0000 7fc609d9b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:37:52.350 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:52.335+0000 7fc609d9b700 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.88:3304/0 2023-09-30T01:37:52.350 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:52.335+0000 7fc609d9b700 1 -- 172.21.15.88:0/2852481013 --> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc60c04bb20 con 0x7fc60c09ffd0 2023-09-30T01:37:52.350 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:52.335+0000 7fc609d9b700 1 --2- 172.21.15.88:0/2852481013 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7fc60c09ffd0 0x7fc60c0a03a0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fc5fc0099e0 tx=0x7fc5fc009250 comp rx=0 tx=0).ready entity=mon.8 client_cookie=af8edb58598c725a server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:37:52.350 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:52.335+0000 7fc60959a700 1 -- 172.21.15.88:0/2852481013 <== mon.8 v2:172.21.15.88:3304/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7fc5fc00bca0 con 0x7fc60c09ffd0 2023-09-30T01:37:52.350 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:52.335+0000 7fc60959a700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:37:52.350 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:52.335+0000 7fc60959a700 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2023-09-30T01:37:52.351 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:52.335+0000 7fc60959a700 10 monclient: dump: 2023-09-30T01:37:52.351 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:37:52.351 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:37:52.351 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:37:52.351 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:37:52.351 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:37:52.351 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:37:52.351 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:37:52.351 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:37:52.351 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:37:52.351 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:37:52.351 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:37:52.351 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:37:52.352 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:37:52.352 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:37:52.352 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:37:52.352 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:37:52.352 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:52.335+0000 7fc60959a700 10 monclient: _finish_auth 0 2023-09-30T01:37:52.352 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:52.335+0000 7fc60959a700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:37:52.352 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:52.335+0000 7fc60b59e700 5 monclient: authenticate success, global_id 4868 2023-09-30T01:37:52.352 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:52.335+0000 7fc60b59e700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:37:52.352 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:52.335+0000 7fc60959a700 1 -- 172.21.15.88:0/2852481013 <== mon.8 v2:172.21.15.88:3304/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc5fc00be20 con 0x7fc60c09ffd0 2023-09-30T01:37:52.353 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:52.335+0000 7fc60959a700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:37:52.353 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:52.335+0000 7fc60b59e700 10 monclient: get_monmap_and_config success 2023-09-30T01:37:52.353 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:52.335+0000 7fc60b59e700 4 set_mon_vals no callback set 2023-09-30T01:37:52.353 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:52.335+0000 7fc60959a700 1 -- 172.21.15.88:0/2852481013 <== mon.8 v2:172.21.15.88:3304/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7fc5fc0118a0 con 0x7fc60c09ffd0 2023-09-30T01:37:52.353 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:52.335+0000 7fc60959a700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:37:52.353 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:52.335+0000 7fc60959a700 10 monclient: got monmap 1 from mon.e (according to old e1) 2023-09-30T01:37:52.353 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:52.335+0000 7fc60959a700 10 monclient: dump: 2023-09-30T01:37:52.353 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:37:52.353 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:37:52.353 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:37:52.354 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:37:52.354 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:37:52.354 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:37:52.354 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:37:52.354 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:37:52.354 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:37:52.354 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:37:52.354 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:37:52.354 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:37:52.355 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:37:52.355 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:37:52.355 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:37:52.355 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:37:52.355 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:52.335+0000 7fc60b59e700 10 monclient: shutdown 2023-09-30T01:37:52.355 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:52.335+0000 7fc60b59e700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:37:52.355 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:52.335+0000 7fc60b59e700 1 -- 172.21.15.88:0/2852481013 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7fc60c09ffd0 msgr2=0x7fc60c0a03a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:37:52.355 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:52.335+0000 7fc60b59e700 1 --2- 172.21.15.88:0/2852481013 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7fc60c09ffd0 0x7fc60c0a03a0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fc5fc0099e0 tx=0x7fc5fc009250 comp rx=0 tx=0).stop 2023-09-30T01:37:52.355 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:52.335+0000 7fc60b59e700 1 -- 172.21.15.88:0/2852481013 shutdown_connections 2023-09-30T01:37:52.356 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:52.335+0000 7fc60b59e700 1 --2- 172.21.15.88:0/2852481013 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7fc60c1333b0 0x7fc60c133820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:37:52.356 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:52.335+0000 7fc60b59e700 1 --2- 172.21.15.88:0/2852481013 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7fc60c09ffd0 0x7fc60c0a03a0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:37:52.356 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:52.335+0000 7fc60b59e700 1 --2- 172.21.15.88:0/2852481013 >> v2:172.21.15.107:3300/0 conn(0x7fc60c0a3380 0x7fc60c0a48b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:37:52.356 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:52.335+0000 7fc60b59e700 1 -- 172.21.15.88:0/2852481013 >> 172.21.15.88:0/2852481013 conn(0x7fc60c06e5e0 msgr2=0x7fc60c06e9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:37:52.356 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:52.335+0000 7fc60b59e700 1 -- 172.21.15.88:0/2852481013 shutdown_connections 2023-09-30T01:37:52.356 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:52.335+0000 7fc60b59e700 1 -- 172.21.15.88:0/2852481013 wait complete. 2023-09-30T01:37:52.356 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:52.335+0000 7fc60b59e700 10 monclient: build_initial_monmap 2023-09-30T01:37:52.356 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:52.335+0000 7fc60b59e700 10 monclient: monmap: 2023-09-30T01:37:52.356 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2023-09-30T01:37:52.356 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:37:52.357 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 0.000000 2023-09-30T01:37:52.357 INFO:teuthology.orchestra.run.smithi088.stderr:created 0.000000 2023-09-30T01:37:52.357 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2023-09-30T01:37:52.357 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 1 2023-09-30T01:37:52.357 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.noname-a 2023-09-30T01:37:52.357 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.noname-f 2023-09-30T01:37:52.357 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.noname-b 2023-09-30T01:37:52.357 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.noname-g 2023-09-30T01:37:52.357 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.noname-c 2023-09-30T01:37:52.357 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.noname-h 2023-09-30T01:37:52.358 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.noname-d 2023-09-30T01:37:52.358 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.noname-i 2023-09-30T01:37:52.358 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.noname-e 2023-09-30T01:37:52.358 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:37:52.358 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:52.339+0000 7fc60b59e700 1 Processor -- start 2023-09-30T01:37:52.358 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:52.339+0000 7fc60b59e700 1 -- start start 2023-09-30T01:37:52.358 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:52.339+0000 7fc60b59e700 10 monclient: init 2023-09-30T01:37:52.358 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:52.339+0000 7fc60b59e700 10 monclient: _reopen_session rank -1 2023-09-30T01:37:52.358 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:52.339+0000 7fc60b59e700 10 monclient: _add_conns ranks=[3,6,2,7,5,1,4,0,8] 2023-09-30T01:37:52.358 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:52.339+0000 7fc60b59e700 1 --2- >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7fc60c09ffd0 0x7fc60c11cf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:37:52.358 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:52.339+0000 7fc60b59e700 10 monclient(hunting): picked mon.noname-g con 0x7fc60c09ffd0 addr [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] 2023-09-30T01:37:52.359 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:52.339+0000 7fc60b59e700 1 --2- >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7fc60c0a3380 0x7fc60c11d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:37:52.359 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:52.339+0000 7fc60b59e700 10 monclient(hunting): picked mon.noname-d con 0x7fc60c0a3380 addr [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] 2023-09-30T01:37:52.359 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:52.339+0000 7fc60b59e700 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc60c1333b0 0x7fc60c11d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:37:52.359 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:52.339+0000 7fc60b59e700 10 monclient(hunting): picked mon.noname-b con 0x7fc60c1333b0 addr [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] 2023-09-30T01:37:52.359 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:52.339+0000 7fc60b59e700 10 monclient(hunting): start opening mon connection 2023-09-30T01:37:52.359 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:52.339+0000 7fc60b59e700 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc60c0a0ac0 con 0x7fc60c1333b0 2023-09-30T01:37:52.359 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:52.339+0000 7fc60b59e700 10 monclient(hunting): start opening mon connection 2023-09-30T01:37:52.359 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:52.339+0000 7fc60b59e700 1 -- --> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc60c097ce0 con 0x7fc60c09ffd0 2023-09-30T01:37:52.359 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:52.339+0000 7fc60b59e700 10 monclient(hunting): start opening mon connection 2023-09-30T01:37:52.359 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:52.339+0000 7fc60b59e700 1 -- --> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] -- mon_getmap magic: 0 v1 -- 0x7fc60c1201e0 con 0x7fc60c0a3380 2023-09-30T01:37:52.360 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:52.339+0000 7fc60b59e700 10 monclient(hunting): _renew_subs 2023-09-30T01:37:52.360 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:52.339+0000 7fc60b59e700 10 monclient(hunting): authenticate will time out at 1144.377459s 2023-09-30T01:37:52.360 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:52.339+0000 7fc609d9b700 1 --2- >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7fc60c0a3380 0x7fc60c11d4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:37:52.360 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:52.339+0000 7fc60a59c700 1 --2- >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7fc60c09ffd0 0x7fc60c11cf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:37:52.360 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:52.339+0000 7fc60a59c700 1 --2- >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7fc60c09ffd0 0x7fc60c11cf60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3301/0 says I am v2:172.21.15.88:46806/0 (socket says 172.21.15.88:46806) 2023-09-30T01:37:52.360 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:52.339+0000 7fc60a59c700 1 -- 172.21.15.88:0/442889189 learned_addr learned my addr 172.21.15.88:0/442889189 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-09-30T01:37:52.360 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:52.339+0000 7fc609d9b700 1 --2- >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7fc60c0a3380 0x7fc60c11d4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3303/0 says I am v2:172.21.15.88:50296/0 (socket says 172.21.15.88:50296) 2023-09-30T01:37:52.360 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:52.339+0000 7fc60a59c700 10 monclient(hunting): get_auth_request con 0x7fc60c09ffd0 auth_method 0 2023-09-30T01:37:52.360 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:52.339+0000 7fc60a59c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:37:52.360 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:52.339+0000 7fc60a59c700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:37:52.360 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:52.339+0000 7fc60a59c700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:37:52.360 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:52.339+0000 7fc609d9b700 10 monclient(hunting): get_auth_request con 0x7fc60c0a3380 auth_method 0 2023-09-30T01:37:52.361 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:52.339+0000 7fc609d9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:37:52.361 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:52.339+0000 7fc609d9b700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:37:52.361 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:52.339+0000 7fc609d9b700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:37:52.361 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:52.339+0000 7fc60ad9d700 1 --2- 172.21.15.88:0/442889189 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc60c1333b0 0x7fc60c11d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:37:52.361 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:52.339+0000 7fc609d9b700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:37:52.361 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:52.339+0000 7fc609d9b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:37:52.361 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:52.339+0000 7fc609d9b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:37:52.361 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:52.339+0000 7fc60a59c700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:37:52.361 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:52.339+0000 7fc60a59c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:37:52.361 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:52.339+0000 7fc60a59c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:37:52.361 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:52.339+0000 7fc60ad9d700 10 monclient(hunting): get_auth_request con 0x7fc60c1333b0 auth_method 0 2023-09-30T01:37:52.361 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:52.339+0000 7fc60ad9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:37:52.362 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:52.339+0000 7fc60ad9d700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:37:52.362 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:52.339+0000 7fc60ad9d700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:37:52.362 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:52.339+0000 7fc60ad9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:37:52.362 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:52.339+0000 7fc60ad9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:37:52.362 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:52.339+0000 7fc60ad9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:37:52.362 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:52.339+0000 7fc609d9b700 10 monclient(hunting): handle_auth_done global_id 4731 payload 824 2023-09-30T01:37:52.362 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:52.339+0000 7fc609d9b700 1 -- 172.21.15.88:0/442889189 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7fc60c09ffd0 msgr2=0x7fc60c11cf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:37:52.362 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:52.339+0000 7fc609d9b700 1 --2- 172.21.15.88:0/442889189 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7fc60c09ffd0 0x7fc60c11cf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:37:52.362 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:52.339+0000 7fc609d9b700 1 -- 172.21.15.88:0/442889189 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc60c1333b0 msgr2=0x7fc60c11d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:37:52.362 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:52.339+0000 7fc609d9b700 1 --2- 172.21.15.88:0/442889189 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc60c1333b0 0x7fc60c11d9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:37:52.362 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:52.339+0000 7fc609d9b700 10 monclient: _finish_hunting 0 2023-09-30T01:37:52.362 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:52.339+0000 7fc609d9b700 1 monclient: found mon.noname-d 2023-09-30T01:37:52.363 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:52.339+0000 7fc609d9b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:37:52.363 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:52.339+0000 7fc609d9b700 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.88:3303/0 2023-09-30T01:37:52.363 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:52.339+0000 7fc609d9b700 1 -- 172.21.15.88:0/442889189 --> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5fc0094f0 con 0x7fc60c0a3380 2023-09-30T01:37:52.363 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:52.339+0000 7fc609d9b700 1 --2- 172.21.15.88:0/442889189 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7fc60c0a3380 0x7fc60c11d4a0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fc5fc013000 tx=0x7fc5fc017000 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:37:52.363 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:52.339+0000 7fc5fa7fc700 1 -- 172.21.15.88:0/442889189 <== mon.6 v2:172.21.15.88:3303/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7fc5fc0118a0 con 0x7fc60c0a3380 2023-09-30T01:37:52.363 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:52.339+0000 7fc5fa7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:37:52.363 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:52.339+0000 7fc5fa7fc700 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2023-09-30T01:37:52.363 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:52.339+0000 7fc5fa7fc700 10 monclient: dump: 2023-09-30T01:37:52.363 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:37:52.363 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:37:52.363 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:37:52.363 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:37:52.363 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:37:52.364 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:37:52.364 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:37:52.364 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:37:52.364 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:37:52.364 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:37:52.364 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:37:52.364 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:37:52.364 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:37:52.364 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:37:52.364 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:37:52.364 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:37:52.364 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:52.339+0000 7fc5fa7fc700 10 monclient: _finish_auth 0 2023-09-30T01:37:52.364 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:52.339+0000 7fc5fa7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:37:52.365 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:52.339+0000 7fc5fa7fc700 1 -- 172.21.15.88:0/442889189 <== mon.6 v2:172.21.15.88:3303/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc5fc011a60 con 0x7fc60c0a3380 2023-09-30T01:37:52.365 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:52.339+0000 7fc5fa7fc700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:37:52.365 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:52.339+0000 7fc5fa7fc700 1 -- 172.21.15.88:0/442889189 <== mon.6 v2:172.21.15.88:3303/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7fc5fc011bd0 con 0x7fc60c0a3380 2023-09-30T01:37:52.365 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:52.339+0000 7fc5fa7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:37:52.365 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:52.339+0000 7fc5fa7fc700 10 monclient: got monmap 1 from mon.d (according to old e1) 2023-09-30T01:37:52.365 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:52.339+0000 7fc5fa7fc700 10 monclient: dump: 2023-09-30T01:37:52.365 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:37:52.365 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:37:52.365 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:37:52.365 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:37:52.365 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:37:52.365 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:37:52.366 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:37:52.366 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:37:52.366 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:37:52.366 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:37:52.366 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:37:52.366 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:37:52.366 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:37:52.366 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:37:52.366 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:37:52.366 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:37:52.366 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:52.339+0000 7fc60b59e700 5 monclient: authenticate success, global_id 4731 2023-09-30T01:37:52.366 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:52.339+0000 7fc60b59e700 10 monclient: _renew_subs 2023-09-30T01:37:52.367 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:52.339+0000 7fc608d99700 4 set_mon_vals no callback set 2023-09-30T01:37:52.367 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:52.339+0000 7fc60b59e700 10 monclient: _send_mon_message to mon.d at v2:172.21.15.88:3303/0 2023-09-30T01:37:52.367 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:52.339+0000 7fc60b59e700 1 -- 172.21.15.88:0/442889189 --> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc60c11ff00 con 0x7fc60c0a3380 2023-09-30T01:37:52.367 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:52.339+0000 7fc60b59e700 10 monclient: _renew_subs 2023-09-30T01:37:52.367 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:52.339+0000 7fc60b59e700 10 monclient: _send_mon_message to mon.d at v2:172.21.15.88:3303/0 2023-09-30T01:37:52.367 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:52.339+0000 7fc60b59e700 1 -- 172.21.15.88:0/442889189 --> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc60c11e360 con 0x7fc60c0a3380 2023-09-30T01:37:52.367 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:52.343+0000 7fc5fa7fc700 1 -- 172.21.15.88:0/442889189 <== mon.6 v2:172.21.15.88:3303/0 4 ==== mgrmap(e 5) v1 ==== 40144+0+0 (secure 0 0 0) 0x7fc5fc011d50 con 0x7fc60c0a3380 2023-09-30T01:37:52.367 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:52.343+0000 7fc60b59e700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:37:52.367 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:52.343+0000 7fc60b59e700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:37:52.367 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:52.343+0000 7fc60b59e700 10 monclient: _send_mon_message to mon.d at v2:172.21.15.88:3303/0 2023-09-30T01:37:52.367 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:52.343+0000 7fc60b59e700 1 -- 172.21.15.88:0/442889189 --> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc60c09bde0 con 0x7fc60c0a3380 2023-09-30T01:37:52.367 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:52.343+0000 7fc5fa7fc700 1 --2- 172.21.15.88:0/442889189 >> v2:172.21.15.107:6812/29926 conn(0x7fc5ec033290 0x7fc5ec035740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:37:52.368 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:52.343+0000 7fc5fa7fc700 1 -- 172.21.15.88:0/442889189 <== mon.6 v2:172.21.15.88:3303/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 3502+0+0 (secure 0 0 0) 0x7fc5fc044800 con 0x7fc60c0a3380 2023-09-30T01:37:52.368 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:52.343+0000 7fc60a59c700 1 --2- 172.21.15.88:0/442889189 >> v2:172.21.15.107:6812/29926 conn(0x7fc5ec033290 0x7fc5ec035740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:37:52.368 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:52.343+0000 7fc60a59c700 10 monclient: get_auth_request con 0x7fc5ec033290 auth_method 0 2023-09-30T01:37:52.368 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:52.343+0000 7fc60a59c700 1 --2- 172.21.15.88:0/442889189 >> v2:172.21.15.107:6812/29926 conn(0x7fc5ec033290 0x7fc5ec035740 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fc600006fd0 tx=0x7fc600008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:37:52.368 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:52.347+0000 7fc5fa7fc700 1 -- 172.21.15.88:0/442889189 <== mon.6 v2:172.21.15.88:3303/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fc5fc0216b0 con 0x7fc60c0a3380 2023-09-30T01:37:52.368 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:52.347+0000 7fc5fa7fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:37:52.368 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:52.347+0000 7fc5fa7fc700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:37:52.548 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:52.547+0000 7fc60b59e700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T01:37:52.549 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:52.547+0000 7fc60b59e700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:37:52.549 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:52.547+0000 7fc60b59e700 10 monclient: _send_mon_message to mon.d at v2:172.21.15.88:3303/0 2023-09-30T01:37:52.549 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:52.547+0000 7fc60b59e700 1 -- 172.21.15.88:0/442889189 --> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fc60c0710d0 con 0x7fc60c0a3380 2023-09-30T01:37:52.564 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:52.563+0000 7fc5fa7fc700 1 -- 172.21.15.88:0/442889189 <== mon.6 v2:172.21.15.88:3303/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7fc5fc028070 con 0x7fc60c0a3380 2023-09-30T01:37:52.565 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:52.563+0000 7fc5fa7fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:37:52.565 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:52.563+0000 7fc5fa7fc700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:37:52.565 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-09-30T01:37:52.565 INFO:teuthology.orchestra.run.smithi088.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:37:52.568 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:52.563+0000 7fc60b59e700 1 -- 172.21.15.88:0/442889189 >> v2:172.21.15.107:6812/29926 conn(0x7fc5ec033290 msgr2=0x7fc5ec035740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:37:52.568 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:52.563+0000 7fc60b59e700 1 --2- 172.21.15.88:0/442889189 >> v2:172.21.15.107:6812/29926 conn(0x7fc5ec033290 0x7fc5ec035740 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fc600006fd0 tx=0x7fc600008040 comp rx=0 tx=0).stop 2023-09-30T01:37:52.568 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:52.563+0000 7fc60b59e700 10 monclient: shutdown 2023-09-30T01:37:52.568 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:52.563+0000 7fc60b59e700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:37:52.568 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:52.563+0000 7fc60b59e700 1 -- 172.21.15.88:0/442889189 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7fc60c0a3380 msgr2=0x7fc60c11d4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:37:52.568 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:52.563+0000 7fc60b59e700 1 --2- 172.21.15.88:0/442889189 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7fc60c0a3380 0x7fc60c11d4a0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fc5fc013000 tx=0x7fc5fc017000 comp rx=0 tx=0).stop 2023-09-30T01:37:52.568 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:52.563+0000 7fc60b59e700 1 -- 172.21.15.88:0/442889189 shutdown_connections 2023-09-30T01:37:52.569 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:52.563+0000 7fc60b59e700 1 --2- 172.21.15.88:0/442889189 >> v2:172.21.15.107:6812/29926 conn(0x7fc5ec033290 0x7fc5ec035740 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:37:52.569 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:52.563+0000 7fc60b59e700 1 --2- 172.21.15.88:0/442889189 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc60c1333b0 0x7fc60c11d9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:37:52.569 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:52.563+0000 7fc60b59e700 1 --2- 172.21.15.88:0/442889189 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7fc60c0a3380 0x7fc60c11d4a0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:37:52.569 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:52.563+0000 7fc60b59e700 1 --2- 172.21.15.88:0/442889189 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7fc60c09ffd0 0x7fc60c11cf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:37:52.569 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:52.563+0000 7fc60b59e700 1 -- 172.21.15.88:0/442889189 >> 172.21.15.88:0/442889189 conn(0x7fc60c06e5e0 msgr2=0x7fc60c098f60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:37:52.569 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:52.563+0000 7fc60b59e700 1 -- 172.21.15.88:0/442889189 shutdown_connections 2023-09-30T01:37:52.569 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:52.563+0000 7fc60b59e700 1 -- 172.21.15.88:0/442889189 wait complete. 2023-09-30T01:37:55.582 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:37:55.684 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:55.683+0000 7f6236061700 10 monclient: get_monmap_and_config 2023-09-30T01:37:55.684 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:55.683+0000 7f6236061700 10 monclient: build_initial_monmap 2023-09-30T01:37:55.684 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:55.683+0000 7f6236061700 10 monclient: monmap: 2023-09-30T01:37:55.684 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2023-09-30T01:37:55.684 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:37:55.684 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:37:55.684085+0000 2023-09-30T01:37:55.684 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:37:55.684085+0000 2023-09-30T01:37:55.684 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2023-09-30T01:37:55.685 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:37:55.685 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a-legacy 2023-09-30T01:37:55.685 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.107:6789/0 mon.noname-f-legacy 2023-09-30T01:37:55.685 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.noname-b 2023-09-30T01:37:55.685 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.noname-g 2023-09-30T01:37:55.685 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.noname-c 2023-09-30T01:37:55.685 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.noname-h 2023-09-30T01:37:55.685 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.noname-d 2023-09-30T01:37:55.685 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.noname-i 2023-09-30T01:37:55.685 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.noname-e 2023-09-30T01:37:55.685 INFO:teuthology.orchestra.run.smithi088.stderr:9: v2:172.21.15.88:3300/0 mon.noname-a 2023-09-30T01:37:55.685 INFO:teuthology.orchestra.run.smithi088.stderr:10: v2:172.21.15.107:3300/0 mon.noname-f 2023-09-30T01:37:55.685 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:37:55.689 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:55.683+0000 7f6236061700 1 Processor -- start 2023-09-30T01:37:55.689 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:55.683+0000 7f6236061700 1 -- start start 2023-09-30T01:37:55.689 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:55.683+0000 7f6236061700 10 monclient: init 2023-09-30T01:37:55.689 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:55.683+0000 7f6236061700 10 monclient: _reopen_session rank -1 2023-09-30T01:37:55.689 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:55.683+0000 7f6236061700 10 monclient: _add_conns ranks=[6,5,0,3,9,2,1,10,7,8,4] 2023-09-30T01:37:55.689 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:55.683+0000 7f6236061700 1 --2- >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7f623009d5f0 0x7f623009d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:37:55.689 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:55.683+0000 7f6236061700 10 monclient(hunting): picked mon.noname-d con 0x7f623009d5f0 addr [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] 2023-09-30T01:37:55.689 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:55.683+0000 7f6236061700 1 --2- >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7f623006e670 0x7f623009df00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:37:55.689 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:55.683+0000 7f6236061700 10 monclient(hunting): picked mon.noname-h con 0x7f623006e670 addr [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] 2023-09-30T01:37:55.690 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:55.683+0000 7f6236061700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f6230131440 addr v1:172.21.15.88:6789/0 2023-09-30T01:37:55.690 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:55.683+0000 7f6236061700 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f623005ecb0 con 0x7f6230131440 2023-09-30T01:37:55.690 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:55.683+0000 7f6236061700 10 monclient(hunting): start opening mon connection 2023-09-30T01:37:55.690 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:55.683+0000 7f6236061700 1 -- --> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] -- mon_getmap magic: 0 v1 -- 0x7f6230097a40 con 0x7f623006e670 2023-09-30T01:37:55.690 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:55.683+0000 7f6236061700 10 monclient(hunting): start opening mon connection 2023-09-30T01:37:55.690 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:55.683+0000 7f6236061700 1 -- --> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] -- mon_getmap magic: 0 v1 -- 0x7f623006da10 con 0x7f623009d5f0 2023-09-30T01:37:55.690 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:55.683+0000 7f6236061700 10 monclient(hunting): _renew_subs 2023-09-30T01:37:55.690 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:55.683+0000 7f6236061700 10 monclient(hunting): authenticate will time out at 1147.723311s 2023-09-30T01:37:55.690 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:55.683+0000 7f622f7fe700 1 --2- >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7f623009d5f0 0x7f623009d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:37:55.691 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:55.683+0000 7f622ffff700 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f6230131440 0x7f6230131850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:50368/0 (socket says 172.21.15.88:50368) 2023-09-30T01:37:55.691 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:55.683+0000 7f622ffff700 1 -- 172.21.15.88:0/2519415386 learned_addr learned my addr 172.21.15.88:0/2519415386 (peer_addr_for_me v1:172.21.15.88:0/0) 2023-09-30T01:37:55.691 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:55.683+0000 7f622f7fe700 10 monclient(hunting): get_auth_request con 0x7f623009d5f0 auth_method 0 2023-09-30T01:37:55.691 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:55.683+0000 7f622f7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:37:55.691 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:55.683+0000 7f622f7fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:37:55.691 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:55.683+0000 7f622f7fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:37:55.691 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:55.683+0000 7f622effd700 1 --2- 172.21.15.88:0/2519415386 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7f623006e670 0x7f623009df00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:37:55.691 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:55.683+0000 7f622f7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:37:55.691 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:55.683+0000 7f622f7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:37:55.691 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:55.683+0000 7f622f7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:37:55.691 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:55.683+0000 7f622effd700 10 monclient(hunting): get_auth_request con 0x7f623006e670 auth_method 0 2023-09-30T01:37:55.692 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:55.683+0000 7f622effd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:37:55.692 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:55.683+0000 7f622effd700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:37:55.692 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:55.683+0000 7f622effd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:37:55.692 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:55.683+0000 7f622effd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:37:55.692 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:55.683+0000 7f622effd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:37:55.692 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:55.683+0000 7f622effd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:37:55.692 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:55.687+0000 7f622f7fe700 10 monclient(hunting): handle_auth_done global_id 4740 payload 274 2023-09-30T01:37:55.692 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:55.687+0000 7f622f7fe700 1 -- 172.21.15.88:0/2519415386 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7f623006e670 msgr2=0x7f623009df00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:37:55.692 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:55.687+0000 7f622f7fe700 1 --2- 172.21.15.88:0/2519415386 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7f623006e670 0x7f623009df00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:37:55.692 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:55.687+0000 7f622f7fe700 1 -- 172.21.15.88:0/2519415386 >> v1:172.21.15.88:6789/0 conn(0x7f6230131440 legacy=0x7f6230131850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:37:55.692 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:55.687+0000 7f622f7fe700 10 monclient: _finish_hunting 0 2023-09-30T01:37:55.693 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:55.687+0000 7f622f7fe700 1 monclient: found mon.noname-d 2023-09-30T01:37:55.693 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:55.687+0000 7f622f7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:37:55.693 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:55.687+0000 7f622f7fe700 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.88:3303/0 2023-09-30T01:37:55.693 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:55.687+0000 7f622f7fe700 1 -- 172.21.15.88:0/2519415386 --> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f623006d450 con 0x7f623009d5f0 2023-09-30T01:37:55.693 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:55.687+0000 7f622f7fe700 1 --2- 172.21.15.88:0/2519415386 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7f623009d5f0 0x7f623009d9c0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f6220009910 tx=0x7f62200091a0 comp rx=0 tx=0).ready entity=mon.6 client_cookie=a5e7577ed6a7bd6e server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:37:55.693 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:55.687+0000 7f622e7fc700 1 -- 172.21.15.88:0/2519415386 <== mon.6 v2:172.21.15.88:3303/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f622000bb00 con 0x7f623009d5f0 2023-09-30T01:37:55.693 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:55.687+0000 7f622e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:37:55.693 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:55.687+0000 7f622e7fc700 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2023-09-30T01:37:55.693 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:55.687+0000 7f622e7fc700 10 monclient: dump: 2023-09-30T01:37:55.693 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:37:55.693 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:37:55.693 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:37:55.693 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:37:55.694 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:37:55.694 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:37:55.694 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:37:55.694 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:37:55.694 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:37:55.694 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:37:55.694 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:37:55.694 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:37:55.694 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:37:55.694 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:37:55.694 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:37:55.694 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:37:55.694 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:55.687+0000 7f622e7fc700 10 monclient: _finish_auth 0 2023-09-30T01:37:55.695 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:55.687+0000 7f622e7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:37:55.695 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:55.687+0000 7f6236061700 5 monclient: authenticate success, global_id 4740 2023-09-30T01:37:55.695 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:55.687+0000 7f6236061700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:37:55.695 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:55.687+0000 7f622e7fc700 1 -- 172.21.15.88:0/2519415386 <== mon.6 v2:172.21.15.88:3303/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f622000bc80 con 0x7f623009d5f0 2023-09-30T01:37:55.695 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:55.687+0000 7f622e7fc700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:37:55.695 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:55.687+0000 7f6236061700 10 monclient: get_monmap_and_config success 2023-09-30T01:37:55.695 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:55.687+0000 7f622e7fc700 1 -- 172.21.15.88:0/2519415386 <== mon.6 v2:172.21.15.88:3303/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f622000bdf0 con 0x7f623009d5f0 2023-09-30T01:37:55.695 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:55.687+0000 7f622e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:37:55.695 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:55.687+0000 7f6236061700 4 set_mon_vals no callback set 2023-09-30T01:37:55.695 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:55.687+0000 7f622e7fc700 10 monclient: got monmap 1 from mon.d (according to old e1) 2023-09-30T01:37:55.695 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:55.687+0000 7f622e7fc700 10 monclient: dump: 2023-09-30T01:37:55.695 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:37:55.696 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:37:55.696 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:37:55.696 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:37:55.696 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:37:55.696 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:37:55.696 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:37:55.696 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:37:55.696 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:37:55.696 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:37:55.696 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:37:55.696 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:37:55.696 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:37:55.696 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:37:55.697 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:37:55.697 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:37:55.701 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:55.687+0000 7f6236061700 10 monclient: shutdown 2023-09-30T01:37:55.701 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:55.687+0000 7f6236061700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:37:55.701 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:55.687+0000 7f6236061700 1 -- 172.21.15.88:0/2519415386 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7f623009d5f0 msgr2=0x7f623009d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:37:55.701 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:55.687+0000 7f6236061700 1 --2- 172.21.15.88:0/2519415386 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7f623009d5f0 0x7f623009d9c0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f6220009910 tx=0x7f62200091a0 comp rx=0 tx=0).stop 2023-09-30T01:37:55.701 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:55.687+0000 7f6236061700 1 -- 172.21.15.88:0/2519415386 shutdown_connections 2023-09-30T01:37:55.701 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:55.687+0000 7f6236061700 1 --2- 172.21.15.88:0/2519415386 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7f623006e670 0x7f623009df00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:37:55.701 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:55.687+0000 7f6236061700 1 --2- 172.21.15.88:0/2519415386 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7f623009d5f0 0x7f623009d9c0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:37:55.701 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:55.687+0000 7f6236061700 1 -- 172.21.15.88:0/2519415386 >> 172.21.15.88:0/2519415386 conn(0x7f62300a01d0 msgr2=0x7f6230062c70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:37:55.701 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:55.687+0000 7f6236061700 1 -- 172.21.15.88:0/2519415386 shutdown_connections 2023-09-30T01:37:55.702 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:55.687+0000 7f6236061700 1 -- 172.21.15.88:0/2519415386 wait complete. 2023-09-30T01:37:55.702 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:55.687+0000 7f6236061700 10 monclient: build_initial_monmap 2023-09-30T01:37:55.702 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:55.687+0000 7f6236061700 10 monclient: monmap: 2023-09-30T01:37:55.702 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2023-09-30T01:37:55.702 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:37:55.702 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 0.000000 2023-09-30T01:37:55.702 INFO:teuthology.orchestra.run.smithi088.stderr:created 0.000000 2023-09-30T01:37:55.702 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2023-09-30T01:37:55.702 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 1 2023-09-30T01:37:55.702 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.noname-a 2023-09-30T01:37:55.702 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.noname-f 2023-09-30T01:37:55.702 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.noname-b 2023-09-30T01:37:55.703 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.noname-g 2023-09-30T01:37:55.703 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.noname-c 2023-09-30T01:37:55.703 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.noname-h 2023-09-30T01:37:55.703 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.noname-d 2023-09-30T01:37:55.703 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.noname-i 2023-09-30T01:37:55.703 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.noname-e 2023-09-30T01:37:55.703 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:37:55.703 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:55.687+0000 7f6236061700 1 Processor -- start 2023-09-30T01:37:55.703 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:55.687+0000 7f6236061700 1 -- start start 2023-09-30T01:37:55.703 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:55.687+0000 7f6236061700 10 monclient: init 2023-09-30T01:37:55.703 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:55.687+0000 7f6236061700 10 monclient: _reopen_session rank -1 2023-09-30T01:37:55.703 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:55.687+0000 7f6236061700 10 monclient: _add_conns ranks=[8,6,0,5,4,1,3,2,7] 2023-09-30T01:37:55.704 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:55.687+0000 7f6236061700 1 --2- >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f623006e670 0x7f623011c8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:37:55.704 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:55.687+0000 7f6236061700 10 monclient(hunting): picked mon.noname-e con 0x7f623006e670 addr [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] 2023-09-30T01:37:55.704 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:55.687+0000 7f6236061700 1 --2- >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7f623009d5f0 0x7f623011ce00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:37:55.704 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:55.687+0000 7f6236061700 10 monclient(hunting): picked mon.noname-d con 0x7f623009d5f0 addr [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] 2023-09-30T01:37:55.704 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:55.687+0000 7f6236061700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6230131440 0x7f623011d340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:37:55.704 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:55.687+0000 7f6236061700 10 monclient(hunting): picked mon.noname-a con 0x7f6230131440 addr [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] 2023-09-30T01:37:55.704 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:55.687+0000 7f6236061700 10 monclient(hunting): start opening mon connection 2023-09-30T01:37:55.704 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:55.687+0000 7f6236061700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f623012ffd0 con 0x7f6230131440 2023-09-30T01:37:55.704 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:55.687+0000 7f6236061700 10 monclient(hunting): start opening mon connection 2023-09-30T01:37:55.704 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:55.687+0000 7f6236061700 1 -- --> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] -- mon_getmap magic: 0 v1 -- 0x7f623006c920 con 0x7f623009d5f0 2023-09-30T01:37:55.704 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:55.687+0000 7f6236061700 10 monclient(hunting): start opening mon connection 2023-09-30T01:37:55.704 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:55.687+0000 7f6236061700 1 -- --> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] -- mon_getmap magic: 0 v1 -- 0x7f623009cef0 con 0x7f623006e670 2023-09-30T01:37:55.704 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:55.687+0000 7f6236061700 10 monclient(hunting): _renew_subs 2023-09-30T01:37:55.705 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:55.691+0000 7f6236061700 10 monclient(hunting): authenticate will time out at 1147.728013s 2023-09-30T01:37:55.705 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:55.691+0000 7f622f7fe700 1 --2- >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f623006e670 0x7f623011c8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:37:55.705 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:55.691+0000 7f622ffff700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6230131440 0x7f623011d340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:37:55.705 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:55.691+0000 7f622f7fe700 1 --2- >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f623006e670 0x7f623011c8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3304/0 says I am v2:172.21.15.88:59762/0 (socket says 172.21.15.88:59762) 2023-09-30T01:37:55.705 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:55.691+0000 7f622effd700 1 --2- >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7f623009d5f0 0x7f623011ce00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:37:55.705 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:55.691+0000 7f622f7fe700 1 -- 172.21.15.88:0/4183345548 learned_addr learned my addr 172.21.15.88:0/4183345548 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-09-30T01:37:55.705 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:55.691+0000 7f622f7fe700 10 monclient(hunting): get_auth_request con 0x7f623006e670 auth_method 0 2023-09-30T01:37:55.705 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:55.691+0000 7f622f7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:37:55.705 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:55.691+0000 7f622f7fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:37:55.705 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:55.691+0000 7f622f7fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:37:55.705 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:55.691+0000 7f622ffff700 10 monclient(hunting): get_auth_request con 0x7f6230131440 auth_method 0 2023-09-30T01:37:55.705 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:55.691+0000 7f622ffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:37:55.706 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:55.691+0000 7f622ffff700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:37:55.706 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:55.691+0000 7f622ffff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:37:55.706 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:55.691+0000 7f622effd700 10 monclient(hunting): get_auth_request con 0x7f623009d5f0 auth_method 0 2023-09-30T01:37:55.706 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:55.691+0000 7f622effd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:37:55.706 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:55.691+0000 7f622effd700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:37:55.706 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:55.691+0000 7f622effd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:37:55.706 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:55.691+0000 7f622f7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:37:55.706 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:55.691+0000 7f622f7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:37:55.706 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:55.691+0000 7f622f7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:37:55.706 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:55.691+0000 7f622ffff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:37:55.706 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:55.691+0000 7f622ffff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:37:55.706 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:55.691+0000 7f622ffff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:37:55.706 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:55.691+0000 7f622effd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:37:55.707 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:55.691+0000 7f622effd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:37:55.707 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:55.691+0000 7f622effd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:37:55.707 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:55.691+0000 7f622f7fe700 10 monclient(hunting): handle_auth_done global_id 4877 payload 824 2023-09-30T01:37:55.707 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:55.691+0000 7f622f7fe700 1 -- 172.21.15.88:0/4183345548 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7f623009d5f0 msgr2=0x7f623011ce00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:37:55.707 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:55.691+0000 7f622f7fe700 1 --2- 172.21.15.88:0/4183345548 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7f623009d5f0 0x7f623011ce00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:37:55.707 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:55.691+0000 7f622f7fe700 1 -- 172.21.15.88:0/4183345548 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6230131440 msgr2=0x7f623011d340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:37:55.707 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:55.691+0000 7f622f7fe700 1 --2- 172.21.15.88:0/4183345548 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6230131440 0x7f623011d340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:37:55.707 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:55.691+0000 7f622f7fe700 10 monclient: _finish_hunting 0 2023-09-30T01:37:55.707 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:55.691+0000 7f622f7fe700 1 monclient: found mon.noname-e 2023-09-30T01:37:55.707 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:55.691+0000 7f622f7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:37:55.707 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:55.691+0000 7f622f7fe700 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.88:3304/0 2023-09-30T01:37:55.707 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:55.691+0000 7f622f7fe700 1 -- 172.21.15.88:0/4183345548 --> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6220009420 con 0x7f623006e670 2023-09-30T01:37:55.708 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:55.691+0000 7f622f7fe700 1 --2- 172.21.15.88:0/4183345548 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f623006e670 0x7f623011c8c0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f6220013000 tx=0x7f6220017000 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:37:55.708 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:55.691+0000 7f62177fe700 1 -- 172.21.15.88:0/4183345548 <== mon.8 v2:172.21.15.88:3304/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f6220006ad0 con 0x7f623006e670 2023-09-30T01:37:55.708 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:55.691+0000 7f62177fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:37:55.708 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:55.691+0000 7f62177fe700 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2023-09-30T01:37:55.708 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:55.691+0000 7f62177fe700 10 monclient: dump: 2023-09-30T01:37:55.708 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:37:55.708 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:37:55.708 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:37:55.708 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:37:55.708 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:37:55.708 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:37:55.708 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:37:55.709 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:37:55.709 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:37:55.709 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:37:55.709 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:37:55.709 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:37:55.709 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:37:55.709 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:37:55.709 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:37:55.709 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:37:55.709 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:55.691+0000 7f62177fe700 10 monclient: _finish_auth 0 2023-09-30T01:37:55.709 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:55.691+0000 7f62177fe700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:37:55.709 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:55.691+0000 7f6236061700 5 monclient: authenticate success, global_id 4877 2023-09-30T01:37:55.709 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:55.691+0000 7f6236061700 10 monclient: _renew_subs 2023-09-30T01:37:55.710 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:55.691+0000 7f62177fe700 1 -- 172.21.15.88:0/4183345548 <== mon.8 v2:172.21.15.88:3304/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f622001a430 con 0x7f623006e670 2023-09-30T01:37:55.710 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:55.691+0000 7f6236061700 10 monclient: _send_mon_message to mon.e at v2:172.21.15.88:3304/0 2023-09-30T01:37:55.710 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:55.691+0000 7f6236061700 1 -- 172.21.15.88:0/4183345548 --> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f623011ff40 con 0x7f623006e670 2023-09-30T01:37:55.710 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:55.691+0000 7f62177fe700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:37:55.710 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:55.691+0000 7f62177fe700 1 -- 172.21.15.88:0/4183345548 <== mon.8 v2:172.21.15.88:3304/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f622001aa20 con 0x7f623006e670 2023-09-30T01:37:55.710 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:55.691+0000 7f62177fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:37:55.710 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:55.691+0000 7f62177fe700 10 monclient: got monmap 1 from mon.e (according to old e1) 2023-09-30T01:37:55.710 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:55.691+0000 7f622dffb700 4 set_mon_vals no callback set 2023-09-30T01:37:55.710 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:55.691+0000 7f62177fe700 10 monclient: dump: 2023-09-30T01:37:55.710 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:37:55.710 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:37:55.710 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:37:55.710 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:37:55.711 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:37:55.711 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:37:55.711 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:37:55.711 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:37:55.711 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:37:55.711 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:37:55.711 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:37:55.711 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:37:55.711 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:37:55.711 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:37:55.711 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:37:55.711 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:37:55.712 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:55.691+0000 7f6236061700 10 monclient: _renew_subs 2023-09-30T01:37:55.712 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:55.691+0000 7f6236061700 10 monclient: _send_mon_message to mon.e at v2:172.21.15.88:3304/0 2023-09-30T01:37:55.712 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:55.691+0000 7f6236061700 1 -- 172.21.15.88:0/4183345548 --> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6230064b80 con 0x7f623006e670 2023-09-30T01:37:55.712 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:55.691+0000 7f6236061700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:37:55.712 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:55.691+0000 7f6236061700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:37:55.712 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:55.691+0000 7f6236061700 10 monclient: _send_mon_message to mon.e at v2:172.21.15.88:3304/0 2023-09-30T01:37:55.712 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:55.691+0000 7f6236061700 1 -- 172.21.15.88:0/4183345548 --> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6230071960 con 0x7f623006e670 2023-09-30T01:37:55.712 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:55.691+0000 7f62177fe700 1 -- 172.21.15.88:0/4183345548 <== mon.8 v2:172.21.15.88:3304/0 4 ==== mgrmap(e 5) v1 ==== 40144+0+0 (secure 0 0 0) 0x7f622001aba0 con 0x7f623006e670 2023-09-30T01:37:55.712 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:55.691+0000 7f62177fe700 1 --2- 172.21.15.88:0/4183345548 >> v2:172.21.15.107:6812/29926 conn(0x7f621c033360 0x7f621c035810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:37:55.712 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:55.691+0000 7f62177fe700 1 -- 172.21.15.88:0/4183345548 <== mon.8 v2:172.21.15.88:3304/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 3502+0+0 (secure 0 0 0) 0x7f6220045010 con 0x7f623006e670 2023-09-30T01:37:55.712 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:55.695+0000 7f622effd700 1 --2- 172.21.15.88:0/4183345548 >> v2:172.21.15.107:6812/29926 conn(0x7f621c033360 0x7f621c035810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:37:55.712 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:55.695+0000 7f622effd700 10 monclient: get_auth_request con 0x7f621c033360 auth_method 0 2023-09-30T01:37:55.712 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:55.695+0000 7f622effd700 1 --2- 172.21.15.88:0/4183345548 >> v2:172.21.15.107:6812/29926 conn(0x7f621c033360 0x7f621c035810 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f6224009db0 tx=0x7f62240091a0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:37:55.713 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:55.699+0000 7f62177fe700 1 -- 172.21.15.88:0/4183345548 <== mon.8 v2:172.21.15.88:3304/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f6220019d80 con 0x7f623006e670 2023-09-30T01:37:55.713 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:55.699+0000 7f62177fe700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:37:55.713 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:55.699+0000 7f62177fe700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:37:55.897 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:55.895+0000 7f6236061700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T01:37:55.897 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:55.895+0000 7f6236061700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:37:55.898 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:55.895+0000 7f6236061700 10 monclient: _send_mon_message to mon.e at v2:172.21.15.88:3304/0 2023-09-30T01:37:55.898 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:55.895+0000 7f6236061700 1 -- 172.21.15.88:0/4183345548 --> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f6230120350 con 0x7f623006e670 2023-09-30T01:37:55.898 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:55.895+0000 7f62177fe700 1 -- 172.21.15.88:0/4183345548 <== mon.8 v2:172.21.15.88:3304/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f6220015020 con 0x7f623006e670 2023-09-30T01:37:55.899 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:55.895+0000 7f62177fe700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:37:55.899 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:55.895+0000 7f62177fe700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:37:55.899 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-09-30T01:37:55.900 INFO:teuthology.orchestra.run.smithi088.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:37:55.901 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:55.899+0000 7f6236061700 1 -- 172.21.15.88:0/4183345548 >> v2:172.21.15.107:6812/29926 conn(0x7f621c033360 msgr2=0x7f621c035810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:37:55.901 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:55.899+0000 7f6236061700 1 --2- 172.21.15.88:0/4183345548 >> v2:172.21.15.107:6812/29926 conn(0x7f621c033360 0x7f621c035810 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f6224009db0 tx=0x7f62240091a0 comp rx=0 tx=0).stop 2023-09-30T01:37:55.901 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:55.899+0000 7f6236061700 10 monclient: shutdown 2023-09-30T01:37:55.901 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:55.899+0000 7f6236061700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:37:55.901 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:55.899+0000 7f6236061700 1 -- 172.21.15.88:0/4183345548 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f623006e670 msgr2=0x7f623011c8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:37:55.901 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:55.899+0000 7f6236061700 1 --2- 172.21.15.88:0/4183345548 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f623006e670 0x7f623011c8c0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f6220013000 tx=0x7f6220017000 comp rx=0 tx=0).stop 2023-09-30T01:37:55.902 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:55.899+0000 7f6236061700 1 -- 172.21.15.88:0/4183345548 shutdown_connections 2023-09-30T01:37:55.902 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:55.899+0000 7f6236061700 1 --2- 172.21.15.88:0/4183345548 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6230131440 0x7f623011d340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:37:55.902 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:55.899+0000 7f6236061700 1 --2- 172.21.15.88:0/4183345548 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7f623009d5f0 0x7f623011ce00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:37:55.902 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:55.899+0000 7f6236061700 1 --2- 172.21.15.88:0/4183345548 >> v2:172.21.15.107:6812/29926 conn(0x7f621c033360 0x7f621c035810 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:37:55.902 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:55.899+0000 7f6236061700 1 --2- 172.21.15.88:0/4183345548 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f623006e670 0x7f623011c8c0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:37:55.902 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:55.899+0000 7f6236061700 1 -- 172.21.15.88:0/4183345548 >> 172.21.15.88:0/4183345548 conn(0x7f62300a01d0 msgr2=0x7f62300a21d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:37:55.902 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:55.899+0000 7f6236061700 1 -- 172.21.15.88:0/4183345548 shutdown_connections 2023-09-30T01:37:55.902 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:55.899+0000 7f6236061700 1 -- 172.21.15.88:0/4183345548 wait complete. 2023-09-30T01:37:58.912 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:37:59.024 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:59.023+0000 7fc5163a7700 10 monclient: get_monmap_and_config 2023-09-30T01:37:59.024 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:59.023+0000 7fc5163a7700 10 monclient: build_initial_monmap 2023-09-30T01:37:59.024 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:59.023+0000 7fc5163a7700 10 monclient: monmap: 2023-09-30T01:37:59.024 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2023-09-30T01:37:59.024 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:37:59.024 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:37:59.024081+0000 2023-09-30T01:37:59.025 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:37:59.024081+0000 2023-09-30T01:37:59.025 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2023-09-30T01:37:59.025 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:37:59.025 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a-legacy 2023-09-30T01:37:59.025 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.107:6789/0 mon.noname-f-legacy 2023-09-30T01:37:59.025 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.noname-b 2023-09-30T01:37:59.025 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.noname-g 2023-09-30T01:37:59.025 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.noname-c 2023-09-30T01:37:59.025 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.noname-h 2023-09-30T01:37:59.025 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.noname-d 2023-09-30T01:37:59.025 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.noname-i 2023-09-30T01:37:59.025 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.noname-e 2023-09-30T01:37:59.025 INFO:teuthology.orchestra.run.smithi088.stderr:9: v2:172.21.15.88:3300/0 mon.noname-a 2023-09-30T01:37:59.026 INFO:teuthology.orchestra.run.smithi088.stderr:10: v2:172.21.15.107:3300/0 mon.noname-f 2023-09-30T01:37:59.026 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:37:59.033 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:59.023+0000 7fc5163a7700 1 Processor -- start 2023-09-30T01:37:59.033 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:59.023+0000 7fc5163a7700 1 -- start start 2023-09-30T01:37:59.033 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:59.023+0000 7fc5163a7700 10 monclient: init 2023-09-30T01:37:59.033 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:59.023+0000 7fc5163a7700 10 monclient: _reopen_session rank -1 2023-09-30T01:37:59.033 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:59.023+0000 7fc5163a7700 10 monclient: _add_conns ranks=[0,3,4,6,2,9,8,1,7,5,10] 2023-09-30T01:37:59.033 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:59.023+0000 7fc5163a7700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fc5100a3380 addr v1:172.21.15.88:6789/0 2023-09-30T01:37:59.034 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:59.023+0000 7fc5163a7700 1 --2- >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7fc51009ffd0 0x7fc5100a03a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:37:59.034 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:59.023+0000 7fc5163a7700 10 monclient(hunting): picked mon.noname-g con 0x7fc51009ffd0 addr [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] 2023-09-30T01:37:59.034 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:59.023+0000 7fc5163a7700 1 --2- >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7fc5101386e0 0x7fc51013aac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:37:59.034 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:59.023+0000 7fc5163a7700 10 monclient(hunting): picked mon.noname-c con 0x7fc5101386e0 addr [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] 2023-09-30T01:37:59.034 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:59.023+0000 7fc5163a7700 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc51004bb20 con 0x7fc5100a3380 2023-09-30T01:37:59.034 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:59.023+0000 7fc5163a7700 10 monclient(hunting): start opening mon connection 2023-09-30T01:37:59.034 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:59.023+0000 7fc5163a7700 1 -- --> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc510097ab0 con 0x7fc51009ffd0 2023-09-30T01:37:59.034 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:59.023+0000 7fc5163a7700 10 monclient(hunting): start opening mon connection 2023-09-30T01:37:59.034 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:59.023+0000 7fc5163a7700 1 -- --> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] -- mon_getmap magic: 0 v1 -- 0x7fc5100a08e0 con 0x7fc5101386e0 2023-09-30T01:37:59.034 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:59.023+0000 7fc5163a7700 10 monclient(hunting): _renew_subs 2023-09-30T01:37:59.034 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:59.023+0000 7fc5163a7700 10 monclient(hunting): authenticate will time out at 1151.061549s 2023-09-30T01:37:59.034 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:59.023+0000 7fc515ba6700 1 --2- >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7fc5101386e0 0x7fc51013aac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:37:59.035 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:59.023+0000 7fc515ba6700 1 --2- >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7fc5101386e0 0x7fc51013aac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3302/0 says I am v2:172.21.15.88:38834/0 (socket says 172.21.15.88:38834) 2023-09-30T01:37:59.035 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:59.023+0000 7fc515ba6700 1 -- 172.21.15.88:0/18075902 learned_addr learned my addr 172.21.15.88:0/18075902 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-09-30T01:37:59.035 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:59.023+0000 7fc515ba6700 10 monclient(hunting): get_auth_request con 0x7fc5101386e0 auth_method 0 2023-09-30T01:37:59.035 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:59.023+0000 7fc515ba6700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:37:59.035 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:59.023+0000 7fc515ba6700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:37:59.035 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:59.023+0000 7fc515ba6700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:37:59.035 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:59.023+0000 7fc515ba6700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:37:59.035 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:59.023+0000 7fc515ba6700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:37:59.035 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:59.023+0000 7fc515ba6700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:37:59.035 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:59.023+0000 7fc514ba4700 1 --2- 172.21.15.88:0/18075902 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7fc51009ffd0 0x7fc5100a03a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:37:59.035 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:59.023+0000 7fc514ba4700 10 monclient(hunting): get_auth_request con 0x7fc51009ffd0 auth_method 0 2023-09-30T01:37:59.035 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:59.023+0000 7fc514ba4700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:37:59.036 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:59.023+0000 7fc514ba4700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:37:59.036 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:59.023+0000 7fc514ba4700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:37:59.036 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:59.023+0000 7fc515ba6700 10 monclient(hunting): handle_auth_done global_id 4864 payload 274 2023-09-30T01:37:59.036 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:59.023+0000 7fc515ba6700 1 -- 172.21.15.88:0/18075902 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7fc51009ffd0 msgr2=0x7fc5100a03a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:37:59.036 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:59.023+0000 7fc515ba6700 1 --2- 172.21.15.88:0/18075902 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7fc51009ffd0 0x7fc5100a03a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:37:59.036 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:59.023+0000 7fc515ba6700 1 -- 172.21.15.88:0/18075902 >> v1:172.21.15.88:6789/0 conn(0x7fc5100a3380 legacy=0x7fc5100a48b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:37:59.036 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:59.023+0000 7fc515ba6700 10 monclient: _finish_hunting 0 2023-09-30T01:37:59.036 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:59.023+0000 7fc515ba6700 1 monclient: found mon.noname-c 2023-09-30T01:37:59.036 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:59.023+0000 7fc515ba6700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:37:59.036 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:59.023+0000 7fc515ba6700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.88:3302/0 2023-09-30T01:37:59.036 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:59.023+0000 7fc515ba6700 1 -- 172.21.15.88:0/18075902 --> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5100984b0 con 0x7fc5101386e0 2023-09-30T01:37:59.036 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:59.023+0000 7fc515ba6700 1 --2- 172.21.15.88:0/18075902 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7fc5101386e0 0x7fc51013aac0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fc5080099e0 tx=0x7fc508009250 comp rx=0 tx=0).ready entity=mon.4 client_cookie=b8a2d3cab174e443 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:37:59.036 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:59.023+0000 7fc50ffff700 1 -- 172.21.15.88:0/18075902 <== mon.4 v2:172.21.15.88:3302/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7fc50800bce0 con 0x7fc5101386e0 2023-09-30T01:37:59.037 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:59.023+0000 7fc50ffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:37:59.037 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:59.023+0000 7fc50ffff700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T01:37:59.037 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:59.023+0000 7fc50ffff700 10 monclient: dump: 2023-09-30T01:37:59.037 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:37:59.037 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:37:59.037 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:37:59.037 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:37:59.037 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:37:59.037 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:37:59.037 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:37:59.037 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:37:59.037 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:37:59.038 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:37:59.038 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:37:59.038 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:37:59.038 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:37:59.038 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:37:59.038 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:37:59.038 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:37:59.038 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:59.023+0000 7fc50ffff700 10 monclient: _finish_auth 0 2023-09-30T01:37:59.038 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:59.023+0000 7fc50ffff700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:37:59.038 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:59.023+0000 7fc5163a7700 5 monclient: authenticate success, global_id 4864 2023-09-30T01:37:59.038 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:59.023+0000 7fc50ffff700 1 -- 172.21.15.88:0/18075902 <== mon.4 v2:172.21.15.88:3302/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc50800be60 con 0x7fc5101386e0 2023-09-30T01:37:59.038 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:59.023+0000 7fc50ffff700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:37:59.038 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:59.023+0000 7fc5163a7700 10 monclient: get_monmap_and_config success 2023-09-30T01:37:59.039 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:59.023+0000 7fc50ffff700 1 -- 172.21.15.88:0/18075902 <== mon.4 v2:172.21.15.88:3302/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7fc5080118a0 con 0x7fc5101386e0 2023-09-30T01:37:59.039 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:59.023+0000 7fc5163a7700 4 set_mon_vals no callback set 2023-09-30T01:37:59.039 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:59.023+0000 7fc50ffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:37:59.039 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:59.023+0000 7fc50ffff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T01:37:59.039 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:59.023+0000 7fc50ffff700 10 monclient: dump: 2023-09-30T01:37:59.039 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:37:59.039 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:37:59.039 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:37:59.039 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:37:59.039 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:37:59.039 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:37:59.039 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:37:59.040 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:37:59.040 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:37:59.040 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:37:59.040 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:37:59.040 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:37:59.040 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:37:59.040 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:37:59.040 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:37:59.040 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:37:59.040 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:59.023+0000 7fc5163a7700 10 monclient: shutdown 2023-09-30T01:37:59.040 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:59.023+0000 7fc5163a7700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:37:59.040 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:59.023+0000 7fc5163a7700 1 -- 172.21.15.88:0/18075902 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7fc5101386e0 msgr2=0x7fc51013aac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:37:59.041 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:59.023+0000 7fc5163a7700 1 --2- 172.21.15.88:0/18075902 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7fc5101386e0 0x7fc51013aac0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fc5080099e0 tx=0x7fc508009250 comp rx=0 tx=0).stop 2023-09-30T01:37:59.041 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:59.023+0000 7fc5163a7700 1 -- 172.21.15.88:0/18075902 shutdown_connections 2023-09-30T01:37:59.041 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:59.023+0000 7fc5163a7700 1 --2- 172.21.15.88:0/18075902 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7fc5101386e0 0x7fc51013aac0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:37:59.041 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:59.023+0000 7fc5163a7700 1 --2- 172.21.15.88:0/18075902 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7fc51009ffd0 0x7fc5100a03a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:37:59.041 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:59.023+0000 7fc5163a7700 1 -- 172.21.15.88:0/18075902 >> 172.21.15.88:0/18075902 conn(0x7fc51006e5e0 msgr2=0x7fc51006e9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:37:59.041 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:59.023+0000 7fc5163a7700 1 -- 172.21.15.88:0/18075902 shutdown_connections 2023-09-30T01:37:59.041 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:59.023+0000 7fc5163a7700 1 -- 172.21.15.88:0/18075902 wait complete. 2023-09-30T01:37:59.041 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:59.027+0000 7fc5163a7700 10 monclient: build_initial_monmap 2023-09-30T01:37:59.041 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:59.027+0000 7fc5163a7700 10 monclient: monmap: 2023-09-30T01:37:59.041 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2023-09-30T01:37:59.041 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:37:59.041 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 0.000000 2023-09-30T01:37:59.041 INFO:teuthology.orchestra.run.smithi088.stderr:created 0.000000 2023-09-30T01:37:59.042 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2023-09-30T01:37:59.042 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 1 2023-09-30T01:37:59.042 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.noname-a 2023-09-30T01:37:59.042 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.noname-f 2023-09-30T01:37:59.042 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.noname-b 2023-09-30T01:37:59.042 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.noname-g 2023-09-30T01:37:59.042 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.noname-c 2023-09-30T01:37:59.042 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.noname-h 2023-09-30T01:37:59.042 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.noname-d 2023-09-30T01:37:59.042 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.noname-i 2023-09-30T01:37:59.042 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.noname-e 2023-09-30T01:37:59.042 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:37:59.043 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:59.027+0000 7fc5163a7700 1 Processor -- start 2023-09-30T01:37:59.043 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:59.027+0000 7fc5163a7700 1 -- start start 2023-09-30T01:37:59.043 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:59.027+0000 7fc5163a7700 10 monclient: init 2023-09-30T01:37:59.043 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:59.027+0000 7fc5163a7700 10 monclient: _reopen_session rank -1 2023-09-30T01:37:59.043 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:59.027+0000 7fc5163a7700 10 monclient: _add_conns ranks=[4,6,7,2,8,5,1,3,0] 2023-09-30T01:37:59.043 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:59.027+0000 7fc5163a7700 1 --2- >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7fc51009ffd0 0x7fc51009b720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:37:59.043 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:59.027+0000 7fc5163a7700 10 monclient(hunting): picked mon.noname-c con 0x7fc51009ffd0 addr [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] 2023-09-30T01:37:59.043 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:59.027+0000 7fc5163a7700 1 --2- >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7fc5100a3380 0x7fc51009bc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:37:59.043 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:59.027+0000 7fc5163a7700 10 monclient(hunting): picked mon.noname-d con 0x7fc5100a3380 addr [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] 2023-09-30T01:37:59.043 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:59.027+0000 7fc5163a7700 1 --2- >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7fc5101386e0 0x7fc51009c1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:37:59.043 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:59.027+0000 7fc5163a7700 10 monclient(hunting): picked mon.noname-i con 0x7fc5101386e0 addr [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] 2023-09-30T01:37:59.043 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:59.027+0000 7fc5163a7700 10 monclient(hunting): start opening mon connection 2023-09-30T01:37:59.043 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:59.027+0000 7fc5163a7700 1 -- --> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] -- mon_getmap magic: 0 v1 -- 0x7fc5100a0ac0 con 0x7fc51009ffd0 2023-09-30T01:37:59.044 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:59.027+0000 7fc5163a7700 10 monclient(hunting): start opening mon connection 2023-09-30T01:37:59.044 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:59.027+0000 7fc5163a7700 1 -- --> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] -- mon_getmap magic: 0 v1 -- 0x7fc510097ce0 con 0x7fc5100a3380 2023-09-30T01:37:59.044 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:59.027+0000 7fc5163a7700 10 monclient(hunting): start opening mon connection 2023-09-30T01:37:59.044 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:59.027+0000 7fc5163a7700 1 -- --> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] -- mon_getmap magic: 0 v1 -- 0x7fc51009f660 con 0x7fc5101386e0 2023-09-30T01:37:59.044 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:59.027+0000 7fc5163a7700 10 monclient(hunting): _renew_subs 2023-09-30T01:37:59.044 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:59.027+0000 7fc5163a7700 10 monclient(hunting): authenticate will time out at 1151.066009s 2023-09-30T01:37:59.044 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:59.027+0000 7fc5153a5700 1 --2- >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7fc51009ffd0 0x7fc51009b720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:37:59.044 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:59.027+0000 7fc514ba4700 1 --2- >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7fc5100a3380 0x7fc51009bc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:37:59.044 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:59.027+0000 7fc5153a5700 1 --2- >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7fc51009ffd0 0x7fc51009b720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3302/0 says I am v2:172.21.15.88:38850/0 (socket says 172.21.15.88:38850) 2023-09-30T01:37:59.044 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:59.027+0000 7fc5153a5700 1 -- 172.21.15.88:0/2622171837 learned_addr learned my addr 172.21.15.88:0/2622171837 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-09-30T01:37:59.044 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:59.027+0000 7fc5153a5700 10 monclient(hunting): get_auth_request con 0x7fc51009ffd0 auth_method 0 2023-09-30T01:37:59.044 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:59.027+0000 7fc5153a5700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:37:59.045 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:59.027+0000 7fc5153a5700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:37:59.045 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:59.027+0000 7fc5153a5700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:37:59.045 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:59.027+0000 7fc514ba4700 10 monclient(hunting): get_auth_request con 0x7fc5100a3380 auth_method 0 2023-09-30T01:37:59.045 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:59.027+0000 7fc514ba4700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:37:59.045 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:59.027+0000 7fc514ba4700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:37:59.045 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:59.027+0000 7fc514ba4700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:37:59.045 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:59.027+0000 7fc515ba6700 1 --2- 172.21.15.88:0/2622171837 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7fc5101386e0 0x7fc51009c1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:37:59.045 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:59.027+0000 7fc5153a5700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:37:59.045 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:59.027+0000 7fc5153a5700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:37:59.045 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:59.027+0000 7fc5153a5700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:37:59.045 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:59.027+0000 7fc515ba6700 10 monclient(hunting): get_auth_request con 0x7fc5101386e0 auth_method 0 2023-09-30T01:37:59.045 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:59.027+0000 7fc515ba6700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:37:59.045 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:59.027+0000 7fc515ba6700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:37:59.046 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:59.027+0000 7fc515ba6700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:37:59.046 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:59.027+0000 7fc514ba4700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:37:59.046 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:59.027+0000 7fc514ba4700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:37:59.046 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:59.027+0000 7fc514ba4700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:37:59.046 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:59.027+0000 7fc515ba6700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:37:59.046 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:59.027+0000 7fc515ba6700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:37:59.046 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:59.027+0000 7fc515ba6700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:37:59.046 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:59.027+0000 7fc5153a5700 10 monclient(hunting): handle_auth_done global_id 4873 payload 824 2023-09-30T01:37:59.046 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:59.027+0000 7fc5153a5700 1 -- 172.21.15.88:0/2622171837 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7fc5101386e0 msgr2=0x7fc51009c1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:37:59.046 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:59.027+0000 7fc5153a5700 1 --2- 172.21.15.88:0/2622171837 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7fc5101386e0 0x7fc51009c1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:37:59.046 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:59.027+0000 7fc5153a5700 1 -- 172.21.15.88:0/2622171837 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7fc5100a3380 msgr2=0x7fc51009bc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:37:59.046 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:59.027+0000 7fc5153a5700 1 --2- 172.21.15.88:0/2622171837 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7fc5100a3380 0x7fc51009bc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:37:59.047 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:59.027+0000 7fc5153a5700 10 monclient: _finish_hunting 0 2023-09-30T01:37:59.047 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:59.027+0000 7fc5153a5700 1 monclient: found mon.noname-c 2023-09-30T01:37:59.047 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:59.027+0000 7fc5153a5700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:37:59.047 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:59.027+0000 7fc5153a5700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.88:3302/0 2023-09-30T01:37:59.047 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:59.027+0000 7fc5153a5700 1 -- 172.21.15.88:0/2622171837 --> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5080094f0 con 0x7fc51009ffd0 2023-09-30T01:37:59.047 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:59.027+0000 7fc5153a5700 1 --2- 172.21.15.88:0/2622171837 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7fc51009ffd0 0x7fc51009b720 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fc50400c790 tx=0x7fc504007f40 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:37:59.047 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:59.027+0000 7fc50d7fa700 1 -- 172.21.15.88:0/2622171837 <== mon.4 v2:172.21.15.88:3302/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7fc504011a10 con 0x7fc51009ffd0 2023-09-30T01:37:59.047 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:59.027+0000 7fc50d7fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:37:59.047 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:59.027+0000 7fc50d7fa700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T01:37:59.047 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:59.027+0000 7fc50d7fa700 10 monclient: dump: 2023-09-30T01:37:59.047 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:37:59.047 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:37:59.048 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:37:59.048 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:37:59.048 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:37:59.048 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:37:59.048 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:37:59.048 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:37:59.048 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:37:59.048 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:37:59.048 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:37:59.048 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:37:59.048 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:37:59.048 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:37:59.048 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:37:59.049 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:37:59.049 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:59.027+0000 7fc50d7fa700 10 monclient: _finish_auth 0 2023-09-30T01:37:59.049 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:59.027+0000 7fc50d7fa700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:37:59.049 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:59.027+0000 7fc5163a7700 5 monclient: authenticate success, global_id 4873 2023-09-30T01:37:59.049 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:59.027+0000 7fc50d7fa700 1 -- 172.21.15.88:0/2622171837 <== mon.4 v2:172.21.15.88:3302/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc504011b90 con 0x7fc51009ffd0 2023-09-30T01:37:59.049 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:59.027+0000 7fc5163a7700 10 monclient: _renew_subs 2023-09-30T01:37:59.049 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:59.027+0000 7fc5163a7700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.88:3302/0 2023-09-30T01:37:59.049 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:59.027+0000 7fc5163a7700 1 -- 172.21.15.88:0/2622171837 --> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc51009f310 con 0x7fc51009ffd0 2023-09-30T01:37:59.049 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:59.027+0000 7fc50d7fa700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:37:59.049 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:59.027+0000 7fc50d7fa700 1 -- 172.21.15.88:0/2622171837 <== mon.4 v2:172.21.15.88:3302/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7fc504011d00 con 0x7fc51009ffd0 2023-09-30T01:37:59.049 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:59.027+0000 7fc50d7fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:37:59.049 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:59.027+0000 7fc50d7fa700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T01:37:59.050 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:59.027+0000 7fc50d7fa700 10 monclient: dump: 2023-09-30T01:37:59.050 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:37:59.050 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:37:59.050 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:37:59.050 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:37:59.050 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:37:59.050 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:37:59.050 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:37:59.050 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:37:59.050 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:37:59.050 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:37:59.050 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:37:59.051 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:37:59.051 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:37:59.051 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:37:59.051 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:37:59.051 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:37:59.051 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:59.027+0000 7fc5163a7700 10 monclient: _renew_subs 2023-09-30T01:37:59.051 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:59.027+0000 7fc5163a7700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.88:3302/0 2023-09-30T01:37:59.051 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:59.027+0000 7fc5163a7700 1 -- 172.21.15.88:0/2622171837 --> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc510137ff0 con 0x7fc51009ffd0 2023-09-30T01:37:59.051 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:59.027+0000 7fc50f7fe700 4 set_mon_vals no callback set 2023-09-30T01:37:59.051 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:59.031+0000 7fc50d7fa700 1 -- 172.21.15.88:0/2622171837 <== mon.4 v2:172.21.15.88:3302/0 4 ==== mgrmap(e 5) v1 ==== 40144+0+0 (secure 0 0 0) 0x7fc504011590 con 0x7fc51009ffd0 2023-09-30T01:37:59.051 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:59.031+0000 7fc5163a7700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:37:59.051 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:59.031+0000 7fc5163a7700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:37:59.051 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:59.031+0000 7fc5163a7700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.88:3302/0 2023-09-30T01:37:59.052 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:59.031+0000 7fc5163a7700 1 -- 172.21.15.88:0/2622171837 --> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc51009eea0 con 0x7fc51009ffd0 2023-09-30T01:37:59.052 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:59.031+0000 7fc50d7fa700 1 --2- 172.21.15.88:0/2622171837 >> v2:172.21.15.107:6812/29926 conn(0x7fc4fc033360 0x7fc4fc035810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:37:59.052 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:59.031+0000 7fc50d7fa700 1 -- 172.21.15.88:0/2622171837 <== mon.4 v2:172.21.15.88:3302/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 3502+0+0 (secure 0 0 0) 0x7fc504042390 con 0x7fc51009ffd0 2023-09-30T01:37:59.052 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:59.031+0000 7fc514ba4700 1 --2- 172.21.15.88:0/2622171837 >> v2:172.21.15.107:6812/29926 conn(0x7fc4fc033360 0x7fc4fc035810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:37:59.052 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:59.031+0000 7fc514ba4700 10 monclient: get_auth_request con 0x7fc4fc033360 auth_method 0 2023-09-30T01:37:59.052 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:59.031+0000 7fc514ba4700 1 --2- 172.21.15.88:0/2622171837 >> v2:172.21.15.107:6812/29926 conn(0x7fc4fc033360 0x7fc4fc035810 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fc500009fd0 tx=0x7fc500009880 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:37:59.053 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:59.039+0000 7fc50d7fa700 1 -- 172.21.15.88:0/2622171837 <== mon.4 v2:172.21.15.88:3302/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fc504011840 con 0x7fc51009ffd0 2023-09-30T01:37:59.053 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:59.039+0000 7fc50d7fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:37:59.053 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:59.039+0000 7fc50d7fa700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:37:59.233 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:59.231+0000 7fc5163a7700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T01:37:59.234 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:59.231+0000 7fc5163a7700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:37:59.234 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:59.231+0000 7fc5163a7700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.88:3302/0 2023-09-30T01:37:59.234 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:59.231+0000 7fc5163a7700 1 -- 172.21.15.88:0/2622171837 --> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fc5100710d0 con 0x7fc51009ffd0 2023-09-30T01:37:59.234 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:59.231+0000 7fc50d7fa700 1 -- 172.21.15.88:0/2622171837 <== mon.4 v2:172.21.15.88:3302/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7fc504017030 con 0x7fc51009ffd0 2023-09-30T01:37:59.234 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:59.231+0000 7fc50d7fa700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:37:59.235 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:59.231+0000 7fc50d7fa700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:37:59.235 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-09-30T01:37:59.235 INFO:teuthology.orchestra.run.smithi088.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:37:59.236 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:59.235+0000 7fc5163a7700 1 -- 172.21.15.88:0/2622171837 >> v2:172.21.15.107:6812/29926 conn(0x7fc4fc033360 msgr2=0x7fc4fc035810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:37:59.236 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:59.235+0000 7fc5163a7700 1 --2- 172.21.15.88:0/2622171837 >> v2:172.21.15.107:6812/29926 conn(0x7fc4fc033360 0x7fc4fc035810 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fc500009fd0 tx=0x7fc500009880 comp rx=0 tx=0).stop 2023-09-30T01:37:59.237 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:59.235+0000 7fc5163a7700 10 monclient: shutdown 2023-09-30T01:37:59.237 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:59.235+0000 7fc5163a7700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:37:59.237 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:59.235+0000 7fc5163a7700 1 -- 172.21.15.88:0/2622171837 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7fc51009ffd0 msgr2=0x7fc51009b720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:37:59.237 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:59.235+0000 7fc5163a7700 1 --2- 172.21.15.88:0/2622171837 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7fc51009ffd0 0x7fc51009b720 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fc50400c790 tx=0x7fc504007f40 comp rx=0 tx=0).stop 2023-09-30T01:37:59.237 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:59.235+0000 7fc5163a7700 1 -- 172.21.15.88:0/2622171837 shutdown_connections 2023-09-30T01:37:59.237 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:59.235+0000 7fc5163a7700 1 --2- 172.21.15.88:0/2622171837 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7fc5101386e0 0x7fc51009c1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:37:59.237 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:59.235+0000 7fc5163a7700 1 --2- 172.21.15.88:0/2622171837 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7fc5100a3380 0x7fc51009bc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:37:59.237 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:59.235+0000 7fc5163a7700 1 --2- 172.21.15.88:0/2622171837 >> v2:172.21.15.107:6812/29926 conn(0x7fc4fc033360 0x7fc4fc035810 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:37:59.237 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:59.235+0000 7fc5163a7700 1 --2- 172.21.15.88:0/2622171837 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7fc51009ffd0 0x7fc51009b720 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:37:59.237 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:59.235+0000 7fc5163a7700 1 -- 172.21.15.88:0/2622171837 >> 172.21.15.88:0/2622171837 conn(0x7fc51006e5e0 msgr2=0x7fc51012e760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:37:59.237 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:59.235+0000 7fc5163a7700 1 -- 172.21.15.88:0/2622171837 shutdown_connections 2023-09-30T01:37:59.238 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:37:59.235+0000 7fc5163a7700 1 -- 172.21.15.88:0/2622171837 wait complete. 2023-09-30T01:38:02.249 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:38:02.360 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:02.359+0000 7fa8b759e700 10 monclient: get_monmap_and_config 2023-09-30T01:38:02.360 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:02.359+0000 7fa8b759e700 10 monclient: build_initial_monmap 2023-09-30T01:38:02.360 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:02.359+0000 7fa8b759e700 10 monclient: monmap: 2023-09-30T01:38:02.360 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2023-09-30T01:38:02.360 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:38:02.360 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:38:02.360052+0000 2023-09-30T01:38:02.361 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:38:02.360052+0000 2023-09-30T01:38:02.361 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2023-09-30T01:38:02.361 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:38:02.361 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a-legacy 2023-09-30T01:38:02.361 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.107:6789/0 mon.noname-f-legacy 2023-09-30T01:38:02.361 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.noname-b 2023-09-30T01:38:02.361 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.noname-g 2023-09-30T01:38:02.361 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.noname-c 2023-09-30T01:38:02.361 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.noname-h 2023-09-30T01:38:02.361 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.noname-d 2023-09-30T01:38:02.361 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.noname-i 2023-09-30T01:38:02.361 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.noname-e 2023-09-30T01:38:02.362 INFO:teuthology.orchestra.run.smithi088.stderr:9: v2:172.21.15.88:3300/0 mon.noname-a 2023-09-30T01:38:02.362 INFO:teuthology.orchestra.run.smithi088.stderr:10: v2:172.21.15.107:3300/0 mon.noname-f 2023-09-30T01:38:02.362 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:38:02.366 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:02.359+0000 7fa8b759e700 1 Processor -- start 2023-09-30T01:38:02.366 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:02.359+0000 7fa8b759e700 1 -- start start 2023-09-30T01:38:02.366 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:02.359+0000 7fa8b759e700 10 monclient: init 2023-09-30T01:38:02.366 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:02.359+0000 7fa8b759e700 10 monclient: _reopen_session rank -1 2023-09-30T01:38:02.366 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:02.359+0000 7fa8b759e700 10 monclient: _add_conns ranks=[5,0,3,6,9,2,7,10,4,8,1] 2023-09-30T01:38:02.366 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:02.359+0000 7fa8b759e700 1 --2- >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7fa8b80a3380 0x7fa8b80a48b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:38:02.366 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:02.359+0000 7fa8b759e700 10 monclient(hunting): picked mon.noname-h con 0x7fa8b80a3380 addr [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] 2023-09-30T01:38:02.366 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:02.359+0000 7fa8b759e700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fa8b809ffd0 addr v1:172.21.15.88:6789/0 2023-09-30T01:38:02.366 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:02.359+0000 7fa8b759e700 1 --2- >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7fa8b80a03a0 0x7fa8b8133ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:38:02.367 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:02.359+0000 7fa8b759e700 10 monclient(hunting): picked mon.noname-g con 0x7fa8b80a03a0 addr [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] 2023-09-30T01:38:02.367 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:02.359+0000 7fa8b759e700 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8b804bb20 con 0x7fa8b809ffd0 2023-09-30T01:38:02.367 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:02.359+0000 7fa8b759e700 10 monclient(hunting): start opening mon connection 2023-09-30T01:38:02.367 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:02.359+0000 7fa8b759e700 1 -- --> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8b8097ab0 con 0x7fa8b80a03a0 2023-09-30T01:38:02.367 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:02.359+0000 7fa8b759e700 10 monclient(hunting): start opening mon connection 2023-09-30T01:38:02.367 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:02.359+0000 7fa8b759e700 1 -- --> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] -- mon_getmap magic: 0 v1 -- 0x7fa8b8098480 con 0x7fa8b80a3380 2023-09-30T01:38:02.367 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:02.359+0000 7fa8b759e700 10 monclient(hunting): _renew_subs 2023-09-30T01:38:02.367 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:02.359+0000 7fa8b759e700 10 monclient(hunting): authenticate will time out at 1154.397549s 2023-09-30T01:38:02.367 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:02.359+0000 7fa8add9b700 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7fa8b809ffd0 0x7fa8b81333b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:49974/0 (socket says 172.21.15.88:49974) 2023-09-30T01:38:02.367 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:02.359+0000 7fa8add9b700 1 -- 172.21.15.88:0/3748015688 learned_addr learned my addr 172.21.15.88:0/3748015688 (peer_addr_for_me v1:172.21.15.88:0/0) 2023-09-30T01:38:02.367 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:02.359+0000 7fa8b6d9d700 1 --2- 172.21.15.88:0/3748015688 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7fa8b80a03a0 0x7fa8b8133ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:38:02.367 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:02.359+0000 7fa8b5d9b700 1 -- 172.21.15.88:0/3748015688 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2430582138 0 0) 0x7fa8b804bb20 con 0x7fa8b809ffd0 2023-09-30T01:38:02.368 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:02.359+0000 7fa8b5d9b700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:38:02.368 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:02.359+0000 7fa8b5d9b700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:38:02.368 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:02.359+0000 7fa8b5d9b700 10 monclient(hunting): my global_id is 5157 2023-09-30T01:38:02.368 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:02.359+0000 7fa8b659c700 1 --2- 172.21.15.88:0/3748015688 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7fa8b80a3380 0x7fa8b80a48b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:38:02.368 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:02.359+0000 7fa8b5d9b700 1 -- 172.21.15.88:0/3748015688 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa8a4003570 con 0x7fa8b809ffd0 2023-09-30T01:38:02.368 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:02.359+0000 7fa8b6d9d700 10 monclient(hunting): get_auth_request con 0x7fa8b80a03a0 auth_method 0 2023-09-30T01:38:02.368 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:02.359+0000 7fa8b6d9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:38:02.368 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:02.359+0000 7fa8b6d9d700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:38:02.368 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:02.359+0000 7fa8b6d9d700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:38:02.368 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:02.359+0000 7fa8b659c700 10 monclient(hunting): get_auth_request con 0x7fa8b80a3380 auth_method 0 2023-09-30T01:38:02.368 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:02.359+0000 7fa8b659c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:38:02.368 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:02.359+0000 7fa8b659c700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:38:02.368 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:02.359+0000 7fa8b659c700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:38:02.369 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:02.359+0000 7fa8b5d9b700 1 -- 172.21.15.88:0/3748015688 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2746685076 0 0) 0x7fa8a4003570 con 0x7fa8b809ffd0 2023-09-30T01:38:02.369 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:02.359+0000 7fa8b5d9b700 1 -- 172.21.15.88:0/3748015688 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7fa8b80a3380 msgr2=0x7fa8b80a48b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:38:02.369 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:02.359+0000 7fa8b5d9b700 1 --2- 172.21.15.88:0/3748015688 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7fa8b80a3380 0x7fa8b80a48b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:38:02.369 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:02.359+0000 7fa8b5d9b700 1 -- 172.21.15.88:0/3748015688 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7fa8b80a03a0 msgr2=0x7fa8b8133ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:38:02.369 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:02.359+0000 7fa8b5d9b700 1 --2- 172.21.15.88:0/3748015688 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7fa8b80a03a0 0x7fa8b8133ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:38:02.369 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:02.359+0000 7fa8b5d9b700 10 monclient: _finish_hunting 0 2023-09-30T01:38:02.369 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:02.359+0000 7fa8b5d9b700 1 monclient: found mon.noname-a-legacy 2023-09-30T01:38:02.369 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:02.359+0000 7fa8b5d9b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:38:02.369 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:02.359+0000 7fa8b5d9b700 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.88:6789/0 2023-09-30T01:38:02.369 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:02.359+0000 7fa8b5d9b700 1 -- 172.21.15.88:0/3748015688 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8b80a3750 con 0x7fa8b809ffd0 2023-09-30T01:38:02.369 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:02.359+0000 7fa8b5d9b700 10 monclient: _finish_auth 0 2023-09-30T01:38:02.369 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:02.359+0000 7fa8b5d9b700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:38:02.370 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:02.359+0000 7fa8b5d9b700 1 -- 172.21.15.88:0/3748015688 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (unknown 176869241 0 0) 0x7fa8a80018f0 con 0x7fa8b809ffd0 2023-09-30T01:38:02.370 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:02.359+0000 7fa8b759e700 5 monclient: authenticate success, global_id 5157 2023-09-30T01:38:02.370 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:02.359+0000 7fa8b759e700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:38:02.370 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:02.359+0000 7fa8b6d9d700 1 --2- 172.21.15.88:0/3748015688 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7fa8b80a03a0 0x7fa8b8133ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-30T01:38:02.370 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:02.359+0000 7fa8b5d9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:38:02.370 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:02.359+0000 7fa8b5d9b700 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2023-09-30T01:38:02.370 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:02.359+0000 7fa8b5d9b700 10 monclient: dump: 2023-09-30T01:38:02.370 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:38:02.370 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:38:02.370 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:38:02.370 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:38:02.370 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:38:02.370 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:38:02.371 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:38:02.371 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:38:02.371 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:38:02.371 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:38:02.371 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:38:02.371 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:38:02.371 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:38:02.371 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:38:02.371 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:38:02.371 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:38:02.371 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:02.359+0000 7fa8b5d9b700 1 monclient: mon.a has (v2) addrs [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] but i'm connected to v1:172.21.15.88:6789/0, reconnecting 2023-09-30T01:38:02.371 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:02.359+0000 7fa8b5d9b700 10 monclient: _reopen_session rank -1 2023-09-30T01:38:02.372 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:02.359+0000 7fa8b5d9b700 1 -- 172.21.15.88:0/3748015688 >> v1:172.21.15.88:6789/0 conn(0x7fa8b809ffd0 legacy=0x7fa8b81333b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:38:02.372 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:02.359+0000 7fa8b5d9b700 10 monclient: _add_conns ranks=[2,8,4,6,5,3,0,1,7] 2023-09-30T01:38:02.372 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:02.359+0000 7fa8b5d9b700 1 --2- 172.21.15.88:0/3748015688 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa8a40042e0 0x7fa8a4004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:38:02.372 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:02.359+0000 7fa8b5d9b700 10 monclient(hunting): picked mon.b con 0x7fa8a40042e0 addr [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] 2023-09-30T01:38:02.372 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:02.359+0000 7fa8b5d9b700 1 --2- 172.21.15.88:0/3748015688 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7fa8a4004e70 0x7fa8a40052e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:38:02.372 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:02.359+0000 7fa8b5d9b700 10 monclient(hunting): picked mon.e con 0x7fa8a4004e70 addr [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] 2023-09-30T01:38:02.372 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:02.359+0000 7fa8b5d9b700 1 --2- 172.21.15.88:0/3748015688 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7fa8a4009cb0 0x7fa8a400c130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:38:02.372 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:02.359+0000 7fa8b5d9b700 10 monclient(hunting): picked mon.c con 0x7fa8a4009cb0 addr [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] 2023-09-30T01:38:02.372 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:02.359+0000 7fa8b5d9b700 10 monclient(hunting): start opening mon connection 2023-09-30T01:38:02.372 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:02.359+0000 7fa8b5d9b700 1 -- 172.21.15.88:0/3748015688 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8b8097ab0 con 0x7fa8a40042e0 2023-09-30T01:38:02.372 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:02.359+0000 7fa8b5d9b700 10 monclient(hunting): start opening mon connection 2023-09-30T01:38:02.372 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:02.359+0000 7fa8b5d9b700 1 -- 172.21.15.88:0/3748015688 --> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] -- mon_getmap magic: 0 v1 -- 0x7fa8b8098480 con 0x7fa8a4009cb0 2023-09-30T01:38:02.373 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:02.359+0000 7fa8b5d9b700 10 monclient(hunting): start opening mon connection 2023-09-30T01:38:02.373 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:02.359+0000 7fa8b5d9b700 1 -- 172.21.15.88:0/3748015688 --> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] -- mon_getmap magic: 0 v1 -- 0x7fa8a400ca40 con 0x7fa8a4004e70 2023-09-30T01:38:02.373 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:02.359+0000 7fa8b5d9b700 10 monclient(hunting): _renew_subs 2023-09-30T01:38:02.373 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:02.359+0000 7fa8b5d9b700 10 monclient(hunting): _finish_auth 0 2023-09-30T01:38:02.373 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:02.359+0000 7fa8b759e700 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2023-09-30T01:38:02.373 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:02.359+0000 7fa8add9b700 1 --2- 172.21.15.88:0/3748015688 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7fa8a4004e70 0x7fa8a40052e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:38:02.373 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:02.359+0000 7fa8b659c700 1 --2- 172.21.15.88:0/3748015688 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa8a40042e0 0x7fa8a4004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:38:02.373 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:02.359+0000 7fa8add9b700 10 monclient(hunting): get_auth_request con 0x7fa8a4004e70 auth_method 0 2023-09-30T01:38:02.373 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:02.359+0000 7fa8add9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:38:02.373 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:02.359+0000 7fa8add9b700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:38:02.373 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:02.359+0000 7fa8add9b700 10 monclient(hunting): _init_auth already have auth, reseting 2023-09-30T01:38:02.373 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:02.359+0000 7fa8b659c700 10 monclient(hunting): get_auth_request con 0x7fa8a40042e0 auth_method 0 2023-09-30T01:38:02.374 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:02.359+0000 7fa8b659c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:38:02.374 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:02.359+0000 7fa8b659c700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:38:02.374 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:02.359+0000 7fa8b659c700 10 monclient(hunting): _init_auth already have auth, reseting 2023-09-30T01:38:02.374 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:02.359+0000 7fa8b6d9d700 1 --2- 172.21.15.88:0/3748015688 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7fa8a4009cb0 0x7fa8a400c130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:38:02.374 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:02.359+0000 7fa8add9b700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:38:02.374 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:02.359+0000 7fa8add9b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:38:02.374 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:02.359+0000 7fa8add9b700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-09-30T01:38:02.374 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:02.359+0000 7fa8b659c700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:38:02.374 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:02.359+0000 7fa8b659c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:38:02.374 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:02.359+0000 7fa8b659c700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-09-30T01:38:02.374 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:02.359+0000 7fa8b6d9d700 10 monclient(hunting): get_auth_request con 0x7fa8a4009cb0 auth_method 0 2023-09-30T01:38:02.374 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:02.359+0000 7fa8b6d9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:38:02.375 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:02.359+0000 7fa8b6d9d700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:38:02.375 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:02.359+0000 7fa8b6d9d700 10 monclient(hunting): _init_auth already have auth, reseting 2023-09-30T01:38:02.375 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:02.359+0000 7fa8b6d9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:38:02.375 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:02.359+0000 7fa8b6d9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:38:02.375 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:02.359+0000 7fa8b6d9d700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-09-30T01:38:02.375 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:02.359+0000 7fa8b659c700 10 monclient(hunting): handle_auth_done global_id 5157 payload 293 2023-09-30T01:38:02.375 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:02.359+0000 7fa8b659c700 1 -- 172.21.15.88:0/3748015688 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7fa8a4004e70 msgr2=0x7fa8a40052e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:38:02.375 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:02.359+0000 7fa8b659c700 1 --2- 172.21.15.88:0/3748015688 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7fa8a4004e70 0x7fa8a40052e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:38:02.375 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:02.359+0000 7fa8b659c700 1 -- 172.21.15.88:0/3748015688 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7fa8a4009cb0 msgr2=0x7fa8a400c130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:38:02.375 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:02.359+0000 7fa8b659c700 1 --2- 172.21.15.88:0/3748015688 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7fa8a4009cb0 0x7fa8a400c130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:38:02.375 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:02.359+0000 7fa8b659c700 10 monclient: _finish_hunting 0 2023-09-30T01:38:02.375 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:02.359+0000 7fa8b659c700 1 monclient: found mon.b 2023-09-30T01:38:02.376 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:02.359+0000 7fa8b659c700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:38:02.376 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:02.359+0000 7fa8b659c700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.88:3301/0 2023-09-30T01:38:02.376 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:02.359+0000 7fa8b659c700 1 -- 172.21.15.88:0/3748015688 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8a4003570 con 0x7fa8a40042e0 2023-09-30T01:38:02.376 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:02.359+0000 7fa8b659c700 10 monclient: _finish_auth 0 2023-09-30T01:38:02.376 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:02.359+0000 7fa8b659c700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:38:02.376 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:02.359+0000 7fa8b6d9d700 1 -- 172.21.15.88:0/3748015688 reap_dead start 2023-09-30T01:38:02.376 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:02.359+0000 7fa8b659c700 1 --2- 172.21.15.88:0/3748015688 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa8a40042e0 0x7fa8a4004810 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fa8a000bb90 tx=0x7fa8a0008040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7d2b801c71236660 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:38:02.376 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:02.359+0000 7fa8b5d9b700 1 -- 172.21.15.88:0/3748015688 <== mon.2 v2:172.21.15.88:3301/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7fa8a000da50 con 0x7fa8a40042e0 2023-09-30T01:38:02.376 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:02.363+0000 7fa8b5d9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:38:02.376 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:02.363+0000 7fa8b5d9b700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T01:38:02.376 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:02.363+0000 7fa8b5d9b700 10 monclient: dump: 2023-09-30T01:38:02.376 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:38:02.376 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:38:02.377 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:38:02.377 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:38:02.377 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:38:02.377 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:38:02.377 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:38:02.377 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:38:02.377 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:38:02.377 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:38:02.377 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:38:02.377 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:38:02.377 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:38:02.377 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:38:02.378 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:38:02.378 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:38:02.378 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:02.363+0000 7fa8b5d9b700 1 -- 172.21.15.88:0/3748015688 <== mon.2 v2:172.21.15.88:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa8a000dbf0 con 0x7fa8a40042e0 2023-09-30T01:38:02.378 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:02.363+0000 7fa8b759e700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:38:02.378 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:02.363+0000 7fa8b5d9b700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:38:02.378 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:02.363+0000 7fa8b759e700 10 monclient: get_monmap_and_config success 2023-09-30T01:38:02.378 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:02.363+0000 7fa8b5d9b700 1 -- 172.21.15.88:0/3748015688 <== mon.2 v2:172.21.15.88:3301/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7fa8a000dd60 con 0x7fa8a40042e0 2023-09-30T01:38:02.378 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:02.363+0000 7fa8b5d9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:38:02.378 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:02.363+0000 7fa8b759e700 4 set_mon_vals no callback set 2023-09-30T01:38:02.378 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:02.363+0000 7fa8b5d9b700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T01:38:02.378 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:02.363+0000 7fa8b5d9b700 10 monclient: dump: 2023-09-30T01:38:02.378 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:38:02.378 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:38:02.379 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:38:02.379 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:38:02.379 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:38:02.379 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:38:02.379 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:38:02.379 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:38:02.379 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:38:02.379 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:38:02.379 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:38:02.379 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:38:02.379 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:38:02.379 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:38:02.379 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:38:02.380 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:38:02.380 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:02.363+0000 7fa8b759e700 10 monclient: shutdown 2023-09-30T01:38:02.380 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:02.363+0000 7fa8b759e700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:38:02.380 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:02.363+0000 7fa8b759e700 1 -- 172.21.15.88:0/3748015688 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa8a40042e0 msgr2=0x7fa8a4004810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:38:02.380 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:02.363+0000 7fa8b759e700 1 --2- 172.21.15.88:0/3748015688 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa8a40042e0 0x7fa8a4004810 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fa8a000bb90 tx=0x7fa8a0008040 comp rx=0 tx=0).stop 2023-09-30T01:38:02.380 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:02.363+0000 7fa8b759e700 1 -- 172.21.15.88:0/3748015688 shutdown_connections 2023-09-30T01:38:02.380 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:02.363+0000 7fa8b759e700 1 --2- 172.21.15.88:0/3748015688 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa8a40042e0 0x7fa8a4004810 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:38:02.380 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:02.363+0000 7fa8b759e700 1 -- 172.21.15.88:0/3748015688 >> 172.21.15.88:0/3748015688 conn(0x7fa8b806e5e0 msgr2=0x7fa8b806e9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:38:02.380 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:02.363+0000 7fa8b759e700 1 -- 172.21.15.88:0/3748015688 shutdown_connections 2023-09-30T01:38:02.380 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:02.363+0000 7fa8b759e700 1 -- 172.21.15.88:0/3748015688 wait complete. 2023-09-30T01:38:02.384 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:02.363+0000 7fa8b759e700 10 monclient: build_initial_monmap 2023-09-30T01:38:02.384 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:02.363+0000 7fa8b759e700 10 monclient: monmap: 2023-09-30T01:38:02.384 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2023-09-30T01:38:02.384 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:38:02.385 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 0.000000 2023-09-30T01:38:02.385 INFO:teuthology.orchestra.run.smithi088.stderr:created 0.000000 2023-09-30T01:38:02.385 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2023-09-30T01:38:02.385 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 1 2023-09-30T01:38:02.385 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.noname-a 2023-09-30T01:38:02.385 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.noname-f 2023-09-30T01:38:02.385 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.noname-b 2023-09-30T01:38:02.385 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.noname-g 2023-09-30T01:38:02.385 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.noname-c 2023-09-30T01:38:02.385 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.noname-h 2023-09-30T01:38:02.385 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.noname-d 2023-09-30T01:38:02.385 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.noname-i 2023-09-30T01:38:02.386 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.noname-e 2023-09-30T01:38:02.386 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:38:02.386 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:02.363+0000 7fa8b759e700 1 Processor -- start 2023-09-30T01:38:02.386 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:02.367+0000 7fa8b759e700 1 -- start start 2023-09-30T01:38:02.386 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:02.367+0000 7fa8b759e700 10 monclient: init 2023-09-30T01:38:02.386 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:02.367+0000 7fa8b759e700 10 monclient: _reopen_session rank -1 2023-09-30T01:38:02.386 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:02.367+0000 7fa8b759e700 10 monclient: _add_conns ranks=[3,0,1,7,8,2,5,4,6] 2023-09-30T01:38:02.386 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:02.367+0000 7fa8b759e700 1 --2- >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7fa8a40042e0 0x7fa8b813e850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:38:02.386 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:02.367+0000 7fa8b759e700 10 monclient(hunting): picked mon.noname-g con 0x7fa8a40042e0 addr [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] 2023-09-30T01:38:02.386 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:02.367+0000 7fa8b759e700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa8b81220d0 0x7fa8b813df00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:38:02.386 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:02.367+0000 7fa8b759e700 10 monclient(hunting): picked mon.noname-a con 0x7fa8b81220d0 addr [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] 2023-09-30T01:38:02.386 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:02.367+0000 7fa8b759e700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa8b8120b00 0x7fa8b8141dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:38:02.386 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:02.367+0000 7fa8b759e700 10 monclient(hunting): picked mon.noname-f con 0x7fa8b8120b00 addr [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] 2023-09-30T01:38:02.387 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:02.367+0000 7fa8b759e700 10 monclient(hunting): start opening mon connection 2023-09-30T01:38:02.387 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:02.367+0000 7fa8b759e700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8b8123280 con 0x7fa8b81220d0 2023-09-30T01:38:02.387 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:02.367+0000 7fa8b759e700 10 monclient(hunting): start opening mon connection 2023-09-30T01:38:02.387 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:02.367+0000 7fa8b759e700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8b8121570 con 0x7fa8b8120b00 2023-09-30T01:38:02.387 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:02.367+0000 7fa8b759e700 10 monclient(hunting): start opening mon connection 2023-09-30T01:38:02.387 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:02.367+0000 7fa8b759e700 1 -- --> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8b813f3b0 con 0x7fa8a40042e0 2023-09-30T01:38:02.387 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:02.367+0000 7fa8b759e700 10 monclient(hunting): _renew_subs 2023-09-30T01:38:02.387 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:02.367+0000 7fa8b759e700 10 monclient(hunting): authenticate will time out at 1154.405386s 2023-09-30T01:38:02.387 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:02.367+0000 7fa8add9b700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa8b81220d0 0x7fa8b813df00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:38:02.387 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:02.367+0000 7fa8add9b700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa8b81220d0 0x7fa8b813df00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:38714/0 (socket says 172.21.15.88:38714) 2023-09-30T01:38:02.387 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:02.367+0000 7fa8add9b700 1 -- 172.21.15.88:0/2028952478 learned_addr learned my addr 172.21.15.88:0/2028952478 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-09-30T01:38:02.387 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:02.367+0000 7fa8add9b700 10 monclient(hunting): get_auth_request con 0x7fa8b81220d0 auth_method 0 2023-09-30T01:38:02.388 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:02.367+0000 7fa8add9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:38:02.388 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:02.367+0000 7fa8add9b700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:38:02.388 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:02.367+0000 7fa8add9b700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:38:02.388 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:02.367+0000 7fa8add9b700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:38:02.388 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:02.367+0000 7fa8add9b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:38:02.388 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:02.367+0000 7fa8b659c700 1 --2- 172.21.15.88:0/2028952478 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7fa8a40042e0 0x7fa8b813e850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:38:02.388 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:02.367+0000 7fa8add9b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:38:02.388 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:02.367+0000 7fa8b659c700 10 monclient(hunting): get_auth_request con 0x7fa8a40042e0 auth_method 0 2023-09-30T01:38:02.388 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:02.367+0000 7fa8b659c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:38:02.388 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:02.367+0000 7fa8b659c700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:38:02.388 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:02.367+0000 7fa8b659c700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:38:02.388 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:02.367+0000 7fa8b6d9d700 1 --2- 172.21.15.88:0/2028952478 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa8b8120b00 0x7fa8b8141dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:38:02.388 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:02.367+0000 7fa8b659c700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:38:02.389 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:02.367+0000 7fa8b659c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:38:02.389 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:02.367+0000 7fa8b659c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:38:02.389 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:02.367+0000 7fa8add9b700 10 monclient(hunting): handle_auth_done global_id 5166 payload 824 2023-09-30T01:38:02.389 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:02.367+0000 7fa8add9b700 1 -- 172.21.15.88:0/2028952478 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7fa8a40042e0 msgr2=0x7fa8b813e850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:38:02.389 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:02.367+0000 7fa8add9b700 1 --2- 172.21.15.88:0/2028952478 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7fa8a40042e0 0x7fa8b813e850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:38:02.389 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:02.367+0000 7fa8add9b700 1 -- 172.21.15.88:0/2028952478 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa8b8120b00 msgr2=0x7fa8b8141dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:38:02.389 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:02.367+0000 7fa8add9b700 1 --2- 172.21.15.88:0/2028952478 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa8b8120b00 0x7fa8b8141dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:38:02.389 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:02.367+0000 7fa8add9b700 10 monclient: _finish_hunting 0 2023-09-30T01:38:02.389 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:02.367+0000 7fa8add9b700 1 monclient: found mon.noname-a 2023-09-30T01:38:02.389 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:02.367+0000 7fa8add9b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:38:02.389 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:02.367+0000 7fa8add9b700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.88:3300/0 2023-09-30T01:38:02.389 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:02.367+0000 7fa8add9b700 1 -- 172.21.15.88:0/2028952478 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8b8122fa0 con 0x7fa8b81220d0 2023-09-30T01:38:02.389 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:02.367+0000 7fa8b6d9d700 10 monclient: get_auth_request con 0x7fa8b8120b00 auth_method 0 2023-09-30T01:38:02.390 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:02.367+0000 7fa8b6d9d700 1 --2- 172.21.15.88:0/2028952478 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa8b8120b00 0x7fa8b8141dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-09-30T01:38:02.390 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:02.367+0000 7fa8add9b700 1 --2- 172.21.15.88:0/2028952478 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa8b81220d0 0x7fa8b813df00 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fa8a8007390 tx=0x7fa8b8098ef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:38:02.390 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:02.367+0000 7fa8aeffd700 1 -- 172.21.15.88:0/2028952478 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7fa8a8018030 con 0x7fa8b81220d0 2023-09-30T01:38:02.390 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:02.367+0000 7fa8aeffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:38:02.390 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:02.367+0000 7fa8aeffd700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T01:38:02.390 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:02.367+0000 7fa8aeffd700 10 monclient: dump: 2023-09-30T01:38:02.390 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:38:02.390 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:38:02.390 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:38:02.390 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:38:02.390 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:38:02.390 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:38:02.391 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:38:02.391 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:38:02.391 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:38:02.391 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:38:02.391 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:38:02.391 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:38:02.391 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:38:02.391 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:38:02.391 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:38:02.391 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:38:02.391 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:02.367+0000 7fa8aeffd700 10 monclient: _finish_auth 0 2023-09-30T01:38:02.391 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:02.367+0000 7fa8aeffd700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:38:02.391 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:02.367+0000 7fa8b759e700 5 monclient: authenticate success, global_id 5166 2023-09-30T01:38:02.392 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:02.367+0000 7fa8b759e700 10 monclient: _renew_subs 2023-09-30T01:38:02.392 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:02.367+0000 7fa8aeffd700 1 -- 172.21.15.88:0/2028952478 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa8a8007750 con 0x7fa8b81220d0 2023-09-30T01:38:02.392 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:02.367+0000 7fa8b759e700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.88:3300/0 2023-09-30T01:38:02.392 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:02.367+0000 7fa8b759e700 1 -- 172.21.15.88:0/2028952478 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa8b8123560 con 0x7fa8b81220d0 2023-09-30T01:38:02.392 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:02.367+0000 7fa8aeffd700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:38:02.392 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:02.367+0000 7fa8aeffd700 1 -- 172.21.15.88:0/2028952478 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7fa8a800fcf0 con 0x7fa8b81220d0 2023-09-30T01:38:02.392 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:02.367+0000 7fa8aeffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:38:02.392 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:02.367+0000 7fa8aeffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:38:02.392 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:02.367+0000 7fa8b559a700 4 set_mon_vals no callback set 2023-09-30T01:38:02.392 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:02.367+0000 7fa8aeffd700 10 monclient: dump: 2023-09-30T01:38:02.392 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:38:02.392 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:38:02.392 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:38:02.393 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:38:02.393 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:38:02.393 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:38:02.393 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:38:02.393 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:38:02.393 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:38:02.393 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:38:02.393 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:38:02.393 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:38:02.393 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:38:02.393 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:38:02.393 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:38:02.394 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:38:02.394 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:02.367+0000 7fa8b759e700 10 monclient: _renew_subs 2023-09-30T01:38:02.394 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:02.367+0000 7fa8b759e700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.88:3300/0 2023-09-30T01:38:02.394 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:02.367+0000 7fa8b759e700 1 -- 172.21.15.88:0/2028952478 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa8b813e5e0 con 0x7fa8b81220d0 2023-09-30T01:38:02.394 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:02.367+0000 7fa8aeffd700 1 -- 172.21.15.88:0/2028952478 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 40144+0+0 (secure 0 0 0) 0x7fa8a801b070 con 0x7fa8b81220d0 2023-09-30T01:38:02.394 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:02.371+0000 7fa8aeffd700 1 --2- 172.21.15.88:0/2028952478 >> v2:172.21.15.107:6812/29926 conn(0x7fa898033390 0x7fa898035840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:38:02.394 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:02.371+0000 7fa8aeffd700 1 -- 172.21.15.88:0/2028952478 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 3502+0+0 (secure 0 0 0) 0x7fa8a8019070 con 0x7fa8b81220d0 2023-09-30T01:38:02.394 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:02.371+0000 7fa8b659c700 1 --2- 172.21.15.88:0/2028952478 >> v2:172.21.15.107:6812/29926 conn(0x7fa898033390 0x7fa898035840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:38:02.394 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:02.371+0000 7fa8b659c700 10 monclient: get_auth_request con 0x7fa898033390 auth_method 0 2023-09-30T01:38:02.394 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:02.371+0000 7fa8b759e700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:38:02.394 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:02.371+0000 7fa8b759e700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:38:02.394 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:02.371+0000 7fa8b759e700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.88:3300/0 2023-09-30T01:38:02.395 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:02.371+0000 7fa8b759e700 1 -- 172.21.15.88:0/2028952478 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa8b80710d0 con 0x7fa8b81220d0 2023-09-30T01:38:02.395 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:02.371+0000 7fa8b659c700 1 --2- 172.21.15.88:0/2028952478 >> v2:172.21.15.107:6812/29926 conn(0x7fa898033390 0x7fa898035840 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fa8a000bfd0 tx=0x7fa8a001b000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:38:02.395 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:02.379+0000 7fa8aeffd700 1 -- 172.21.15.88:0/2028952478 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fa8a800fb00 con 0x7fa8b81220d0 2023-09-30T01:38:02.395 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:02.379+0000 7fa8aeffd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:38:02.395 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:02.379+0000 7fa8aeffd700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:38:02.570 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:02.567+0000 7fa8b759e700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T01:38:02.570 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:02.567+0000 7fa8b759e700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:38:02.570 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:02.567+0000 7fa8b759e700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.88:3300/0 2023-09-30T01:38:02.570 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:02.567+0000 7fa8b759e700 1 -- 172.21.15.88:0/2028952478 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fa8b8067b60 con 0x7fa8b81220d0 2023-09-30T01:38:02.571 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:02.567+0000 7fa8aeffd700 1 -- 172.21.15.88:0/2028952478 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7fa8a801c370 con 0x7fa8b81220d0 2023-09-30T01:38:02.571 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:02.567+0000 7fa8aeffd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:38:02.571 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:02.567+0000 7fa8aeffd700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:38:02.571 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-09-30T01:38:02.571 INFO:teuthology.orchestra.run.smithi088.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:38:02.573 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:02.571+0000 7fa8b759e700 1 -- 172.21.15.88:0/2028952478 >> v2:172.21.15.107:6812/29926 conn(0x7fa898033390 msgr2=0x7fa898035840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:38:02.573 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:02.571+0000 7fa8b759e700 1 --2- 172.21.15.88:0/2028952478 >> v2:172.21.15.107:6812/29926 conn(0x7fa898033390 0x7fa898035840 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fa8a000bfd0 tx=0x7fa8a001b000 comp rx=0 tx=0).stop 2023-09-30T01:38:02.573 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:02.571+0000 7fa8b759e700 10 monclient: shutdown 2023-09-30T01:38:02.573 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:02.571+0000 7fa8b759e700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:38:02.573 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:02.571+0000 7fa8b759e700 1 -- 172.21.15.88:0/2028952478 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa8b81220d0 msgr2=0x7fa8b813df00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:38:02.574 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:02.571+0000 7fa8b759e700 1 --2- 172.21.15.88:0/2028952478 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa8b81220d0 0x7fa8b813df00 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fa8a8007390 tx=0x7fa8b8098ef0 comp rx=0 tx=0).stop 2023-09-30T01:38:02.574 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:02.571+0000 7fa8b759e700 1 -- 172.21.15.88:0/2028952478 shutdown_connections 2023-09-30T01:38:02.574 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:02.571+0000 7fa8b759e700 1 --2- 172.21.15.88:0/2028952478 >> v2:172.21.15.107:6812/29926 conn(0x7fa898033390 0x7fa898035840 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:38:02.574 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:02.571+0000 7fa8b759e700 1 --2- 172.21.15.88:0/2028952478 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa8b8120b00 0x7fa8b8141dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:38:02.574 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:02.571+0000 7fa8b759e700 1 --2- 172.21.15.88:0/2028952478 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa8b81220d0 0x7fa8b813df00 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:38:02.574 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:02.571+0000 7fa8b759e700 1 --2- 172.21.15.88:0/2028952478 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7fa8a40042e0 0x7fa8b813e850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:38:02.574 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:02.571+0000 7fa8b759e700 1 -- 172.21.15.88:0/2028952478 >> 172.21.15.88:0/2028952478 conn(0x7fa8b806e5e0 msgr2=0x7fa8b80a1bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:38:02.574 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:02.571+0000 7fa8b759e700 1 -- 172.21.15.88:0/2028952478 shutdown_connections 2023-09-30T01:38:02.574 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:02.571+0000 7fa8b759e700 1 -- 172.21.15.88:0/2028952478 wait complete. 2023-09-30T01:38:05.585 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:38:05.689 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:05.687+0000 7f0337298700 10 monclient: get_monmap_and_config 2023-09-30T01:38:05.690 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:05.687+0000 7f0337298700 10 monclient: build_initial_monmap 2023-09-30T01:38:05.690 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:05.687+0000 7f0337298700 10 monclient: monmap: 2023-09-30T01:38:05.690 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2023-09-30T01:38:05.690 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:38:05.690 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:38:05.689436+0000 2023-09-30T01:38:05.690 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:38:05.689436+0000 2023-09-30T01:38:05.690 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2023-09-30T01:38:05.690 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:38:05.691 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a-legacy 2023-09-30T01:38:05.691 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.107:6789/0 mon.noname-f-legacy 2023-09-30T01:38:05.691 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.noname-b 2023-09-30T01:38:05.691 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.noname-g 2023-09-30T01:38:05.691 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.noname-c 2023-09-30T01:38:05.691 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.noname-h 2023-09-30T01:38:05.691 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.noname-d 2023-09-30T01:38:05.691 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.noname-i 2023-09-30T01:38:05.691 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.noname-e 2023-09-30T01:38:05.691 INFO:teuthology.orchestra.run.smithi088.stderr:9: v2:172.21.15.88:3300/0 mon.noname-a 2023-09-30T01:38:05.692 INFO:teuthology.orchestra.run.smithi088.stderr:10: v2:172.21.15.107:3300/0 mon.noname-f 2023-09-30T01:38:05.692 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:38:05.700 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:05.687+0000 7f0337298700 1 Processor -- start 2023-09-30T01:38:05.701 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:05.687+0000 7f0337298700 1 -- start start 2023-09-30T01:38:05.701 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:05.687+0000 7f0337298700 10 monclient: init 2023-09-30T01:38:05.701 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:05.687+0000 7f0337298700 10 monclient: _reopen_session rank -1 2023-09-30T01:38:05.701 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:05.687+0000 7f0337298700 10 monclient: _add_conns ranks=[4,1,2,9,10,0,8,6,3,7,5] 2023-09-30T01:38:05.701 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:05.687+0000 7f0337298700 1 --2- >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f03300966a0 0x7f0330096a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:38:05.701 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:05.687+0000 7f0337298700 10 monclient(hunting): picked mon.noname-c con 0x7f03300966a0 addr [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] 2023-09-30T01:38:05.701 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:05.687+0000 7f0337298700 10 monclient(hunting): picked mon.noname-f-legacy con 0x7f0330097cd0 addr v1:172.21.15.107:6789/0 2023-09-30T01:38:05.701 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:05.687+0000 7f0337298700 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f03300980a0 0x7f033009f490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:38:05.701 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:05.687+0000 7f0337298700 10 monclient(hunting): picked mon.noname-b con 0x7f03300980a0 addr [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] 2023-09-30T01:38:05.702 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:05.687+0000 7f0337298700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f033004bb20 con 0x7f0330097cd0 2023-09-30T01:38:05.702 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:05.687+0000 7f0337298700 10 monclient(hunting): start opening mon connection 2023-09-30T01:38:05.702 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:05.687+0000 7f0337298700 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0330097920 con 0x7f03300980a0 2023-09-30T01:38:05.702 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:05.687+0000 7f0337298700 10 monclient(hunting): start opening mon connection 2023-09-30T01:38:05.702 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:05.687+0000 7f0337298700 1 -- --> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] -- mon_getmap magic: 0 v1 -- 0x7f033009a340 con 0x7f03300966a0 2023-09-30T01:38:05.702 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:05.687+0000 7f0337298700 10 monclient(hunting): _renew_subs 2023-09-30T01:38:05.702 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:05.687+0000 7f0337298700 10 monclient(hunting): authenticate will time out at 1157.727428s 2023-09-30T01:38:05.702 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:05.687+0000 7f0336296700 1 --2- >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f03300966a0 0x7f0330096a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:38:05.702 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:05.687+0000 7f0336296700 1 --2- >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f03300966a0 0x7f0330096a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3302/0 says I am v2:172.21.15.88:38870/0 (socket says 172.21.15.88:38870) 2023-09-30T01:38:05.702 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:05.687+0000 7f0336a97700 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f03300980a0 0x7f033009f490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:38:05.702 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:05.687+0000 7f0336296700 1 -- 172.21.15.88:0/256875644 learned_addr learned my addr 172.21.15.88:0/256875644 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-09-30T01:38:05.702 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:05.687+0000 7f0336296700 10 monclient(hunting): get_auth_request con 0x7f03300966a0 auth_method 0 2023-09-30T01:38:05.703 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:05.687+0000 7f0336296700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:38:05.703 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:05.687+0000 7f0336296700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:38:05.703 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:05.687+0000 7f0336296700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:38:05.703 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:05.687+0000 7f0336a97700 10 monclient(hunting): get_auth_request con 0x7f03300980a0 auth_method 0 2023-09-30T01:38:05.703 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:05.687+0000 7f0336a97700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:38:05.703 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:05.687+0000 7f0336a97700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:38:05.703 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:05.687+0000 7f0336a97700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:38:05.703 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:05.687+0000 7f0336296700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:38:05.703 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:05.687+0000 7f0336296700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:38:05.703 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:05.687+0000 7f0336296700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:38:05.703 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:05.687+0000 7f0336a97700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:38:05.704 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:05.687+0000 7f0336a97700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:38:05.704 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:05.687+0000 7f0336a97700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:38:05.704 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:05.691+0000 7f0336296700 10 monclient(hunting): handle_auth_done global_id 4882 payload 274 2023-09-30T01:38:05.704 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:05.691+0000 7f0336296700 1 -- 172.21.15.88:0/256875644 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f03300980a0 msgr2=0x7f033009f490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:38:05.704 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:05.691+0000 7f0336296700 1 --2- 172.21.15.88:0/256875644 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f03300980a0 0x7f033009f490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:38:05.704 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:05.691+0000 7f0336296700 1 -- 172.21.15.88:0/256875644 >> v1:172.21.15.107:6789/0 conn(0x7f0330097cd0 legacy=0x7f033009ed80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:38:05.704 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:05.691+0000 7f0336296700 10 monclient: _finish_hunting 0 2023-09-30T01:38:05.704 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:05.691+0000 7f0336296700 1 monclient: found mon.noname-c 2023-09-30T01:38:05.704 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:05.691+0000 7f0336296700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:38:05.704 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:05.691+0000 7f0336296700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.88:3302/0 2023-09-30T01:38:05.704 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:05.691+0000 7f0336296700 1 -- 172.21.15.88:0/256875644 --> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03300975d0 con 0x7f03300966a0 2023-09-30T01:38:05.704 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:05.691+0000 7f0336a97700 1 --2- 172.21.15.88:0/256875644 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f03300980a0 0x7f033009f490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:38:05.705 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:05.691+0000 7f0336296700 1 --2- 172.21.15.88:0/256875644 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f03300966a0 0x7f0330096a70 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f032c0099c0 tx=0x7f032c009220 comp rx=0 tx=0).ready entity=mon.4 client_cookie=bccc22c908d1b7f3 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:38:05.705 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:05.691+0000 7f0335294700 1 -- 172.21.15.88:0/256875644 <== mon.4 v2:172.21.15.88:3302/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f032c005c10 con 0x7f03300966a0 2023-09-30T01:38:05.705 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:05.691+0000 7f0335294700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:38:05.705 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:05.691+0000 7f0335294700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T01:38:05.705 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:05.691+0000 7f0335294700 10 monclient: dump: 2023-09-30T01:38:05.705 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:38:05.705 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:38:05.705 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:38:05.705 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:38:05.705 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:38:05.705 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:38:05.705 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:38:05.706 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:38:05.706 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:38:05.706 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:38:05.706 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:38:05.706 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:38:05.706 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:38:05.706 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:38:05.706 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:38:05.706 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:38:05.706 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:05.691+0000 7f0335294700 10 monclient: _finish_auth 0 2023-09-30T01:38:05.706 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:05.691+0000 7f0335294700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:38:05.706 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:05.691+0000 7f0337298700 5 monclient: authenticate success, global_id 4882 2023-09-30T01:38:05.707 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:05.691+0000 7f0337298700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:38:05.707 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:05.691+0000 7f0335294700 1 -- 172.21.15.88:0/256875644 <== mon.4 v2:172.21.15.88:3302/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f032c005d90 con 0x7f03300966a0 2023-09-30T01:38:05.707 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:05.691+0000 7f0335294700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:38:05.707 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:05.691+0000 7f0337298700 10 monclient: get_monmap_and_config success 2023-09-30T01:38:05.707 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:05.691+0000 7f0337298700 4 set_mon_vals no callback set 2023-09-30T01:38:05.707 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:05.691+0000 7f0335294700 1 -- 172.21.15.88:0/256875644 <== mon.4 v2:172.21.15.88:3302/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f032c006030 con 0x7f03300966a0 2023-09-30T01:38:05.707 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:05.691+0000 7f0335294700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:38:05.707 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:05.691+0000 7f0335294700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T01:38:05.707 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:05.691+0000 7f0335294700 10 monclient: dump: 2023-09-30T01:38:05.707 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:38:05.707 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:38:05.708 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:38:05.708 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:38:05.708 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:38:05.708 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:38:05.708 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:38:05.708 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:38:05.708 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:38:05.708 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:38:05.708 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:38:05.708 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:38:05.708 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:38:05.708 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:38:05.708 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:38:05.709 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:38:05.709 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:05.691+0000 7f0337298700 10 monclient: shutdown 2023-09-30T01:38:05.709 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:05.691+0000 7f0337298700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:38:05.709 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:05.691+0000 7f0337298700 1 -- 172.21.15.88:0/256875644 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f03300966a0 msgr2=0x7f0330096a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:38:05.709 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:05.691+0000 7f0337298700 1 --2- 172.21.15.88:0/256875644 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f03300966a0 0x7f0330096a70 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f032c0099c0 tx=0x7f032c009220 comp rx=0 tx=0).stop 2023-09-30T01:38:05.709 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:05.691+0000 7f0337298700 1 -- 172.21.15.88:0/256875644 shutdown_connections 2023-09-30T01:38:05.709 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:05.691+0000 7f0337298700 1 --2- 172.21.15.88:0/256875644 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f03300980a0 0x7f033009f490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:38:05.709 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:05.691+0000 7f0337298700 1 --2- 172.21.15.88:0/256875644 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f03300966a0 0x7f0330096a70 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:38:05.709 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:05.691+0000 7f0337298700 1 -- 172.21.15.88:0/256875644 >> 172.21.15.88:0/256875644 conn(0x7f03300a0910 msgr2=0x7f03300a33c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:38:05.709 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:05.691+0000 7f0337298700 1 -- 172.21.15.88:0/256875644 shutdown_connections 2023-09-30T01:38:05.709 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:05.691+0000 7f0337298700 1 -- 172.21.15.88:0/256875644 wait complete. 2023-09-30T01:38:05.709 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:05.691+0000 7f0337298700 10 monclient: build_initial_monmap 2023-09-30T01:38:05.710 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:05.691+0000 7f0337298700 10 monclient: monmap: 2023-09-30T01:38:05.710 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2023-09-30T01:38:05.710 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:38:05.710 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 0.000000 2023-09-30T01:38:05.710 INFO:teuthology.orchestra.run.smithi088.stderr:created 0.000000 2023-09-30T01:38:05.710 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2023-09-30T01:38:05.710 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 1 2023-09-30T01:38:05.710 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.noname-a 2023-09-30T01:38:05.710 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.noname-f 2023-09-30T01:38:05.710 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.noname-b 2023-09-30T01:38:05.710 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.noname-g 2023-09-30T01:38:05.710 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.noname-c 2023-09-30T01:38:05.711 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.noname-h 2023-09-30T01:38:05.711 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.noname-d 2023-09-30T01:38:05.711 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.noname-i 2023-09-30T01:38:05.711 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.noname-e 2023-09-30T01:38:05.711 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:38:05.711 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:05.691+0000 7f0337298700 1 Processor -- start 2023-09-30T01:38:05.711 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:05.691+0000 7f0337298700 1 -- start start 2023-09-30T01:38:05.711 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:05.691+0000 7f0337298700 10 monclient: init 2023-09-30T01:38:05.711 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:05.691+0000 7f0337298700 10 monclient: _reopen_session rank -1 2023-09-30T01:38:05.711 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:05.691+0000 7f0337298700 10 monclient: _add_conns ranks=[4,7,1,2,0,5,6,8,3] 2023-09-30T01:38:05.711 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:05.691+0000 7f0337298700 1 --2- >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f03300966a0 0x7f03301365d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:38:05.711 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:05.691+0000 7f0337298700 10 monclient(hunting): picked mon.noname-c con 0x7f03300966a0 addr [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] 2023-09-30T01:38:05.712 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:05.691+0000 7f0337298700 1 --2- >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7f0330097cd0 0x7f0330136b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:38:05.712 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:05.691+0000 7f0337298700 10 monclient(hunting): picked mon.noname-i con 0x7f0330097cd0 addr [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] 2023-09-30T01:38:05.712 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:05.695+0000 7f0337298700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f03300980a0 0x7f0330137050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:38:05.712 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:05.695+0000 7f0337298700 10 monclient(hunting): picked mon.noname-f con 0x7f03300980a0 addr [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] 2023-09-30T01:38:05.712 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:05.695+0000 7f0337298700 10 monclient(hunting): start opening mon connection 2023-09-30T01:38:05.712 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:05.695+0000 7f0337298700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0330090390 con 0x7f03300980a0 2023-09-30T01:38:05.712 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:05.695+0000 7f0337298700 10 monclient(hunting): start opening mon connection 2023-09-30T01:38:05.712 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:05.695+0000 7f0337298700 1 -- --> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] -- mon_getmap magic: 0 v1 -- 0x7f03300996c0 con 0x7f03300966a0 2023-09-30T01:38:05.712 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:05.695+0000 7f0337298700 10 monclient(hunting): start opening mon connection 2023-09-30T01:38:05.712 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:05.695+0000 7f0337298700 1 -- --> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] -- mon_getmap magic: 0 v1 -- 0x7f0330063630 con 0x7f0330097cd0 2023-09-30T01:38:05.712 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:05.695+0000 7f0337298700 10 monclient(hunting): _renew_subs 2023-09-30T01:38:05.713 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:05.695+0000 7f0337298700 10 monclient(hunting): authenticate will time out at 1157.732124s 2023-09-30T01:38:05.713 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:05.695+0000 7f0336296700 1 --2- >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f03300966a0 0x7f03301365d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:38:05.713 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:05.695+0000 7f0336296700 1 --2- >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f03300966a0 0x7f03301365d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3302/0 says I am v2:172.21.15.88:38880/0 (socket says 172.21.15.88:38880) 2023-09-30T01:38:05.713 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:05.695+0000 7f0336296700 1 -- 172.21.15.88:0/413655672 learned_addr learned my addr 172.21.15.88:0/413655672 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-09-30T01:38:05.713 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:05.695+0000 7f0336296700 10 monclient(hunting): get_auth_request con 0x7f03300966a0 auth_method 0 2023-09-30T01:38:05.713 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:05.695+0000 7f0336296700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:38:05.713 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:05.695+0000 7f0336296700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:38:05.713 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:05.695+0000 7f0336296700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:38:05.713 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:05.695+0000 7f0335a95700 1 --2- 172.21.15.88:0/413655672 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7f0330097cd0 0x7f0330136b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:38:05.713 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:05.695+0000 7f0336296700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:38:05.713 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:05.695+0000 7f0336296700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:38:05.714 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:05.695+0000 7f0336a97700 1 --2- 172.21.15.88:0/413655672 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f03300980a0 0x7f0330137050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:38:05.714 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:05.695+0000 7f0336296700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:38:05.714 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:05.695+0000 7f0335a95700 10 monclient(hunting): get_auth_request con 0x7f0330097cd0 auth_method 0 2023-09-30T01:38:05.714 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:05.695+0000 7f0335a95700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:38:05.714 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:05.695+0000 7f0335a95700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:38:05.714 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:05.695+0000 7f0335a95700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:38:05.714 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:05.695+0000 7f0336a97700 10 monclient(hunting): get_auth_request con 0x7f03300980a0 auth_method 0 2023-09-30T01:38:05.714 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:05.695+0000 7f0336a97700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:38:05.714 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:05.695+0000 7f0336a97700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:38:05.714 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:05.695+0000 7f0336a97700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:38:05.714 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:05.695+0000 7f0335a95700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:38:05.714 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:05.695+0000 7f0335a95700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:38:05.715 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:05.695+0000 7f0335a95700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:38:05.715 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:05.695+0000 7f0336a97700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:38:05.715 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:05.695+0000 7f0336a97700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:38:05.715 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:05.695+0000 7f0336a97700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:38:05.715 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:05.695+0000 7f0336296700 10 monclient(hunting): handle_auth_done global_id 4891 payload 824 2023-09-30T01:38:05.715 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:05.695+0000 7f0336296700 1 -- 172.21.15.88:0/413655672 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7f0330097cd0 msgr2=0x7f0330136b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:38:05.715 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:05.695+0000 7f0336296700 1 --2- 172.21.15.88:0/413655672 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7f0330097cd0 0x7f0330136b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:38:05.715 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:05.695+0000 7f0336296700 1 -- 172.21.15.88:0/413655672 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f03300980a0 msgr2=0x7f0330137050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:38:05.715 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:05.695+0000 7f0336296700 1 --2- 172.21.15.88:0/413655672 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f03300980a0 0x7f0330137050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:38:05.715 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:05.695+0000 7f0336296700 10 monclient: _finish_hunting 0 2023-09-30T01:38:05.715 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:05.695+0000 7f0336296700 1 monclient: found mon.noname-c 2023-09-30T01:38:05.715 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:05.695+0000 7f0336296700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:38:05.716 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:05.695+0000 7f0336296700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.88:3302/0 2023-09-30T01:38:05.716 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:05.695+0000 7f0336296700 1 -- 172.21.15.88:0/413655672 --> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0328009a20 con 0x7f03300966a0 2023-09-30T01:38:05.716 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:05.695+0000 7f0336296700 1 --2- 172.21.15.88:0/413655672 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f03300966a0 0x7f03301365d0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f0330063140 tx=0x7f032c016000 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:38:05.716 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:05.695+0000 7f03267fc700 1 -- 172.21.15.88:0/413655672 <== mon.4 v2:172.21.15.88:3302/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f032c005c10 con 0x7f03300966a0 2023-09-30T01:38:05.716 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:05.695+0000 7f03267fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:38:05.716 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:05.695+0000 7f03267fc700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T01:38:05.716 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:05.695+0000 7f03267fc700 10 monclient: dump: 2023-09-30T01:38:05.716 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:38:05.716 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:38:05.716 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:38:05.716 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:38:05.716 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:38:05.717 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:38:05.717 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:38:05.717 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:38:05.717 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:38:05.717 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:38:05.717 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:38:05.717 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:38:05.717 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:38:05.717 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:38:05.717 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:38:05.718 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:38:05.718 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:05.695+0000 7f03267fc700 10 monclient: _finish_auth 0 2023-09-30T01:38:05.718 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:05.695+0000 7f03267fc700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:38:05.718 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:05.695+0000 7f03267fc700 1 -- 172.21.15.88:0/413655672 <== mon.4 v2:172.21.15.88:3302/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f032c009220 con 0x7f03300966a0 2023-09-30T01:38:05.718 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:05.695+0000 7f0337298700 5 monclient: authenticate success, global_id 4891 2023-09-30T01:38:05.718 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:05.695+0000 7f0337298700 10 monclient: _renew_subs 2023-09-30T01:38:05.718 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:05.695+0000 7f0337298700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.88:3302/0 2023-09-30T01:38:05.718 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:05.695+0000 7f0337298700 1 -- 172.21.15.88:0/413655672 --> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f032c009480 con 0x7f03300966a0 2023-09-30T01:38:05.718 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:05.695+0000 7f03267fc700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:38:05.718 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:05.695+0000 7f03267fc700 1 -- 172.21.15.88:0/413655672 <== mon.4 v2:172.21.15.88:3302/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f032c0066b0 con 0x7f03300966a0 2023-09-30T01:38:05.719 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:05.695+0000 7f03267fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:38:05.719 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:05.695+0000 7f03267fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T01:38:05.719 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:05.695+0000 7f03267fc700 10 monclient: dump: 2023-09-30T01:38:05.719 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:38:05.719 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:38:05.719 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:38:05.719 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:38:05.719 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:38:05.719 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:38:05.719 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:38:05.719 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:38:05.719 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:38:05.720 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:38:05.720 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:38:05.720 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:38:05.720 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:38:05.720 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:38:05.720 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:38:05.720 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:38:05.720 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:05.695+0000 7f0337298700 10 monclient: _renew_subs 2023-09-30T01:38:05.720 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:05.695+0000 7f0327fff700 4 set_mon_vals no callback set 2023-09-30T01:38:05.720 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:05.695+0000 7f0337298700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.88:3302/0 2023-09-30T01:38:05.720 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:05.695+0000 7f0337298700 1 -- 172.21.15.88:0/413655672 --> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0330091620 con 0x7f03300966a0 2023-09-30T01:38:05.720 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:05.695+0000 7f0337298700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:38:05.721 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:05.695+0000 7f0337298700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:38:05.721 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:05.695+0000 7f0337298700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.88:3302/0 2023-09-30T01:38:05.721 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:05.695+0000 7f0337298700 1 -- 172.21.15.88:0/413655672 --> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0330067520 con 0x7f03300966a0 2023-09-30T01:38:05.721 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:05.695+0000 7f03267fc700 1 -- 172.21.15.88:0/413655672 <== mon.4 v2:172.21.15.88:3302/0 4 ==== mgrmap(e 5) v1 ==== 40144+0+0 (secure 0 0 0) 0x7f032c022070 con 0x7f03300966a0 2023-09-30T01:38:05.721 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:05.699+0000 7f03267fc700 1 --2- 172.21.15.88:0/413655672 >> v2:172.21.15.107:6812/29926 conn(0x7f0314033360 0x7f0314035810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:38:05.721 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:05.699+0000 7f03267fc700 1 -- 172.21.15.88:0/413655672 <== mon.4 v2:172.21.15.88:3302/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 3502+0+0 (secure 0 0 0) 0x7f032c0451c0 con 0x7f03300966a0 2023-09-30T01:38:05.721 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:05.699+0000 7f0335a95700 1 --2- 172.21.15.88:0/413655672 >> v2:172.21.15.107:6812/29926 conn(0x7f0314033360 0x7f0314035810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:38:05.721 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:05.699+0000 7f0335a95700 10 monclient: get_auth_request con 0x7f0314033360 auth_method 0 2023-09-30T01:38:05.721 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:05.699+0000 7f0335a95700 1 --2- 172.21.15.88:0/413655672 >> v2:172.21.15.107:6812/29926 conn(0x7f0314033360 0x7f0314035810 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f0320003aa0 tx=0x7f0320000bc0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:38:05.722 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:05.707+0000 7f03267fc700 1 -- 172.21.15.88:0/413655672 <== mon.4 v2:172.21.15.88:3302/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f032c023370 con 0x7f03300966a0 2023-09-30T01:38:05.722 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:05.707+0000 7f03267fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:38:05.722 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:05.707+0000 7f03267fc700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:38:05.900 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:05.899+0000 7f0337298700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T01:38:05.900 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:05.899+0000 7f0337298700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:38:05.901 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:05.899+0000 7f0337298700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.88:3302/0 2023-09-30T01:38:05.901 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:05.899+0000 7f0337298700 1 -- 172.21.15.88:0/413655672 --> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f0330071960 con 0x7f03300966a0 2023-09-30T01:38:05.901 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:05.899+0000 7f03267fc700 1 -- 172.21.15.88:0/413655672 <== mon.4 v2:172.21.15.88:3302/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f032c023e90 con 0x7f03300966a0 2023-09-30T01:38:05.902 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:05.899+0000 7f03267fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:38:05.902 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:05.899+0000 7f03267fc700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:38:05.902 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-09-30T01:38:05.902 INFO:teuthology.orchestra.run.smithi088.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:38:05.903 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:05.899+0000 7f0337298700 1 -- 172.21.15.88:0/413655672 >> v2:172.21.15.107:6812/29926 conn(0x7f0314033360 msgr2=0x7f0314035810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:38:05.903 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:05.899+0000 7f0337298700 1 --2- 172.21.15.88:0/413655672 >> v2:172.21.15.107:6812/29926 conn(0x7f0314033360 0x7f0314035810 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f0320003aa0 tx=0x7f0320000bc0 comp rx=0 tx=0).stop 2023-09-30T01:38:05.903 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:05.899+0000 7f0337298700 10 monclient: shutdown 2023-09-30T01:38:05.904 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:05.899+0000 7f0337298700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:38:05.904 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:05.899+0000 7f0337298700 1 -- 172.21.15.88:0/413655672 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f03300966a0 msgr2=0x7f03301365d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:38:05.904 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:05.899+0000 7f0337298700 1 --2- 172.21.15.88:0/413655672 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f03300966a0 0x7f03301365d0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f0330063140 tx=0x7f032c016000 comp rx=0 tx=0).stop 2023-09-30T01:38:05.904 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:05.899+0000 7f0337298700 1 -- 172.21.15.88:0/413655672 shutdown_connections 2023-09-30T01:38:05.904 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:05.899+0000 7f0337298700 1 --2- 172.21.15.88:0/413655672 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f03300980a0 0x7f0330137050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:38:05.904 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:05.899+0000 7f0337298700 1 --2- 172.21.15.88:0/413655672 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7f0330097cd0 0x7f0330136b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:38:05.904 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:05.899+0000 7f0337298700 1 --2- 172.21.15.88:0/413655672 >> v2:172.21.15.107:6812/29926 conn(0x7f0314033360 0x7f0314035810 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:38:05.904 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:05.899+0000 7f0337298700 1 --2- 172.21.15.88:0/413655672 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f03300966a0 0x7f03301365d0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:38:05.904 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:05.899+0000 7f0337298700 1 -- 172.21.15.88:0/413655672 >> 172.21.15.88:0/413655672 conn(0x7f03300a0910 msgr2=0x7f03300a18b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:38:05.904 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:05.899+0000 7f0337298700 1 -- 172.21.15.88:0/413655672 shutdown_connections 2023-09-30T01:38:05.904 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:05.899+0000 7f0337298700 1 -- 172.21.15.88:0/413655672 wait complete. 2023-09-30T01:38:08.914 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:38:09.028 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.027+0000 7f2345a58700 10 monclient: get_monmap_and_config 2023-09-30T01:38:09.028 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.027+0000 7f2345a58700 10 monclient: build_initial_monmap 2023-09-30T01:38:09.028 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.027+0000 7f2345a58700 10 monclient: monmap: 2023-09-30T01:38:09.029 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2023-09-30T01:38:09.029 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:38:09.029 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:38:09.028122+0000 2023-09-30T01:38:09.029 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:38:09.028122+0000 2023-09-30T01:38:09.029 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2023-09-30T01:38:09.030 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:38:09.030 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a-legacy 2023-09-30T01:38:09.030 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.107:6789/0 mon.noname-f-legacy 2023-09-30T01:38:09.030 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.noname-b 2023-09-30T01:38:09.030 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.noname-g 2023-09-30T01:38:09.031 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.noname-c 2023-09-30T01:38:09.031 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.noname-h 2023-09-30T01:38:09.031 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.noname-d 2023-09-30T01:38:09.031 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.noname-i 2023-09-30T01:38:09.031 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.noname-e 2023-09-30T01:38:09.031 INFO:teuthology.orchestra.run.smithi088.stderr:9: v2:172.21.15.88:3300/0 mon.noname-a 2023-09-30T01:38:09.031 INFO:teuthology.orchestra.run.smithi088.stderr:10: v2:172.21.15.107:3300/0 mon.noname-f 2023-09-30T01:38:09.031 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:38:09.040 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.027+0000 7f2345a58700 1 Processor -- start 2023-09-30T01:38:09.040 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.027+0000 7f2345a58700 1 -- start start 2023-09-30T01:38:09.040 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.027+0000 7f2345a58700 10 monclient: init 2023-09-30T01:38:09.040 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.027+0000 7f2345a58700 10 monclient: _reopen_session rank -1 2023-09-30T01:38:09.040 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.027+0000 7f2345a58700 10 monclient: _add_conns ranks=[5,2,0,7,8,10,4,3,1,9,6] 2023-09-30T01:38:09.040 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.027+0000 7f2345a58700 1 --2- >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7f23400a3380 0x7f23400a48b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:38:09.040 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.027+0000 7f2345a58700 10 monclient(hunting): picked mon.noname-h con 0x7f23400a3380 addr [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] 2023-09-30T01:38:09.040 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.027+0000 7f2345a58700 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f234009ffd0 0x7f23400a03a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:38:09.040 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.027+0000 7f2345a58700 10 monclient(hunting): picked mon.noname-b con 0x7f234009ffd0 addr [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] 2023-09-30T01:38:09.040 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.027+0000 7f2345a58700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f23401333b0 addr v1:172.21.15.88:6789/0 2023-09-30T01:38:09.041 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.027+0000 7f2345a58700 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f234004bb20 con 0x7f23401333b0 2023-09-30T01:38:09.041 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.027+0000 7f2345a58700 10 monclient(hunting): start opening mon connection 2023-09-30T01:38:09.041 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.027+0000 7f2345a58700 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2340097ab0 con 0x7f234009ffd0 2023-09-30T01:38:09.041 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.027+0000 7f2345a58700 10 monclient(hunting): start opening mon connection 2023-09-30T01:38:09.041 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.027+0000 7f2345a58700 1 -- --> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] -- mon_getmap magic: 0 v1 -- 0x7f23400a08e0 con 0x7f23400a3380 2023-09-30T01:38:09.041 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.027+0000 7f2345a58700 10 monclient(hunting): _renew_subs 2023-09-30T01:38:09.041 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.027+0000 7f2345a58700 10 monclient(hunting): authenticate will time out at 1161.065671s 2023-09-30T01:38:09.041 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.027+0000 7f233ffff700 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f234009ffd0 0x7f23400a03a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:38:09.041 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.027+0000 7f233ffff700 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f234009ffd0 0x7f23400a03a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3301/0 says I am v2:172.21.15.88:53454/0 (socket says 172.21.15.88:53454) 2023-09-30T01:38:09.041 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.027+0000 7f233ffff700 1 -- 172.21.15.88:0/2491224914 learned_addr learned my addr 172.21.15.88:0/2491224914 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-09-30T01:38:09.041 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.027+0000 7f233ffff700 10 monclient(hunting): get_auth_request con 0x7f234009ffd0 auth_method 0 2023-09-30T01:38:09.042 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.027+0000 7f233ffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:38:09.042 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.027+0000 7f233ffff700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:38:09.042 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.027+0000 7f233ffff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:38:09.042 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.027+0000 7f233ffff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:38:09.042 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.027+0000 7f233ffff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:38:09.042 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.027+0000 7f233ffff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:38:09.042 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.027+0000 7f2344a56700 1 --2- 172.21.15.88:0/2491224914 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7f23400a3380 0x7f23400a48b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:38:09.042 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.027+0000 7f2344a56700 10 monclient(hunting): get_auth_request con 0x7f23400a3380 auth_method 0 2023-09-30T01:38:09.042 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.027+0000 7f2344a56700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:38:09.042 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.027+0000 7f2344a56700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:38:09.042 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.027+0000 7f2344a56700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:38:09.042 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.027+0000 7f233ffff700 10 monclient(hunting): handle_auth_done global_id 4790 payload 274 2023-09-30T01:38:09.043 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.027+0000 7f233ffff700 1 -- 172.21.15.88:0/2491224914 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7f23400a3380 msgr2=0x7f23400a48b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:38:09.043 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.027+0000 7f233ffff700 1 --2- 172.21.15.88:0/2491224914 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7f23400a3380 0x7f23400a48b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:38:09.043 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.027+0000 7f233ffff700 1 -- 172.21.15.88:0/2491224914 >> v1:172.21.15.88:6789/0 conn(0x7f23401333b0 legacy=0x7f2340133800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:38:09.043 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.027+0000 7f233ffff700 10 monclient: _finish_hunting 0 2023-09-30T01:38:09.043 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.027+0000 7f233ffff700 1 monclient: found mon.noname-b 2023-09-30T01:38:09.043 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.027+0000 7f233ffff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:38:09.043 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.027+0000 7f233ffff700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.88:3301/0 2023-09-30T01:38:09.043 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.027+0000 7f233ffff700 1 -- 172.21.15.88:0/2491224914 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2340098480 con 0x7f234009ffd0 2023-09-30T01:38:09.043 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.027+0000 7f233ffff700 1 --2- 172.21.15.88:0/2491224914 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f234009ffd0 0x7f23400a03a0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f2330009f50 tx=0x7f2330009840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=500de20fbeb29fb1 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:38:09.043 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.027+0000 7f233f7fe700 1 -- 172.21.15.88:0/2491224914 <== mon.2 v2:172.21.15.88:3301/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f233000e070 con 0x7f234009ffd0 2023-09-30T01:38:09.043 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.027+0000 7f233f7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:38:09.044 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.027+0000 7f233f7fe700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T01:38:09.044 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.027+0000 7f233f7fe700 10 monclient: dump: 2023-09-30T01:38:09.044 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:38:09.044 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:38:09.044 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:38:09.044 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:38:09.044 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:38:09.044 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:38:09.044 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:38:09.044 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:38:09.044 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:38:09.044 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:38:09.045 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:38:09.045 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:38:09.045 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:38:09.045 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:38:09.045 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:38:09.045 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:38:09.045 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.027+0000 7f233f7fe700 10 monclient: _finish_auth 0 2023-09-30T01:38:09.045 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.027+0000 7f233f7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:38:09.045 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.027+0000 7f2345a58700 5 monclient: authenticate success, global_id 4790 2023-09-30T01:38:09.045 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.027+0000 7f2345a58700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:38:09.045 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.027+0000 7f233f7fe700 1 -- 172.21.15.88:0/2491224914 <== mon.2 v2:172.21.15.88:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2330007400 con 0x7f234009ffd0 2023-09-30T01:38:09.045 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.027+0000 7f233f7fe700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:38:09.046 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.027+0000 7f2345a58700 10 monclient: get_monmap_and_config success 2023-09-30T01:38:09.046 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.027+0000 7f233f7fe700 1 -- 172.21.15.88:0/2491224914 <== mon.2 v2:172.21.15.88:3301/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f23300092e0 con 0x7f234009ffd0 2023-09-30T01:38:09.046 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.027+0000 7f2345a58700 4 set_mon_vals no callback set 2023-09-30T01:38:09.046 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.027+0000 7f233f7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:38:09.046 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.027+0000 7f233f7fe700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T01:38:09.046 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.027+0000 7f233f7fe700 10 monclient: dump: 2023-09-30T01:38:09.046 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:38:09.046 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:38:09.046 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:38:09.046 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:38:09.046 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:38:09.046 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:38:09.047 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:38:09.047 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:38:09.047 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:38:09.047 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:38:09.047 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:38:09.047 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:38:09.047 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:38:09.047 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:38:09.047 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:38:09.047 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:38:09.047 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.031+0000 7f2345a58700 10 monclient: shutdown 2023-09-30T01:38:09.047 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.031+0000 7f2345a58700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:38:09.048 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.031+0000 7f2345a58700 1 -- 172.21.15.88:0/2491224914 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f234009ffd0 msgr2=0x7f23400a03a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:38:09.048 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.031+0000 7f2345a58700 1 --2- 172.21.15.88:0/2491224914 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f234009ffd0 0x7f23400a03a0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f2330009f50 tx=0x7f2330009840 comp rx=0 tx=0).stop 2023-09-30T01:38:09.048 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.031+0000 7f2345a58700 1 -- 172.21.15.88:0/2491224914 shutdown_connections 2023-09-30T01:38:09.048 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.031+0000 7f2345a58700 1 --2- 172.21.15.88:0/2491224914 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f234009ffd0 0x7f23400a03a0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:38:09.048 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.031+0000 7f2345a58700 1 --2- 172.21.15.88:0/2491224914 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7f23400a3380 0x7f23400a48b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:38:09.048 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.031+0000 7f2345a58700 1 -- 172.21.15.88:0/2491224914 >> 172.21.15.88:0/2491224914 conn(0x7f234006e5e0 msgr2=0x7f234006e9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:38:09.048 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.031+0000 7f2345a58700 1 -- 172.21.15.88:0/2491224914 shutdown_connections 2023-09-30T01:38:09.048 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.031+0000 7f2345a58700 1 -- 172.21.15.88:0/2491224914 wait complete. 2023-09-30T01:38:09.048 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.031+0000 7f2345a58700 10 monclient: build_initial_monmap 2023-09-30T01:38:09.048 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.031+0000 7f2345a58700 10 monclient: monmap: 2023-09-30T01:38:09.048 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2023-09-30T01:38:09.048 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:38:09.049 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 0.000000 2023-09-30T01:38:09.049 INFO:teuthology.orchestra.run.smithi088.stderr:created 0.000000 2023-09-30T01:38:09.049 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2023-09-30T01:38:09.049 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 1 2023-09-30T01:38:09.049 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.noname-a 2023-09-30T01:38:09.049 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.noname-f 2023-09-30T01:38:09.049 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.noname-b 2023-09-30T01:38:09.049 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.noname-g 2023-09-30T01:38:09.049 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.noname-c 2023-09-30T01:38:09.049 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.noname-h 2023-09-30T01:38:09.049 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.noname-d 2023-09-30T01:38:09.049 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.noname-i 2023-09-30T01:38:09.049 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.noname-e 2023-09-30T01:38:09.050 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:38:09.050 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.031+0000 7f2345a58700 1 Processor -- start 2023-09-30T01:38:09.050 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.031+0000 7f2345a58700 1 -- start start 2023-09-30T01:38:09.050 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.031+0000 7f2345a58700 10 monclient: init 2023-09-30T01:38:09.050 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.035+0000 7f2345a58700 10 monclient: _reopen_session rank -1 2023-09-30T01:38:09.050 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.035+0000 7f2345a58700 10 monclient: _add_conns ranks=[3,8,4,0,5,2,6,1,7] 2023-09-30T01:38:09.050 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.035+0000 7f2345a58700 1 --2- >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7f234009ffd0 0x7f234011ead0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:38:09.050 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.035+0000 7f2345a58700 10 monclient(hunting): picked mon.noname-g con 0x7f234009ffd0 addr [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] 2023-09-30T01:38:09.050 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.035+0000 7f2345a58700 1 --2- >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f23400a3380 0x7f234011f010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:38:09.050 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.035+0000 7f2345a58700 10 monclient(hunting): picked mon.noname-e con 0x7f23400a3380 addr [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] 2023-09-30T01:38:09.050 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.035+0000 7f2345a58700 1 --2- >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f23401333b0 0x7f234011f550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:38:09.050 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.035+0000 7f2345a58700 10 monclient(hunting): picked mon.noname-c con 0x7f23401333b0 addr [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] 2023-09-30T01:38:09.051 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.035+0000 7f2345a58700 10 monclient(hunting): start opening mon connection 2023-09-30T01:38:09.051 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.035+0000 7f2345a58700 1 -- --> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23400a0ac0 con 0x7f234009ffd0 2023-09-30T01:38:09.051 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.035+0000 7f2345a58700 10 monclient(hunting): start opening mon connection 2023-09-30T01:38:09.051 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.035+0000 7f2345a58700 1 -- --> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] -- mon_getmap magic: 0 v1 -- 0x7f2340097ce0 con 0x7f23401333b0 2023-09-30T01:38:09.051 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.035+0000 7f2345a58700 10 monclient(hunting): start opening mon connection 2023-09-30T01:38:09.051 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.035+0000 7f2345a58700 1 -- --> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] -- mon_getmap magic: 0 v1 -- 0x7f2340122090 con 0x7f23400a3380 2023-09-30T01:38:09.051 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.035+0000 7f2345a58700 10 monclient(hunting): _renew_subs 2023-09-30T01:38:09.051 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.035+0000 7f2345a58700 10 monclient(hunting): authenticate will time out at 1161.072472s 2023-09-30T01:38:09.051 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.035+0000 7f233ffff700 1 --2- >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f23400a3380 0x7f234011f010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:38:09.051 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.035+0000 7f233ffff700 1 --2- >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f23400a3380 0x7f234011f010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3304/0 says I am v2:172.21.15.88:35702/0 (socket says 172.21.15.88:35702) 2023-09-30T01:38:09.051 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.035+0000 7f233ffff700 1 -- 172.21.15.88:0/1055714569 learned_addr learned my addr 172.21.15.88:0/1055714569 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-09-30T01:38:09.051 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.035+0000 7f233ffff700 10 monclient(hunting): get_auth_request con 0x7f23400a3380 auth_method 0 2023-09-30T01:38:09.052 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.035+0000 7f233ffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:38:09.052 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.035+0000 7f233ffff700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:38:09.052 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.035+0000 7f233ffff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:38:09.052 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.035+0000 7f2344a56700 1 --2- 172.21.15.88:0/1055714569 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7f234009ffd0 0x7f234011ead0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:38:09.052 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.035+0000 7f2345257700 1 --2- 172.21.15.88:0/1055714569 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f23401333b0 0x7f234011f550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:38:09.052 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.035+0000 7f2344a56700 10 monclient(hunting): get_auth_request con 0x7f234009ffd0 auth_method 0 2023-09-30T01:38:09.052 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.035+0000 7f2344a56700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:38:09.052 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.035+0000 7f2344a56700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:38:09.052 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.035+0000 7f2344a56700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:38:09.052 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.035+0000 7f233ffff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:38:09.052 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.035+0000 7f233ffff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:38:09.052 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.035+0000 7f233ffff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:38:09.053 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.035+0000 7f2345257700 10 monclient(hunting): get_auth_request con 0x7f23401333b0 auth_method 0 2023-09-30T01:38:09.053 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.035+0000 7f2345257700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:38:09.053 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.035+0000 7f2345257700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:38:09.053 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.035+0000 7f2345257700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:38:09.053 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.035+0000 7f2344a56700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:38:09.053 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.035+0000 7f2344a56700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:38:09.053 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.035+0000 7f2344a56700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:38:09.053 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.035+0000 7f2345257700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:38:09.053 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.035+0000 7f2345257700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:38:09.053 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.035+0000 7f2345257700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:38:09.053 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.035+0000 7f233ffff700 10 monclient(hunting): handle_auth_done global_id 4886 payload 824 2023-09-30T01:38:09.054 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.035+0000 7f233ffff700 1 -- 172.21.15.88:0/1055714569 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f23401333b0 msgr2=0x7f234011f550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:38:09.054 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.035+0000 7f233ffff700 1 --2- 172.21.15.88:0/1055714569 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f23401333b0 0x7f234011f550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:38:09.054 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.035+0000 7f233ffff700 1 -- 172.21.15.88:0/1055714569 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7f234009ffd0 msgr2=0x7f234011ead0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:38:09.054 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.035+0000 7f233ffff700 1 --2- 172.21.15.88:0/1055714569 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7f234009ffd0 0x7f234011ead0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:38:09.054 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.035+0000 7f233ffff700 10 monclient: _finish_hunting 0 2023-09-30T01:38:09.054 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.035+0000 7f233ffff700 1 monclient: found mon.noname-e 2023-09-30T01:38:09.054 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.035+0000 7f233ffff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:38:09.054 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.035+0000 7f233ffff700 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.88:3304/0 2023-09-30T01:38:09.054 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.035+0000 7f233ffff700 1 -- 172.21.15.88:0/1055714569 --> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2330009ae0 con 0x7f23400a3380 2023-09-30T01:38:09.054 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.035+0000 7f2344a56700 1 --2- 172.21.15.88:0/1055714569 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7f234009ffd0 0x7f234011ead0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:38:09.054 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.035+0000 7f233ffff700 1 --2- 172.21.15.88:0/1055714569 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f23400a3380 0x7f234011f010 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f23400a3d10 tx=0x7f2330018000 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:38:09.055 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.035+0000 7f233cff9700 1 -- 172.21.15.88:0/1055714569 <== mon.8 v2:172.21.15.88:3304/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f2330007c00 con 0x7f23400a3380 2023-09-30T01:38:09.055 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.035+0000 7f233cff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:38:09.055 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.035+0000 7f233cff9700 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2023-09-30T01:38:09.055 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.035+0000 7f233cff9700 10 monclient: dump: 2023-09-30T01:38:09.055 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:38:09.055 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:38:09.055 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:38:09.055 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:38:09.055 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:38:09.055 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:38:09.055 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:38:09.055 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:38:09.055 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:38:09.056 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:38:09.056 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:38:09.056 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:38:09.056 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:38:09.056 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:38:09.056 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:38:09.056 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:38:09.056 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.035+0000 7f233cff9700 10 monclient: _finish_auth 0 2023-09-30T01:38:09.056 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.035+0000 7f233cff9700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:38:09.056 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.035+0000 7f233cff9700 1 -- 172.21.15.88:0/1055714569 <== mon.8 v2:172.21.15.88:3304/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2330007d80 con 0x7f23400a3380 2023-09-30T01:38:09.056 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.035+0000 7f2345a58700 5 monclient: authenticate success, global_id 4886 2023-09-30T01:38:09.057 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.035+0000 7f233cff9700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:38:09.057 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.035+0000 7f233cff9700 1 -- 172.21.15.88:0/1055714569 <== mon.8 v2:172.21.15.88:3304/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f23300268e0 con 0x7f23400a3380 2023-09-30T01:38:09.057 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.035+0000 7f233cff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:38:09.057 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.035+0000 7f233cff9700 10 monclient: got monmap 1 from mon.e (according to old e1) 2023-09-30T01:38:09.057 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.035+0000 7f233cff9700 10 monclient: dump: 2023-09-30T01:38:09.057 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:38:09.057 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:38:09.057 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:38:09.057 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:38:09.057 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:38:09.057 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:38:09.057 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:38:09.058 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:38:09.058 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:38:09.058 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:38:09.058 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:38:09.058 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:38:09.058 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:38:09.058 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:38:09.058 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:38:09.058 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:38:09.058 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.035+0000 7f233effd700 4 set_mon_vals no callback set 2023-09-30T01:38:09.058 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.035+0000 7f2345a58700 10 monclient: _renew_subs 2023-09-30T01:38:09.058 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.035+0000 7f2345a58700 10 monclient: _send_mon_message to mon.e at v2:172.21.15.88:3304/0 2023-09-30T01:38:09.059 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.035+0000 7f2345a58700 1 -- 172.21.15.88:0/1055714569 --> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23401225c0 con 0x7f23400a3380 2023-09-30T01:38:09.059 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.035+0000 7f2345a58700 10 monclient: _renew_subs 2023-09-30T01:38:09.059 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.035+0000 7f2345a58700 10 monclient: _send_mon_message to mon.e at v2:172.21.15.88:3304/0 2023-09-30T01:38:09.059 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.035+0000 7f2345a58700 1 -- 172.21.15.88:0/1055714569 --> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f23401204d0 con 0x7f23400a3380 2023-09-30T01:38:09.059 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.035+0000 7f2345a58700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:38:09.059 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.035+0000 7f2345a58700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:38:09.059 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.035+0000 7f2345a58700 10 monclient: _send_mon_message to mon.e at v2:172.21.15.88:3304/0 2023-09-30T01:38:09.059 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.035+0000 7f2345a58700 1 -- 172.21.15.88:0/1055714569 --> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2340122240 con 0x7f23400a3380 2023-09-30T01:38:09.059 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.039+0000 7f233cff9700 1 -- 172.21.15.88:0/1055714569 <== mon.8 v2:172.21.15.88:3304/0 4 ==== mgrmap(e 5) v1 ==== 40144+0+0 (secure 0 0 0) 0x7f2330026460 con 0x7f23400a3380 2023-09-30T01:38:09.059 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.039+0000 7f233cff9700 1 --2- 172.21.15.88:0/1055714569 >> v2:172.21.15.107:6812/29926 conn(0x7f232c033360 0x7f232c035810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:38:09.059 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.039+0000 7f233cff9700 1 -- 172.21.15.88:0/1055714569 <== mon.8 v2:172.21.15.88:3304/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 3502+0+0 (secure 0 0 0) 0x7f2330044fd0 con 0x7f23400a3380 2023-09-30T01:38:09.060 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.039+0000 7f2344a56700 1 --2- 172.21.15.88:0/1055714569 >> v2:172.21.15.107:6812/29926 conn(0x7f232c033360 0x7f232c035810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:38:09.060 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.039+0000 7f2344a56700 10 monclient: get_auth_request con 0x7f232c033360 auth_method 0 2023-09-30T01:38:09.060 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.039+0000 7f2344a56700 1 --2- 172.21.15.88:0/1055714569 >> v2:172.21.15.107:6812/29926 conn(0x7f232c033360 0x7f232c035810 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f2334006fd0 tx=0x7f2334008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:38:09.060 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.047+0000 7f233cff9700 1 -- 172.21.15.88:0/1055714569 <== mon.8 v2:172.21.15.88:3304/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f2330026710 con 0x7f23400a3380 2023-09-30T01:38:09.061 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.047+0000 7f233cff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:38:09.061 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.047+0000 7f233cff9700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:38:09.237 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.235+0000 7f2345a58700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T01:38:09.237 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.235+0000 7f2345a58700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:38:09.237 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.235+0000 7f2345a58700 10 monclient: _send_mon_message to mon.e at v2:172.21.15.88:3304/0 2023-09-30T01:38:09.237 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.235+0000 7f2345a58700 1 -- 172.21.15.88:0/1055714569 --> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f234009d6a0 con 0x7f23400a3380 2023-09-30T01:38:09.238 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.235+0000 7f233cff9700 1 -- 172.21.15.88:0/1055714569 <== mon.8 v2:172.21.15.88:3304/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f2330021070 con 0x7f23400a3380 2023-09-30T01:38:09.238 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.235+0000 7f233cff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:38:09.238 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.235+0000 7f233cff9700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:38:09.238 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-09-30T01:38:09.238 INFO:teuthology.orchestra.run.smithi088.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:38:09.240 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.235+0000 7f2345a58700 1 -- 172.21.15.88:0/1055714569 >> v2:172.21.15.107:6812/29926 conn(0x7f232c033360 msgr2=0x7f232c035810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:38:09.240 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.235+0000 7f2345a58700 1 --2- 172.21.15.88:0/1055714569 >> v2:172.21.15.107:6812/29926 conn(0x7f232c033360 0x7f232c035810 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f2334006fd0 tx=0x7f2334008040 comp rx=0 tx=0).stop 2023-09-30T01:38:09.240 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.235+0000 7f2345a58700 10 monclient: shutdown 2023-09-30T01:38:09.240 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.235+0000 7f2345a58700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:38:09.240 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.235+0000 7f2345a58700 1 -- 172.21.15.88:0/1055714569 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f23400a3380 msgr2=0x7f234011f010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:38:09.240 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.235+0000 7f2345a58700 1 --2- 172.21.15.88:0/1055714569 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f23400a3380 0x7f234011f010 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f23400a3d10 tx=0x7f2330018000 comp rx=0 tx=0).stop 2023-09-30T01:38:09.240 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.235+0000 7f2345a58700 1 -- 172.21.15.88:0/1055714569 shutdown_connections 2023-09-30T01:38:09.240 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.235+0000 7f2345a58700 1 --2- 172.21.15.88:0/1055714569 >> v2:172.21.15.107:6812/29926 conn(0x7f232c033360 0x7f232c035810 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:38:09.240 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.235+0000 7f2345a58700 1 --2- 172.21.15.88:0/1055714569 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f23401333b0 0x7f234011f550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:38:09.240 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.235+0000 7f2345a58700 1 --2- 172.21.15.88:0/1055714569 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f23400a3380 0x7f234011f010 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:38:09.241 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.235+0000 7f2345a58700 1 --2- 172.21.15.88:0/1055714569 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7f234009ffd0 0x7f234011ead0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:38:09.241 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.235+0000 7f2345a58700 1 -- 172.21.15.88:0/1055714569 >> 172.21.15.88:0/1055714569 conn(0x7f234006e5e0 msgr2=0x7f2340132470 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:38:09.241 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.239+0000 7f2345a58700 1 -- 172.21.15.88:0/1055714569 shutdown_connections 2023-09-30T01:38:09.241 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.239+0000 7f2345a58700 1 -- 172.21.15.88:0/1055714569 wait complete. 2023-09-30T01:38:09.251 INFO:tasks.ceph.ceph_manager.ceph:Canceling any pending splits or merges... 2023-09-30T01:38:09.251 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-09-30T01:38:09.356 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.355+0000 7f7fd4379700 10 monclient: get_monmap_and_config 2023-09-30T01:38:09.356 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.355+0000 7f7fd4379700 10 monclient: build_initial_monmap 2023-09-30T01:38:09.357 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.355+0000 7f7fd4379700 10 monclient: monmap: 2023-09-30T01:38:09.357 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2023-09-30T01:38:09.357 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:38:09.357 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:38:09.356009+0000 2023-09-30T01:38:09.357 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:38:09.356009+0000 2023-09-30T01:38:09.357 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2023-09-30T01:38:09.357 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:38:09.357 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a-legacy 2023-09-30T01:38:09.357 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.107:6789/0 mon.noname-f-legacy 2023-09-30T01:38:09.357 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.noname-b 2023-09-30T01:38:09.357 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.noname-g 2023-09-30T01:38:09.357 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.noname-c 2023-09-30T01:38:09.358 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.noname-h 2023-09-30T01:38:09.358 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.noname-d 2023-09-30T01:38:09.358 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.noname-i 2023-09-30T01:38:09.358 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.noname-e 2023-09-30T01:38:09.358 INFO:teuthology.orchestra.run.smithi088.stderr:9: v2:172.21.15.88:3300/0 mon.noname-a 2023-09-30T01:38:09.358 INFO:teuthology.orchestra.run.smithi088.stderr:10: v2:172.21.15.107:3300/0 mon.noname-f 2023-09-30T01:38:09.358 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:38:09.358 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.355+0000 7f7fd4379700 1 Processor -- start 2023-09-30T01:38:09.358 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.355+0000 7f7fd4379700 1 -- start start 2023-09-30T01:38:09.358 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.355+0000 7f7fd4379700 10 monclient: init 2023-09-30T01:38:09.366 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.355+0000 7f7fd4379700 10 monclient: _reopen_session rank -1 2023-09-30T01:38:09.366 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.355+0000 7f7fd4379700 10 monclient: _add_conns ranks=[6,2,5,9,0,7,4,10,3,1,8] 2023-09-30T01:38:09.367 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.355+0000 7f7fd4379700 1 --2- >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7f7fcc0a3270 0x7f7fcc0a0160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:38:09.367 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.355+0000 7f7fd4379700 10 monclient(hunting): picked mon.noname-d con 0x7f7fcc0a3270 addr [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] 2023-09-30T01:38:09.368 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.355+0000 7f7fd4379700 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f7fcc0a4b00 0x7f7fcc0a06a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:38:09.368 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.355+0000 7f7fd4379700 10 monclient(hunting): picked mon.noname-b con 0x7f7fcc0a4b00 addr [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] 2023-09-30T01:38:09.368 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.355+0000 7f7fd4379700 1 --2- >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7f7fcc1333c0 0x7f7fcc1337b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:38:09.368 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.355+0000 7f7fd4379700 10 monclient(hunting): picked mon.noname-h con 0x7f7fcc1333c0 addr [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] 2023-09-30T01:38:09.368 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.355+0000 7f7fd4379700 10 monclient(hunting): start opening mon connection 2023-09-30T01:38:09.368 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.355+0000 7f7fd4379700 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7fcc0a4ed0 con 0x7f7fcc0a4b00 2023-09-30T01:38:09.368 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.355+0000 7f7fd4379700 10 monclient(hunting): start opening mon connection 2023-09-30T01:38:09.368 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.355+0000 7f7fd4379700 1 -- --> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] -- mon_getmap magic: 0 v1 -- 0x7f7fcc0a6330 con 0x7f7fcc1333c0 2023-09-30T01:38:09.368 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.355+0000 7f7fd4379700 10 monclient(hunting): start opening mon connection 2023-09-30T01:38:09.368 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.355+0000 7f7fd4379700 1 -- --> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] -- mon_getmap magic: 0 v1 -- 0x7f7fcc098610 con 0x7f7fcc0a3270 2023-09-30T01:38:09.369 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.355+0000 7f7fd4379700 10 monclient(hunting): _renew_subs 2023-09-30T01:38:09.369 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.355+0000 7f7fd4379700 10 monclient(hunting): authenticate will time out at 1161.393419s 2023-09-30T01:38:09.369 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.355+0000 7f7fd1914700 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f7fcc0a4b00 0x7f7fcc0a06a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:38:09.369 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.355+0000 7f7fd2115700 1 --2- >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7f7fcc0a3270 0x7f7fcc0a0160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:38:09.369 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.355+0000 7f7fd1914700 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f7fcc0a4b00 0x7f7fcc0a06a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3301/0 says I am v2:172.21.15.88:53458/0 (socket says 172.21.15.88:53458) 2023-09-30T01:38:09.369 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.355+0000 7f7fd1914700 1 -- 172.21.15.88:0/675749921 learned_addr learned my addr 172.21.15.88:0/675749921 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-09-30T01:38:09.369 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.355+0000 7f7fd1914700 10 monclient(hunting): get_auth_request con 0x7f7fcc0a4b00 auth_method 0 2023-09-30T01:38:09.369 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.355+0000 7f7fd1914700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:38:09.369 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.355+0000 7f7fd1914700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:38:09.369 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.355+0000 7f7fd1914700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:38:09.369 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.355+0000 7f7fd2115700 10 monclient(hunting): get_auth_request con 0x7f7fcc0a3270 auth_method 0 2023-09-30T01:38:09.370 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.355+0000 7f7fd2115700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:38:09.370 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.355+0000 7f7fd2115700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:38:09.370 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.355+0000 7f7fd2115700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:38:09.370 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.355+0000 7f7fd2115700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:38:09.370 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.355+0000 7f7fd2916700 1 --2- 172.21.15.88:0/675749921 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7f7fcc1333c0 0x7f7fcc1337b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:38:09.370 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.355+0000 7f7fd2115700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:38:09.370 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.355+0000 7f7fd2115700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:38:09.370 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.355+0000 7f7fd1914700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:38:09.370 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.355+0000 7f7fd1914700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:38:09.370 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.355+0000 7f7fd1914700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:38:09.370 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.355+0000 7f7fd2916700 10 monclient(hunting): get_auth_request con 0x7f7fcc1333c0 auth_method 0 2023-09-30T01:38:09.370 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.355+0000 7f7fd2916700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:38:09.371 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.355+0000 7f7fd2916700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:38:09.371 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.355+0000 7f7fd2916700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:38:09.371 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.355+0000 7f7fd2916700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:38:09.371 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.355+0000 7f7fd2916700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:38:09.371 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.355+0000 7f7fd2916700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:38:09.371 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.355+0000 7f7fd2115700 10 monclient(hunting): handle_auth_done global_id 4767 payload 274 2023-09-30T01:38:09.371 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.355+0000 7f7fd2115700 1 -- 172.21.15.88:0/675749921 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7f7fcc1333c0 msgr2=0x7f7fcc1337b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:38:09.371 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.355+0000 7f7fd2115700 1 --2- 172.21.15.88:0/675749921 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7f7fcc1333c0 0x7f7fcc1337b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:38:09.371 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.355+0000 7f7fd2115700 1 -- 172.21.15.88:0/675749921 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f7fcc0a4b00 msgr2=0x7f7fcc0a06a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:38:09.371 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.355+0000 7f7fd2115700 1 --2- 172.21.15.88:0/675749921 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f7fcc0a4b00 0x7f7fcc0a06a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:38:09.371 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.355+0000 7f7fd2115700 10 monclient: _finish_hunting 0 2023-09-30T01:38:09.371 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.355+0000 7f7fd2115700 1 monclient: found mon.noname-d 2023-09-30T01:38:09.372 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.355+0000 7f7fd2115700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:38:09.372 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.355+0000 7f7fd2115700 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.88:3303/0 2023-09-30T01:38:09.372 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.355+0000 7f7fd2115700 1 -- 172.21.15.88:0/675749921 --> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7fcc05ecb0 con 0x7f7fcc0a3270 2023-09-30T01:38:09.372 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.355+0000 7f7fd1914700 1 --2- 172.21.15.88:0/675749921 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f7fcc0a4b00 0x7f7fcc0a06a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:38:09.372 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.355+0000 7f7fd2115700 1 --2- 172.21.15.88:0/675749921 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7f7fcc0a3270 0x7f7fcc0a0160 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f7fc0009910 tx=0x7f7fc00091a0 comp rx=0 tx=0).ready entity=mon.6 client_cookie=8d747241fa2329b8 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:38:09.372 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.355+0000 7f7fd1113700 1 -- 172.21.15.88:0/675749921 <== mon.6 v2:172.21.15.88:3303/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f7fc0011ab0 con 0x7f7fcc0a3270 2023-09-30T01:38:09.372 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.355+0000 7f7fd1113700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:38:09.372 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.355+0000 7f7fd1113700 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2023-09-30T01:38:09.372 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.355+0000 7f7fd1113700 10 monclient: dump: 2023-09-30T01:38:09.372 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:38:09.373 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:38:09.373 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:38:09.373 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:38:09.373 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:38:09.373 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:38:09.373 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:38:09.373 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:38:09.373 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:38:09.373 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:38:09.373 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:38:09.373 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:38:09.373 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:38:09.374 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:38:09.374 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:38:09.374 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:38:09.374 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.355+0000 7f7fd1113700 10 monclient: _finish_auth 0 2023-09-30T01:38:09.374 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.355+0000 7f7fd1113700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:38:09.374 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.355+0000 7f7fd1113700 1 -- 172.21.15.88:0/675749921 <== mon.6 v2:172.21.15.88:3303/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7fc0011c30 con 0x7f7fcc0a3270 2023-09-30T01:38:09.374 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.355+0000 7f7fd4379700 5 monclient: authenticate success, global_id 4767 2023-09-30T01:38:09.374 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.355+0000 7f7fd4379700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:38:09.374 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.355+0000 7f7fd1113700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:38:09.374 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.355+0000 7f7fd4379700 10 monclient: get_monmap_and_config success 2023-09-30T01:38:09.374 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.355+0000 7f7fd1113700 1 -- 172.21.15.88:0/675749921 <== mon.6 v2:172.21.15.88:3303/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f7fc0011da0 con 0x7f7fcc0a3270 2023-09-30T01:38:09.374 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.355+0000 7f7fd1113700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:38:09.374 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.355+0000 7f7fd4379700 4 set_mon_vals no callback set 2023-09-30T01:38:09.375 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.355+0000 7f7fd1113700 10 monclient: got monmap 1 from mon.d (according to old e1) 2023-09-30T01:38:09.375 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.355+0000 7f7fd1113700 10 monclient: dump: 2023-09-30T01:38:09.375 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:38:09.375 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:38:09.375 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:38:09.375 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:38:09.375 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:38:09.375 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:38:09.375 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:38:09.375 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:38:09.375 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:38:09.375 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:38:09.376 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:38:09.376 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:38:09.376 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:38:09.376 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:38:09.376 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:38:09.376 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:38:09.376 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.359+0000 7f7fd4379700 10 monclient: shutdown 2023-09-30T01:38:09.376 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.359+0000 7f7fd4379700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:38:09.376 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.359+0000 7f7fd4379700 1 -- 172.21.15.88:0/675749921 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7f7fcc0a3270 msgr2=0x7f7fcc0a0160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:38:09.376 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.359+0000 7f7fd4379700 1 --2- 172.21.15.88:0/675749921 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7f7fcc0a3270 0x7f7fcc0a0160 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f7fc0009910 tx=0x7f7fc00091a0 comp rx=0 tx=0).stop 2023-09-30T01:38:09.376 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.359+0000 7f7fd4379700 1 -- 172.21.15.88:0/675749921 shutdown_connections 2023-09-30T01:38:09.376 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.359+0000 7f7fd4379700 1 --2- 172.21.15.88:0/675749921 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f7fcc0a4b00 0x7f7fcc0a06a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:38:09.377 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.359+0000 7f7fd4379700 1 --2- 172.21.15.88:0/675749921 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7f7fcc1333c0 0x7f7fcc1337b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:38:09.377 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.359+0000 7f7fd4379700 1 --2- 172.21.15.88:0/675749921 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7f7fcc0a3270 0x7f7fcc0a0160 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:38:09.377 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.359+0000 7f7fd4379700 1 -- 172.21.15.88:0/675749921 >> 172.21.15.88:0/675749921 conn(0x7f7fcc0a67f0 msgr2=0x7f7fcc0a6bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:38:09.377 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.359+0000 7f7fd4379700 1 -- 172.21.15.88:0/675749921 shutdown_connections 2023-09-30T01:38:09.377 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.359+0000 7f7fd4379700 1 -- 172.21.15.88:0/675749921 wait complete. 2023-09-30T01:38:09.377 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.359+0000 7f7fd4379700 10 monclient: build_initial_monmap 2023-09-30T01:38:09.377 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.359+0000 7f7fd4379700 10 monclient: monmap: 2023-09-30T01:38:09.377 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2023-09-30T01:38:09.377 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:38:09.377 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 0.000000 2023-09-30T01:38:09.377 INFO:teuthology.orchestra.run.smithi088.stderr:created 0.000000 2023-09-30T01:38:09.377 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2023-09-30T01:38:09.379 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 1 2023-09-30T01:38:09.379 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.noname-a 2023-09-30T01:38:09.379 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.noname-f 2023-09-30T01:38:09.379 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.noname-b 2023-09-30T01:38:09.379 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.noname-g 2023-09-30T01:38:09.380 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.noname-c 2023-09-30T01:38:09.380 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.noname-h 2023-09-30T01:38:09.380 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.noname-d 2023-09-30T01:38:09.380 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.noname-i 2023-09-30T01:38:09.380 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.noname-e 2023-09-30T01:38:09.380 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:38:09.380 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.359+0000 7f7fd4379700 1 Processor -- start 2023-09-30T01:38:09.380 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.359+0000 7f7fd4379700 1 -- start start 2023-09-30T01:38:09.380 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.359+0000 7f7fd4379700 10 monclient: init 2023-09-30T01:38:09.380 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.359+0000 7f7fd4379700 10 monclient: _reopen_session rank -1 2023-09-30T01:38:09.380 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.359+0000 7f7fd4379700 10 monclient: _add_conns ranks=[3,1,5,7,0,2,8,6,4] 2023-09-30T01:38:09.380 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.359+0000 7f7fd4379700 1 --2- >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7f7fcc0a3270 0x7f7fcc142880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:38:09.380 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.359+0000 7f7fd4379700 10 monclient(hunting): picked mon.noname-g con 0x7f7fcc0a3270 addr [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] 2023-09-30T01:38:09.381 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.359+0000 7f7fd4379700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7fcc0a4b00 0x7f7fcc142dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:38:09.381 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.359+0000 7f7fd4379700 10 monclient(hunting): picked mon.noname-f con 0x7f7fcc0a4b00 addr [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] 2023-09-30T01:38:09.381 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.359+0000 7f7fd4379700 1 --2- >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7f7fcc1333c0 0x7f7fcc143300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:38:09.381 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.359+0000 7f7fd4379700 10 monclient(hunting): picked mon.noname-h con 0x7f7fcc1333c0 addr [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] 2023-09-30T01:38:09.381 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.359+0000 7f7fd4379700 10 monclient(hunting): start opening mon connection 2023-09-30T01:38:09.381 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.359+0000 7f7fd4379700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fcc121160 con 0x7f7fcc0a4b00 2023-09-30T01:38:09.381 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.359+0000 7f7fd4379700 10 monclient(hunting): start opening mon connection 2023-09-30T01:38:09.381 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.359+0000 7f7fd4379700 1 -- --> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7fcc121690 con 0x7f7fcc0a3270 2023-09-30T01:38:09.381 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.359+0000 7f7fd4379700 10 monclient(hunting): start opening mon connection 2023-09-30T01:38:09.381 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.359+0000 7f7fd4379700 1 -- --> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] -- mon_getmap magic: 0 v1 -- 0x7f7fcc097c70 con 0x7f7fcc1333c0 2023-09-30T01:38:09.381 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.359+0000 7f7fd4379700 10 monclient(hunting): _renew_subs 2023-09-30T01:38:09.381 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.359+0000 7f7fd4379700 10 monclient(hunting): authenticate will time out at 1161.399681s 2023-09-30T01:38:09.382 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.359+0000 7f7fd1914700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7fcc0a4b00 0x7f7fcc142dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:38:09.382 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.363+0000 7f7fd2115700 1 --2- >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7f7fcc0a3270 0x7f7fcc142880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:38:09.382 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.363+0000 7f7fd2916700 1 --2- >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7f7fcc1333c0 0x7f7fcc143300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:38:09.382 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.363+0000 7f7fd2115700 1 --2- >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7f7fcc0a3270 0x7f7fcc142880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3301/0 says I am v2:172.21.15.88:45758/0 (socket says 172.21.15.88:45758) 2023-09-30T01:38:09.382 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.363+0000 7f7fd2115700 1 -- 172.21.15.88:0/53403478 learned_addr learned my addr 172.21.15.88:0/53403478 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-09-30T01:38:09.382 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.363+0000 7f7fd2115700 10 monclient(hunting): get_auth_request con 0x7f7fcc0a3270 auth_method 0 2023-09-30T01:38:09.382 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.363+0000 7f7fd2115700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:38:09.382 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.363+0000 7f7fd2115700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:38:09.382 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.363+0000 7f7fd2115700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:38:09.382 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.363+0000 7f7fd1914700 10 monclient(hunting): get_auth_request con 0x7f7fcc0a4b00 auth_method 0 2023-09-30T01:38:09.382 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.363+0000 7f7fd1914700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:38:09.382 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.363+0000 7f7fd1914700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:38:09.383 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.363+0000 7f7fd1914700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:38:09.383 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.363+0000 7f7fd2916700 10 monclient(hunting): get_auth_request con 0x7f7fcc1333c0 auth_method 0 2023-09-30T01:38:09.383 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.363+0000 7f7fd2916700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:38:09.383 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.363+0000 7f7fd2916700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:38:09.383 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.363+0000 7f7fd2916700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:38:09.383 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.363+0000 7f7fd2115700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:38:09.383 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.363+0000 7f7fd2115700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:38:09.383 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.363+0000 7f7fd2115700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:38:09.383 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.363+0000 7f7fd1914700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:38:09.383 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.363+0000 7f7fd1914700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:38:09.383 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.363+0000 7f7fd1914700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:38:09.383 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.363+0000 7f7fd2916700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:38:09.383 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.363+0000 7f7fd2916700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:38:09.384 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.363+0000 7f7fd2916700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:38:09.384 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.363+0000 7f7fd2115700 10 monclient(hunting): handle_auth_done global_id 4719 payload 824 2023-09-30T01:38:09.384 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.363+0000 7f7fd2115700 1 -- 172.21.15.88:0/53403478 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7f7fcc1333c0 msgr2=0x7f7fcc143300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:38:09.384 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.363+0000 7f7fd2115700 1 --2- 172.21.15.88:0/53403478 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7f7fcc1333c0 0x7f7fcc143300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:38:09.384 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.363+0000 7f7fd2115700 1 -- 172.21.15.88:0/53403478 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7fcc0a4b00 msgr2=0x7f7fcc142dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:38:09.384 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.363+0000 7f7fd2115700 1 --2- 172.21.15.88:0/53403478 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7fcc0a4b00 0x7f7fcc142dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:38:09.384 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.363+0000 7f7fd2115700 10 monclient: _finish_hunting 0 2023-09-30T01:38:09.384 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.363+0000 7f7fd2115700 1 monclient: found mon.noname-g 2023-09-30T01:38:09.384 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.363+0000 7f7fd2115700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:38:09.384 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.363+0000 7f7fd2115700 10 monclient: _send_mon_message to mon.noname-g at v2:172.21.15.107:3301/0 2023-09-30T01:38:09.384 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.363+0000 7f7fd2115700 1 -- 172.21.15.88:0/53403478 --> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7fbc009ae0 con 0x7f7fcc0a3270 2023-09-30T01:38:09.384 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.363+0000 7f7fd2115700 1 --2- 172.21.15.88:0/53403478 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7f7fcc0a3270 0x7f7fcc142880 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f7fc0013000 tx=0x7f7fc0017000 comp rx=0 tx=0).ready entity=mon.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:38:09.384 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.363+0000 7f7fca7fc700 1 -- 172.21.15.88:0/53403478 <== mon.3 v2:172.21.15.107:3301/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f7fc001f070 con 0x7f7fcc0a3270 2023-09-30T01:38:09.385 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.363+0000 7f7fca7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:38:09.385 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.363+0000 7f7fca7fc700 10 monclient: got monmap 1 from mon.noname-g (according to old e1) 2023-09-30T01:38:09.385 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.363+0000 7f7fca7fc700 10 monclient: dump: 2023-09-30T01:38:09.385 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:38:09.385 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:38:09.385 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:38:09.385 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:38:09.385 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:38:09.385 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:38:09.385 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:38:09.385 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:38:09.385 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:38:09.386 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:38:09.386 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:38:09.386 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:38:09.386 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:38:09.386 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:38:09.386 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:38:09.386 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:38:09.386 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.363+0000 7f7fca7fc700 10 monclient: _finish_auth 0 2023-09-30T01:38:09.386 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.363+0000 7f7fca7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:38:09.386 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.363+0000 7f7fd4379700 5 monclient: authenticate success, global_id 4719 2023-09-30T01:38:09.386 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.363+0000 7f7fd4379700 10 monclient: _renew_subs 2023-09-30T01:38:09.386 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.363+0000 7f7fca7fc700 1 -- 172.21.15.88:0/53403478 <== mon.3 v2:172.21.15.107:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7fc0002f30 con 0x7f7fcc0a3270 2023-09-30T01:38:09.386 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.363+0000 7f7fd4379700 10 monclient: _send_mon_message to mon.g at v2:172.21.15.107:3301/0 2023-09-30T01:38:09.387 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.363+0000 7f7fd4379700 1 -- 172.21.15.88:0/53403478 --> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7fc0009420 con 0x7f7fcc0a3270 2023-09-30T01:38:09.387 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.363+0000 7f7fca7fc700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:38:09.387 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.363+0000 7f7fca7fc700 1 -- 172.21.15.88:0/53403478 <== mon.3 v2:172.21.15.107:3301/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f7fc0006a90 con 0x7f7fcc0a3270 2023-09-30T01:38:09.387 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.363+0000 7f7fca7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:38:09.387 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.363+0000 7f7fd0912700 4 set_mon_vals no callback set 2023-09-30T01:38:09.387 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.363+0000 7f7fca7fc700 10 monclient: got monmap 1 from mon.g (according to old e1) 2023-09-30T01:38:09.387 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.363+0000 7f7fca7fc700 10 monclient: dump: 2023-09-30T01:38:09.387 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:38:09.387 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:38:09.387 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:38:09.387 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:38:09.387 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:38:09.387 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:38:09.388 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:38:09.388 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:38:09.388 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:38:09.388 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:38:09.388 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:38:09.388 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:38:09.388 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:38:09.388 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:38:09.388 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:38:09.388 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:38:09.388 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.363+0000 7f7fd4379700 10 monclient: _renew_subs 2023-09-30T01:38:09.388 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.363+0000 7f7fd4379700 10 monclient: _send_mon_message to mon.g at v2:172.21.15.107:3301/0 2023-09-30T01:38:09.388 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.363+0000 7f7fd4379700 1 -- 172.21.15.88:0/53403478 --> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7fcc0a0e80 con 0x7f7fcc0a3270 2023-09-30T01:38:09.389 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.363+0000 7f7fd4379700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:38:09.389 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.363+0000 7f7fd4379700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:38:09.389 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.363+0000 7f7fd4379700 10 monclient: _send_mon_message to mon.g at v2:172.21.15.107:3301/0 2023-09-30T01:38:09.389 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.363+0000 7f7fd4379700 1 -- 172.21.15.88:0/53403478 --> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7fcc120170 con 0x7f7fcc0a3270 2023-09-30T01:38:09.389 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.363+0000 7f7fca7fc700 1 -- 172.21.15.88:0/53403478 <== mon.3 v2:172.21.15.107:3301/0 4 ==== mgrmap(e 5) v1 ==== 40144+0+0 (secure 0 0 0) 0x7f7fc0023070 con 0x7f7fcc0a3270 2023-09-30T01:38:09.390 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.363+0000 7f7fca7fc700 1 --2- 172.21.15.88:0/53403478 >> v2:172.21.15.107:6812/29926 conn(0x7f7fb8033360 0x7f7fb8035810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:38:09.390 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.363+0000 7f7fca7fc700 1 -- 172.21.15.88:0/53403478 <== mon.3 v2:172.21.15.107:3301/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 3502+0+0 (secure 0 0 0) 0x7f7fc0044040 con 0x7f7fcc0a3270 2023-09-30T01:38:09.390 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.363+0000 7f7fd1914700 1 --2- 172.21.15.88:0/53403478 >> v2:172.21.15.107:6812/29926 conn(0x7f7fb8033360 0x7f7fb8035810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:38:09.390 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.363+0000 7f7fd1914700 10 monclient: get_auth_request con 0x7f7fb8033360 auth_method 0 2023-09-30T01:38:09.390 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.367+0000 7f7fd1914700 1 --2- 172.21.15.88:0/53403478 >> v2:172.21.15.107:6812/29926 conn(0x7f7fb8033360 0x7f7fb8035810 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f7fbc0072d0 tx=0x7f7fbc00b040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:38:09.390 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.375+0000 7f7fca7fc700 1 -- 172.21.15.88:0/53403478 <== mon.3 v2:172.21.15.107:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f7fc001ae30 con 0x7f7fcc0a3270 2023-09-30T01:38:09.390 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.375+0000 7f7fca7fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:38:09.391 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.375+0000 7f7fca7fc700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:38:09.539 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.535+0000 7f7fd4379700 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2023-09-30T01:38:09.539 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.535+0000 7f7fd4379700 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2023-09-30T01:38:09.539 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.535+0000 7f7fd4379700 10 monclient: _send_mon_message to mon.g at v2:172.21.15.107:3301/0 2023-09-30T01:38:09.539 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.535+0000 7f7fd4379700 1 -- 172.21.15.88:0/53403478 --> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7fcc0a5f70 con 0x7f7fcc0a3270 2023-09-30T01:38:09.540 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.539+0000 7f7fca7fc700 1 -- 172.21.15.88:0/53403478 <== mon.3 v2:172.21.15.107:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v15) v1 ==== 74+0+9976 (secure 0 0 0) 0x7f7fc0015020 con 0x7f7fcc0a3270 2023-09-30T01:38:09.540 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.539+0000 7f7fca7fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2023-09-30T01:38:09.540 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.539+0000 7f7fca7fc700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:38:09.541 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-09-30T01:38:09.541 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":15,"fsid":"650badb5-899b-4472-88a6-cfdccee24d52","created":"2023-09-30T01:32:51.511154+0000","modified":"2023-09-30T01:33:04.280780+0000","last_up_change":"2023-09-30T01:32:59.732301+0000","last_in_change":"2023-09-30T01:32:53.067458+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":"rbd","create_time":"2023-09-30T01:33:01.387368+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":".mgr","create_time":"2023-09-30T01:33:01.712290+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"1243f825-b99c-4546-a77d-90b8637698b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6808","nonce":30177}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6809","nonce":30177}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6811","nonce":30177}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6810","nonce":30177}]},"public_addr":"172.21.15.88:6808/30177","cluster_addr":"172.21.15.88:6809/30177","heartbeat_back_addr":"172.21.15.88:6811/30177","heartbeat_front_addr":"172.21.15.88:6810/30177","state":["exists","up"]},{"osd":1,"uuid":"e9831b75-399b-4d6d-bf97-9ef104adfdb5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6804","nonce":30175}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6805","nonce":30175}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6807","nonce":30175}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6806","nonce":30175}]},"public_addr":"172.21.15.88:6804/30175","cluster_addr":"172.21.15.88:6805/30175","heartbeat_back_addr":"172.21.15.88:6807/30175","heartbeat_front_addr":"172.21.15.88:6806/30175","state":["exists","up"]},{"osd":2,"uuid":"fbc7be74-cd43-4c72-89c1-469c17055b85","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6800","nonce":30176}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6801","nonce":30176}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6803","nonce":30176}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6802","nonce":30176}]},"public_addr":"172.21.15.88:6800/30176","cluster_addr":"172.21.15.88:6801/30176","heartbeat_back_addr":"172.21.15.88:6803/30176","heartbeat_front_addr":"172.21.15.88:6802/30176","state":["exists","up"]},{"osd":3,"uuid":"ca888e00-8f75-4dc4-b70c-6cd46880fe74","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":30274}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6805","nonce":30274}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6807","nonce":30274}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":30274}]},"public_addr":"172.21.15.107:6804/30274","cluster_addr":"172.21.15.107:6805/30274","heartbeat_back_addr":"172.21.15.107:6807/30274","heartbeat_front_addr":"172.21.15.107:6806/30274","state":["exists","up"]},{"osd":4,"uuid":"c8c73bd1-7674-4529-9104-8690e25af2fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":30276}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6809","nonce":30276}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6811","nonce":30276}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":30276}]},"public_addr":"172.21.15.107:6808/30276","cluster_addr":"172.21.15.107:6809/30276","heartbeat_back_addr":"172.21.15.107:6811/30276","heartbeat_front_addr":"172.21.15.107:6810/30276","state":["exists","up"]},{"osd":5,"uuid":"a3994780-9d6a-4bcd-b718-09f9d8774b3d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":30275}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6801","nonce":30275}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6803","nonce":30275}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":30275}]},"public_addr":"172.21.15.107:6800/30275","cluster_addr":"172.21.15.107:6801/30275","heartbeat_back_addr":"172.21.15.107:6803/30275","heartbeat_front_addr":"172.21.15.107:6802/30275","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T01:32:58.527368+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T01:32:58.431507+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T01:32:58.347215+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T01:32:58.501409+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T01:32:58.565697+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T01:32:58.313349+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-09-30T01:38:09.545 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.543+0000 7f7fd4379700 1 -- 172.21.15.88:0/53403478 >> v2:172.21.15.107:6812/29926 conn(0x7f7fb8033360 msgr2=0x7f7fb8035810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:38:09.545 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.543+0000 7f7fd4379700 1 --2- 172.21.15.88:0/53403478 >> v2:172.21.15.107:6812/29926 conn(0x7f7fb8033360 0x7f7fb8035810 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f7fbc0072d0 tx=0x7f7fbc00b040 comp rx=0 tx=0).stop 2023-09-30T01:38:09.545 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.543+0000 7f7fd4379700 10 monclient: shutdown 2023-09-30T01:38:09.545 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.543+0000 7f7fd4379700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:38:09.545 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.543+0000 7f7fd4379700 1 -- 172.21.15.88:0/53403478 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7f7fcc0a3270 msgr2=0x7f7fcc142880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:38:09.545 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.543+0000 7f7fd4379700 1 --2- 172.21.15.88:0/53403478 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7f7fcc0a3270 0x7f7fcc142880 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f7fc0013000 tx=0x7f7fc0017000 comp rx=0 tx=0).stop 2023-09-30T01:38:09.546 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.543+0000 7f7fd4379700 1 -- 172.21.15.88:0/53403478 shutdown_connections 2023-09-30T01:38:09.546 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.543+0000 7f7fd4379700 1 --2- 172.21.15.88:0/53403478 >> v2:172.21.15.107:6812/29926 conn(0x7f7fb8033360 0x7f7fb8035810 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:38:09.546 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.543+0000 7f7fd4379700 1 --2- 172.21.15.88:0/53403478 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7f7fcc1333c0 0x7f7fcc143300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:38:09.546 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.543+0000 7f7fd4379700 1 --2- 172.21.15.88:0/53403478 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7fcc0a4b00 0x7f7fcc142dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:38:09.546 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.543+0000 7f7fd4379700 1 --2- 172.21.15.88:0/53403478 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7f7fcc0a3270 0x7f7fcc142880 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:38:09.546 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.543+0000 7f7fd4379700 1 -- 172.21.15.88:0/53403478 >> 172.21.15.88:0/53403478 conn(0x7f7fcc0a67f0 msgr2=0x7f7fcc0a20e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:38:09.546 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.543+0000 7f7fd4379700 1 -- 172.21.15.88:0/53403478 shutdown_connections 2023-09-30T01:38:09.546 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.543+0000 7f7fd4379700 1 -- 172.21.15.88:0/53403478 wait complete. 2023-09-30T01:38:09.562 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2023-09-30T01:38:09.562 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-09-30T01:38:09.665 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.663+0000 7f44d5432700 10 monclient: get_monmap_and_config 2023-09-30T01:38:09.666 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.663+0000 7f44d5432700 10 monclient: build_initial_monmap 2023-09-30T01:38:09.666 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.663+0000 7f44d5432700 10 monclient: monmap: 2023-09-30T01:38:09.666 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2023-09-30T01:38:09.666 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:38:09.666 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:38:09.665261+0000 2023-09-30T01:38:09.666 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:38:09.665261+0000 2023-09-30T01:38:09.666 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2023-09-30T01:38:09.666 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:38:09.666 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a-legacy 2023-09-30T01:38:09.666 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.107:6789/0 mon.noname-f-legacy 2023-09-30T01:38:09.666 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.noname-b 2023-09-30T01:38:09.667 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.noname-g 2023-09-30T01:38:09.667 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.noname-c 2023-09-30T01:38:09.667 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.noname-h 2023-09-30T01:38:09.667 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.noname-d 2023-09-30T01:38:09.667 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.noname-i 2023-09-30T01:38:09.667 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.noname-e 2023-09-30T01:38:09.667 INFO:teuthology.orchestra.run.smithi088.stderr:9: v2:172.21.15.88:3300/0 mon.noname-a 2023-09-30T01:38:09.667 INFO:teuthology.orchestra.run.smithi088.stderr:10: v2:172.21.15.107:3300/0 mon.noname-f 2023-09-30T01:38:09.667 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:38:09.667 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.663+0000 7f44d5432700 1 Processor -- start 2023-09-30T01:38:09.667 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.663+0000 7f44d5432700 1 -- start start 2023-09-30T01:38:09.667 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.663+0000 7f44d5432700 10 monclient: init 2023-09-30T01:38:09.675 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.663+0000 7f44d5432700 10 monclient: _reopen_session rank -1 2023-09-30T01:38:09.675 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.663+0000 7f44d5432700 10 monclient: _add_conns ranks=[6,0,3,9,5,8,1,7,4,10,2] 2023-09-30T01:38:09.675 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.663+0000 7f44d5432700 1 --2- >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7f44d00a3270 0x7f44d00a0160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:38:09.675 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.663+0000 7f44d5432700 10 monclient(hunting): picked mon.noname-d con 0x7f44d00a3270 addr [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] 2023-09-30T01:38:09.675 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.663+0000 7f44d5432700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f44d00a4b00 addr v1:172.21.15.88:6789/0 2023-09-30T01:38:09.675 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.663+0000 7f44d5432700 1 --2- >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7f44d0138750 0x7f44d013ab30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:38:09.675 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.663+0000 7f44d5432700 10 monclient(hunting): picked mon.noname-g con 0x7f44d0138750 addr [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] 2023-09-30T01:38:09.675 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.663+0000 7f44d5432700 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44d005ecb0 con 0x7f44d00a4b00 2023-09-30T01:38:09.675 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.663+0000 7f44d5432700 10 monclient(hunting): start opening mon connection 2023-09-30T01:38:09.676 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.663+0000 7f44d5432700 1 -- --> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44d00a4ed0 con 0x7f44d0138750 2023-09-30T01:38:09.676 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.663+0000 7f44d5432700 10 monclient(hunting): start opening mon connection 2023-09-30T01:38:09.676 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.663+0000 7f44d5432700 1 -- --> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] -- mon_getmap magic: 0 v1 -- 0x7f44d00a6330 con 0x7f44d00a3270 2023-09-30T01:38:09.676 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.663+0000 7f44d5432700 10 monclient(hunting): _renew_subs 2023-09-30T01:38:09.676 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.663+0000 7f44d5432700 10 monclient(hunting): authenticate will time out at 1161.702698s 2023-09-30T01:38:09.676 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.663+0000 7f44ceffd700 1 --2- >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7f44d00a3270 0x7f44d00a0160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:38:09.676 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.663+0000 7f44ce7fc700 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f44d00a4b00 0x7f44d00a06a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:57992/0 (socket says 172.21.15.88:57992) 2023-09-30T01:38:09.676 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.663+0000 7f44ce7fc700 1 -- 172.21.15.88:0/3078017351 learned_addr learned my addr 172.21.15.88:0/3078017351 (peer_addr_for_me v1:172.21.15.88:0/0) 2023-09-30T01:38:09.676 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.663+0000 7f44ceffd700 10 monclient(hunting): get_auth_request con 0x7f44d00a3270 auth_method 0 2023-09-30T01:38:09.676 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.663+0000 7f44ceffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:38:09.676 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.663+0000 7f44ceffd700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:38:09.676 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.663+0000 7f44ceffd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:38:09.677 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.663+0000 7f44ceffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:38:09.677 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.663+0000 7f44ceffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:38:09.677 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.663+0000 7f44ceffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:38:09.677 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.663+0000 7f44cf7fe700 1 --2- 172.21.15.88:0/3078017351 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7f44d0138750 0x7f44d013ab30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:38:09.677 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.663+0000 7f44cf7fe700 10 monclient(hunting): get_auth_request con 0x7f44d0138750 auth_method 0 2023-09-30T01:38:09.677 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.663+0000 7f44cf7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:38:09.677 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.663+0000 7f44cf7fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:38:09.677 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.663+0000 7f44cf7fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:38:09.677 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.663+0000 7f44cdffb700 1 -- 172.21.15.88:0/3078017351 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 49844042 0 0) 0x7f44d005ecb0 con 0x7f44d00a4b00 2023-09-30T01:38:09.677 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.663+0000 7f44cdffb700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:38:09.677 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.663+0000 7f44cdffb700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:38:09.677 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.663+0000 7f44cdffb700 10 monclient(hunting): my global_id is 5184 2023-09-30T01:38:09.678 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.663+0000 7f44cdffb700 1 -- 172.21.15.88:0/3078017351 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f44b8003570 con 0x7f44d00a4b00 2023-09-30T01:38:09.678 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.663+0000 7f44ceffd700 10 monclient(hunting): handle_auth_done global_id 4776 payload 274 2023-09-30T01:38:09.678 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.663+0000 7f44ceffd700 1 -- 172.21.15.88:0/3078017351 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7f44d0138750 msgr2=0x7f44d013ab30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:38:09.678 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.663+0000 7f44ceffd700 1 --2- 172.21.15.88:0/3078017351 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7f44d0138750 0x7f44d013ab30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:38:09.678 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.663+0000 7f44ceffd700 1 -- 172.21.15.88:0/3078017351 >> v1:172.21.15.88:6789/0 conn(0x7f44d00a4b00 legacy=0x7f44d00a06a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:38:09.678 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.663+0000 7f44ceffd700 10 monclient: _finish_hunting 0 2023-09-30T01:38:09.678 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.663+0000 7f44ceffd700 1 monclient: found mon.noname-d 2023-09-30T01:38:09.678 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.663+0000 7f44ceffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:38:09.678 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.663+0000 7f44ceffd700 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.88:3303/0 2023-09-30T01:38:09.678 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.663+0000 7f44ceffd700 1 -- 172.21.15.88:0/3078017351 --> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44d0098610 con 0x7f44d00a3270 2023-09-30T01:38:09.678 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.663+0000 7f44ceffd700 1 --2- 172.21.15.88:0/3078017351 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7f44d00a3270 0x7f44d00a0160 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f44c8009e90 tx=0x7f44c8009780 comp rx=0 tx=0).ready entity=mon.6 client_cookie=21bd4b3c1e381940 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:38:09.678 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.663+0000 7f44cdffb700 1 -- 172.21.15.88:0/3078017351 <== mon.6 v2:172.21.15.88:3303/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f44c800e070 con 0x7f44d00a3270 2023-09-30T01:38:09.679 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.663+0000 7f44cdffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:38:09.679 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.663+0000 7f44cdffb700 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2023-09-30T01:38:09.679 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.663+0000 7f44cdffb700 10 monclient: dump: 2023-09-30T01:38:09.679 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:38:09.679 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:38:09.679 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:38:09.679 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:38:09.679 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:38:09.679 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:38:09.679 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:38:09.680 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:38:09.680 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:38:09.680 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:38:09.680 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:38:09.680 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:38:09.680 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:38:09.680 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:38:09.680 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:38:09.680 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:38:09.680 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.663+0000 7f44cdffb700 10 monclient: _finish_auth 0 2023-09-30T01:38:09.681 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.663+0000 7f44cdffb700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:38:09.681 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.663+0000 7f44cdffb700 1 -- 172.21.15.88:0/3078017351 <== mon.6 v2:172.21.15.88:3303/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f44c800bc30 con 0x7f44d00a3270 2023-09-30T01:38:09.681 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.663+0000 7f44d5432700 5 monclient: authenticate success, global_id 4776 2023-09-30T01:38:09.681 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.663+0000 7f44d5432700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:38:09.681 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.663+0000 7f44cdffb700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:38:09.681 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.663+0000 7f44d5432700 10 monclient: get_monmap_and_config success 2023-09-30T01:38:09.681 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.663+0000 7f44cdffb700 1 -- 172.21.15.88:0/3078017351 <== mon.6 v2:172.21.15.88:3303/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f44c800bda0 con 0x7f44d00a3270 2023-09-30T01:38:09.681 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.663+0000 7f44d5432700 4 set_mon_vals no callback set 2023-09-30T01:38:09.682 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.663+0000 7f44cdffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:38:09.682 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.663+0000 7f44cdffb700 10 monclient: got monmap 1 from mon.d (according to old e1) 2023-09-30T01:38:09.682 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.663+0000 7f44cdffb700 10 monclient: dump: 2023-09-30T01:38:09.682 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:38:09.682 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:38:09.682 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:38:09.682 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:38:09.682 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:38:09.682 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:38:09.683 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:38:09.683 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:38:09.683 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:38:09.683 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:38:09.683 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:38:09.683 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:38:09.683 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:38:09.683 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:38:09.683 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:38:09.683 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:38:09.683 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.667+0000 7f44d5432700 10 monclient: shutdown 2023-09-30T01:38:09.683 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.667+0000 7f44d5432700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:38:09.684 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.667+0000 7f44d5432700 1 -- 172.21.15.88:0/3078017351 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7f44d00a3270 msgr2=0x7f44d00a0160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:38:09.684 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.667+0000 7f44d5432700 1 --2- 172.21.15.88:0/3078017351 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7f44d00a3270 0x7f44d00a0160 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f44c8009e90 tx=0x7f44c8009780 comp rx=0 tx=0).stop 2023-09-30T01:38:09.684 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.667+0000 7f44d5432700 1 -- 172.21.15.88:0/3078017351 shutdown_connections 2023-09-30T01:38:09.684 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.667+0000 7f44d5432700 1 --2- 172.21.15.88:0/3078017351 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7f44d0138750 0x7f44d013ab30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:38:09.684 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.667+0000 7f44d5432700 1 --2- 172.21.15.88:0/3078017351 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7f44d00a3270 0x7f44d00a0160 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:38:09.684 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.667+0000 7f44d5432700 1 -- 172.21.15.88:0/3078017351 >> 172.21.15.88:0/3078017351 conn(0x7f44d00a67f0 msgr2=0x7f44d00a6bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:38:09.684 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.667+0000 7f44d5432700 1 -- 172.21.15.88:0/3078017351 shutdown_connections 2023-09-30T01:38:09.684 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.667+0000 7f44d5432700 1 -- 172.21.15.88:0/3078017351 wait complete. 2023-09-30T01:38:09.684 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.667+0000 7f44d5432700 10 monclient: build_initial_monmap 2023-09-30T01:38:09.684 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.667+0000 7f44d5432700 10 monclient: monmap: 2023-09-30T01:38:09.684 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2023-09-30T01:38:09.684 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:38:09.684 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 0.000000 2023-09-30T01:38:09.685 INFO:teuthology.orchestra.run.smithi088.stderr:created 0.000000 2023-09-30T01:38:09.685 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2023-09-30T01:38:09.685 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 1 2023-09-30T01:38:09.685 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.noname-a 2023-09-30T01:38:09.685 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.noname-f 2023-09-30T01:38:09.685 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.noname-b 2023-09-30T01:38:09.685 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.noname-g 2023-09-30T01:38:09.685 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.noname-c 2023-09-30T01:38:09.685 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.noname-h 2023-09-30T01:38:09.685 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.noname-d 2023-09-30T01:38:09.685 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.noname-i 2023-09-30T01:38:09.685 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.noname-e 2023-09-30T01:38:09.685 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:38:09.686 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.667+0000 7f44d5432700 1 Processor -- start 2023-09-30T01:38:09.686 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.667+0000 7f44d5432700 1 -- start start 2023-09-30T01:38:09.686 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.667+0000 7f44d5432700 10 monclient: init 2023-09-30T01:38:09.686 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.667+0000 7f44d5432700 10 monclient: _reopen_session rank -1 2023-09-30T01:38:09.686 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.667+0000 7f44d5432700 10 monclient: _add_conns ranks=[3,4,8,2,7,0,1,5,6] 2023-09-30T01:38:09.686 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.667+0000 7f44d5432700 1 --2- >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7f44d00a3270 0x7f44d009fb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:38:09.686 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.667+0000 7f44d5432700 10 monclient(hunting): picked mon.noname-g con 0x7f44d00a3270 addr [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] 2023-09-30T01:38:09.686 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.667+0000 7f44d5432700 1 --2- >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f44d00a4b00 0x7f44d0135d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:38:09.686 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.667+0000 7f44d5432700 10 monclient(hunting): picked mon.noname-c con 0x7f44d00a4b00 addr [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] 2023-09-30T01:38:09.686 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.667+0000 7f44d5432700 1 --2- >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f44d0138750 0x7f44d0136280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:38:09.686 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.667+0000 7f44d5432700 10 monclient(hunting): picked mon.noname-e con 0x7f44d0138750 addr [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] 2023-09-30T01:38:09.686 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.667+0000 7f44d5432700 10 monclient(hunting): start opening mon connection 2023-09-30T01:38:09.687 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.667+0000 7f44d5432700 1 -- --> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44d0099f20 con 0x7f44d00a3270 2023-09-30T01:38:09.687 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.667+0000 7f44d5432700 10 monclient(hunting): start opening mon connection 2023-09-30T01:38:09.687 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.667+0000 7f44d5432700 1 -- --> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] -- mon_getmap magic: 0 v1 -- 0x7f44d0099c40 con 0x7f44d00a4b00 2023-09-30T01:38:09.687 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.667+0000 7f44d5432700 10 monclient(hunting): start opening mon connection 2023-09-30T01:38:09.687 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.667+0000 7f44d5432700 1 -- --> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] -- mon_getmap magic: 0 v1 -- 0x7f44d009a200 con 0x7f44d0138750 2023-09-30T01:38:09.687 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.667+0000 7f44d5432700 10 monclient(hunting): _renew_subs 2023-09-30T01:38:09.687 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.667+0000 7f44d5432700 10 monclient(hunting): authenticate will time out at 1161.706020s 2023-09-30T01:38:09.687 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.667+0000 7f44ce7fc700 1 --2- >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f44d00a4b00 0x7f44d0135d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:38:09.687 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.667+0000 7f44ce7fc700 1 --2- >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f44d00a4b00 0x7f44d0135d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3302/0 says I am v2:172.21.15.88:33914/0 (socket says 172.21.15.88:33914) 2023-09-30T01:38:09.687 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.667+0000 7f44ce7fc700 1 -- 172.21.15.88:0/4155424723 learned_addr learned my addr 172.21.15.88:0/4155424723 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-09-30T01:38:09.687 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.667+0000 7f44ce7fc700 10 monclient(hunting): get_auth_request con 0x7f44d00a4b00 auth_method 0 2023-09-30T01:38:09.687 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.667+0000 7f44ce7fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:38:09.688 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.667+0000 7f44ce7fc700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:38:09.688 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.667+0000 7f44ce7fc700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:38:09.688 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.667+0000 7f44cf7fe700 1 --2- 172.21.15.88:0/4155424723 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f44d0138750 0x7f44d0136280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:38:09.688 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.667+0000 7f44cf7fe700 10 monclient(hunting): get_auth_request con 0x7f44d0138750 auth_method 0 2023-09-30T01:38:09.688 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.667+0000 7f44cf7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:38:09.688 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.667+0000 7f44cf7fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:38:09.688 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.667+0000 7f44cf7fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:38:09.688 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.667+0000 7f44ce7fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:38:09.688 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.667+0000 7f44ce7fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:38:09.688 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.667+0000 7f44ce7fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:38:09.688 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.667+0000 7f44ceffd700 1 --2- 172.21.15.88:0/4155424723 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7f44d00a3270 0x7f44d009fb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:38:09.688 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.667+0000 7f44cf7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:38:09.689 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.667+0000 7f44cf7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:38:09.689 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.667+0000 7f44cf7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:38:09.689 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.667+0000 7f44ceffd700 10 monclient(hunting): get_auth_request con 0x7f44d00a3270 auth_method 0 2023-09-30T01:38:09.689 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.667+0000 7f44ceffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:38:09.689 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.667+0000 7f44ceffd700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:38:09.689 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.667+0000 7f44ceffd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:38:09.689 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.667+0000 7f44ceffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:38:09.689 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.667+0000 7f44ceffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:38:09.689 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.667+0000 7f44ceffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:38:09.689 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.667+0000 7f44ce7fc700 10 monclient(hunting): handle_auth_done global_id 4909 payload 824 2023-09-30T01:38:09.689 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.667+0000 7f44ce7fc700 1 -- 172.21.15.88:0/4155424723 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f44d0138750 msgr2=0x7f44d0136280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:38:09.689 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.667+0000 7f44ce7fc700 1 --2- 172.21.15.88:0/4155424723 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f44d0138750 0x7f44d0136280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:38:09.689 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.667+0000 7f44ce7fc700 1 -- 172.21.15.88:0/4155424723 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7f44d00a3270 msgr2=0x7f44d009fb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:38:09.690 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.667+0000 7f44ce7fc700 1 --2- 172.21.15.88:0/4155424723 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7f44d00a3270 0x7f44d009fb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:38:09.690 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.667+0000 7f44ce7fc700 10 monclient: _finish_hunting 0 2023-09-30T01:38:09.690 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.667+0000 7f44ce7fc700 1 monclient: found mon.noname-c 2023-09-30T01:38:09.690 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.667+0000 7f44ce7fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:38:09.690 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.667+0000 7f44ce7fc700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.88:3302/0 2023-09-30T01:38:09.690 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.667+0000 7f44ce7fc700 1 -- 172.21.15.88:0/4155424723 --> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44c8009a20 con 0x7f44d00a4b00 2023-09-30T01:38:09.690 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.667+0000 7f44cf7fe700 1 --2- 172.21.15.88:0/4155424723 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f44d0138750 0x7f44d0136280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:38:09.690 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.667+0000 7f44ce7fc700 1 --2- 172.21.15.88:0/4155424723 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f44d00a4b00 0x7f44d0135d40 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f44c40011e0 tx=0x7f44c4007df0 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:38:09.690 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.667+0000 7f44b6ffd700 1 -- 172.21.15.88:0/4155424723 <== mon.4 v2:172.21.15.88:3302/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f44c40070d0 con 0x7f44d00a4b00 2023-09-30T01:38:09.690 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.667+0000 7f44b6ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:38:09.690 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.667+0000 7f44b6ffd700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T01:38:09.691 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.667+0000 7f44b6ffd700 10 monclient: dump: 2023-09-30T01:38:09.691 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:38:09.691 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:38:09.691 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:38:09.691 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:38:09.691 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:38:09.691 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:38:09.691 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:38:09.691 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:38:09.691 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:38:09.692 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:38:09.692 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:38:09.692 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:38:09.692 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:38:09.692 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:38:09.692 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:38:09.692 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:38:09.692 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.667+0000 7f44b6ffd700 10 monclient: _finish_auth 0 2023-09-30T01:38:09.692 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.667+0000 7f44b6ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:38:09.692 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.667+0000 7f44d5432700 5 monclient: authenticate success, global_id 4909 2023-09-30T01:38:09.692 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.667+0000 7f44b6ffd700 1 -- 172.21.15.88:0/4155424723 <== mon.4 v2:172.21.15.88:3302/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f44c4007250 con 0x7f44d00a4b00 2023-09-30T01:38:09.692 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.667+0000 7f44b6ffd700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:38:09.692 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.667+0000 7f44d5432700 10 monclient: _renew_subs 2023-09-30T01:38:09.693 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.667+0000 7f44b6ffd700 1 -- 172.21.15.88:0/4155424723 <== mon.4 v2:172.21.15.88:3302/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f44c4011360 con 0x7f44d00a4b00 2023-09-30T01:38:09.693 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.667+0000 7f44d5432700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.88:3302/0 2023-09-30T01:38:09.693 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.667+0000 7f44d5432700 1 -- 172.21.15.88:0/4155424723 --> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f44d009a900 con 0x7f44d00a4b00 2023-09-30T01:38:09.693 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.667+0000 7f44b6ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:38:09.693 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.667+0000 7f44ccff9700 4 set_mon_vals no callback set 2023-09-30T01:38:09.693 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.667+0000 7f44b6ffd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T01:38:09.693 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.667+0000 7f44b6ffd700 10 monclient: dump: 2023-09-30T01:38:09.693 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:38:09.693 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:38:09.693 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:38:09.693 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:38:09.693 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:38:09.694 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:38:09.694 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:38:09.694 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:38:09.694 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:38:09.694 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:38:09.694 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:38:09.694 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:38:09.694 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:38:09.694 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:38:09.694 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:38:09.694 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:38:09.694 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.667+0000 7f44d5432700 10 monclient: _renew_subs 2023-09-30T01:38:09.694 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.667+0000 7f44d5432700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.88:3302/0 2023-09-30T01:38:09.695 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.667+0000 7f44d5432700 1 -- 172.21.15.88:0/4155424723 --> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f44d01383f0 con 0x7f44d00a4b00 2023-09-30T01:38:09.695 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.671+0000 7f44b6ffd700 1 -- 172.21.15.88:0/4155424723 <== mon.4 v2:172.21.15.88:3302/0 4 ==== mgrmap(e 5) v1 ==== 40144+0+0 (secure 0 0 0) 0x7f44c4005b30 con 0x7f44d00a4b00 2023-09-30T01:38:09.695 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.671+0000 7f44d5432700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:38:09.695 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.671+0000 7f44d5432700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:38:09.695 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.671+0000 7f44d5432700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.88:3302/0 2023-09-30T01:38:09.695 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.671+0000 7f44b6ffd700 1 --2- 172.21.15.88:0/4155424723 >> v2:172.21.15.107:6812/29926 conn(0x7f44bc033360 0x7f44bc035810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:38:09.695 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.671+0000 7f44d5432700 1 -- 172.21.15.88:0/4155424723 --> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f44d0071960 con 0x7f44d00a4b00 2023-09-30T01:38:09.695 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.671+0000 7f44b6ffd700 1 -- 172.21.15.88:0/4155424723 <== mon.4 v2:172.21.15.88:3302/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 3502+0+0 (secure 0 0 0) 0x7f44c40423e0 con 0x7f44d00a4b00 2023-09-30T01:38:09.695 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.671+0000 7f44ceffd700 1 --2- 172.21.15.88:0/4155424723 >> v2:172.21.15.107:6812/29926 conn(0x7f44bc033360 0x7f44bc035810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:38:09.695 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.671+0000 7f44ceffd700 10 monclient: get_auth_request con 0x7f44bc033360 auth_method 0 2023-09-30T01:38:09.695 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.671+0000 7f44ceffd700 1 --2- 172.21.15.88:0/4155424723 >> v2:172.21.15.107:6812/29926 conn(0x7f44bc033360 0x7f44bc035810 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f44c8000f80 tx=0x7f44c8019000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:38:09.696 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.675+0000 7f44b6ffd700 1 -- 172.21.15.88:0/4155424723 <== mon.4 v2:172.21.15.88:3302/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f44c4017070 con 0x7f44d00a4b00 2023-09-30T01:38:09.696 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.675+0000 7f44b6ffd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:38:09.696 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.675+0000 7f44b6ffd700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:38:09.837 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.835+0000 7f44d5432700 1 -- 172.21.15.88:0/4155424723 --> v2:172.21.15.107:6812/29926 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f44d0099db0 con 0x7f44bc033360 2023-09-30T01:38:09.843 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.839+0000 7f44b6ffd700 1 -- 172.21.15.88:0/4155424723 <== mgr.4108 v2:172.21.15.107:6812/29926 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+53337 (secure 0 0 0) 0x7f44d0099db0 con 0x7f44bc033360 2023-09-30T01:38:09.844 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-09-30T01:38:09.846 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.843+0000 7f44d5432700 1 -- 172.21.15.88:0/4155424723 >> v2:172.21.15.107:6812/29926 conn(0x7f44bc033360 msgr2=0x7f44bc035810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:38:09.846 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.843+0000 7f44d5432700 1 --2- 172.21.15.88:0/4155424723 >> v2:172.21.15.107:6812/29926 conn(0x7f44bc033360 0x7f44bc035810 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f44c8000f80 tx=0x7f44c8019000 comp rx=0 tx=0).stop 2023-09-30T01:38:09.846 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.843+0000 7f44d5432700 10 monclient: shutdown 2023-09-30T01:38:09.846 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.843+0000 7f44d5432700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:38:09.846 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.843+0000 7f44d5432700 1 -- 172.21.15.88:0/4155424723 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f44d00a4b00 msgr2=0x7f44d0135d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:38:09.846 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.843+0000 7f44d5432700 1 --2- 172.21.15.88:0/4155424723 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f44d00a4b00 0x7f44d0135d40 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f44c40011e0 tx=0x7f44c4007df0 comp rx=0 tx=0).stop 2023-09-30T01:38:09.846 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.843+0000 7f44d5432700 1 -- 172.21.15.88:0/4155424723 shutdown_connections 2023-09-30T01:38:09.846 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.843+0000 7f44d5432700 1 --2- 172.21.15.88:0/4155424723 >> v2:172.21.15.107:6812/29926 conn(0x7f44bc033360 0x7f44bc035810 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:38:09.847 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.843+0000 7f44d5432700 1 --2- 172.21.15.88:0/4155424723 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f44d0138750 0x7f44d0136280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:38:09.847 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.843+0000 7f44d5432700 1 --2- 172.21.15.88:0/4155424723 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f44d00a4b00 0x7f44d0135d40 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:38:09.847 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.843+0000 7f44d5432700 1 --2- 172.21.15.88:0/4155424723 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7f44d00a3270 0x7f44d009fb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:38:09.847 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.843+0000 7f44d5432700 1 -- 172.21.15.88:0/4155424723 >> 172.21.15.88:0/4155424723 conn(0x7f44d00a67f0 msgr2=0x7f44d00a20e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:38:09.847 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.843+0000 7f44d5432700 1 -- 172.21.15.88:0/4155424723 shutdown_connections 2023-09-30T01:38:09.847 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.843+0000 7f44d5432700 1 -- 172.21.15.88:0/4155424723 wait complete. 2023-09-30T01:38:09.847 INFO:teuthology.orchestra.run.smithi088.stderr:dumped all 2023-09-30T01:38:09.858 INFO:teuthology.orchestra.run.smithi088.stdout:{"pg_ready":true,"pg_map":{"version":165,"stamp":"2023-09-30T01:38:09.743539+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":32704,"kb_used_data":1808,"kb_used_omap":0,"kb_used_meta":30848,"kb_avail":566198336,"statfs":{"total":579820584960,"available":579787096064,"internally_reserved":0,"allocated":1851392,"data_stored":1284542,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":31588352},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003602"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-09-30T01:33:03.296691+0000","last_change":"2023-09-30T01:33:03.292315+0000","last_active":"2023-09-30T01:33:03.296691+0000","last_peered":"2023-09-30T01:33:03.296691+0000","last_clean":"2023-09-30T01:33:03.296691+0000","last_became_active":"2023-09-30T01:33:03.291889+0000","last_became_peered":"2023-09-30T01:33:03.291889+0000","last_unstale":"2023-09-30T01:33:03.296691+0000","last_undegraded":"2023-09-30T01:33:03.296691+0000","last_fullsized":"2023-09-30T01:33:03.296691+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T01:33:02.252599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T01:33:02.252599+0000","last_clean_scrub_stamp":"2023-09-30T01:33:02.252599+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T11:44:00.574379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-09-30T01:33:03.297822+0000","last_change":"2023-09-30T01:33:03.295081+0000","last_active":"2023-09-30T01:33:03.297822+0000","last_peered":"2023-09-30T01:33:03.297822+0000","last_clean":"2023-09-30T01:33:03.297822+0000","last_became_active":"2023-09-30T01:33:03.294535+0000","last_became_peered":"2023-09-30T01:33:03.294535+0000","last_unstale":"2023-09-30T01:33:03.297822+0000","last_undegraded":"2023-09-30T01:33:03.297822+0000","last_fullsized":"2023-09-30T01:33:03.297822+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T01:33:02.252599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T01:33:02.252599+0000","last_clean_scrub_stamp":"2023-09-30T01:33:02.252599+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T05:49:49.642723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-09-30T01:33:03.300117+0000","last_change":"2023-09-30T01:33:03.297392+0000","last_active":"2023-09-30T01:33:03.300117+0000","last_peered":"2023-09-30T01:33:03.300117+0000","last_clean":"2023-09-30T01:33:03.300117+0000","last_became_active":"2023-09-30T01:33:03.296959+0000","last_became_peered":"2023-09-30T01:33:03.296959+0000","last_unstale":"2023-09-30T01:33:03.300117+0000","last_undegraded":"2023-09-30T01:33:03.300117+0000","last_fullsized":"2023-09-30T01:33:03.300117+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T01:33:02.252599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T01:33:02.252599+0000","last_clean_scrub_stamp":"2023-09-30T01:33:02.252599+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T11:57:13.112607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-09-30T01:33:03.296932+0000","last_change":"2023-09-30T01:33:03.295478+0000","last_active":"2023-09-30T01:33:03.296932+0000","last_peered":"2023-09-30T01:33:03.296932+0000","last_clean":"2023-09-30T01:33:03.296932+0000","last_became_active":"2023-09-30T01:33:03.295106+0000","last_became_peered":"2023-09-30T01:33:03.295106+0000","last_unstale":"2023-09-30T01:33:03.296932+0000","last_undegraded":"2023-09-30T01:33:03.296932+0000","last_fullsized":"2023-09-30T01:33:03.296932+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T01:33:02.252599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T01:33:02.252599+0000","last_clean_scrub_stamp":"2023-09-30T01:33:02.252599+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T07:34:36.123333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"15'117","reported_seq":159,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-09-30T01:33:05.391246+0000","last_change":"2023-09-30T01:33:03.296872+0000","last_active":"2023-09-30T01:33:05.391246+0000","last_peered":"2023-09-30T01:33:05.391246+0000","last_clean":"2023-09-30T01:33:05.391246+0000","last_became_active":"2023-09-30T01:33:03.295200+0000","last_became_peered":"2023-09-30T01:33:03.295200+0000","last_unstale":"2023-09-30T01:33:05.391246+0000","last_undegraded":"2023-09-30T01:33:05.391246+0000","last_fullsized":"2023-09-30T01:33:05.391246+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T01:33:02.252599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T01:33:02.252599+0000","last_clean_scrub_stamp":"2023-09-30T01:33:02.252599+0000","objects_scrubbed":0,"log_size":117,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T01:51:03.655484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-09-30T01:33:03.296258+0000","last_change":"2023-09-30T01:33:03.293385+0000","last_active":"2023-09-30T01:33:03.296258+0000","last_peered":"2023-09-30T01:33:03.296258+0000","last_clean":"2023-09-30T01:33:03.296258+0000","last_became_active":"2023-09-30T01:33:03.293004+0000","last_became_peered":"2023-09-30T01:33:03.293004+0000","last_unstale":"2023-09-30T01:33:03.296258+0000","last_undegraded":"2023-09-30T01:33:03.296258+0000","last_fullsized":"2023-09-30T01:33:03.296258+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T01:33:02.252599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T01:33:02.252599+0000","last_clean_scrub_stamp":"2023-09-30T01:33:02.252599+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T03:32:32.404638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-09-30T01:33:03.299431+0000","last_change":"2023-09-30T01:33:03.296945+0000","last_active":"2023-09-30T01:33:03.299431+0000","last_peered":"2023-09-30T01:33:03.299431+0000","last_clean":"2023-09-30T01:33:03.299431+0000","last_became_active":"2023-09-30T01:33:03.296465+0000","last_became_peered":"2023-09-30T01:33:03.296465+0000","last_unstale":"2023-09-30T01:33:03.299431+0000","last_undegraded":"2023-09-30T01:33:03.299431+0000","last_fullsized":"2023-09-30T01:33:03.299431+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T01:33:02.252599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T01:33:02.252599+0000","last_clean_scrub_stamp":"2023-09-30T01:33:02.252599+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T05:32:13.855128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-09-30T01:33:03.296569+0000","last_change":"2023-09-30T01:33:03.291527+0000","last_active":"2023-09-30T01:33:03.296569+0000","last_peered":"2023-09-30T01:33:03.296569+0000","last_clean":"2023-09-30T01:33:03.296569+0000","last_became_active":"2023-09-30T01:33:03.291084+0000","last_became_peered":"2023-09-30T01:33:03.291084+0000","last_unstale":"2023-09-30T01:33:03.296569+0000","last_undegraded":"2023-09-30T01:33:03.296569+0000","last_fullsized":"2023-09-30T01:33:03.296569+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T01:33:02.252599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T01:33:02.252599+0000","last_clean_scrub_stamp":"2023-09-30T01:33:02.252599+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T12:10:57.124090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-09-30T01:33:03.299303+0000","last_change":"2023-09-30T01:33:03.296313+0000","last_active":"2023-09-30T01:33:03.299303+0000","last_peered":"2023-09-30T01:33:03.299303+0000","last_clean":"2023-09-30T01:33:03.299303+0000","last_became_active":"2023-09-30T01:33:03.294604+0000","last_became_peered":"2023-09-30T01:33:03.294604+0000","last_unstale":"2023-09-30T01:33:03.299303+0000","last_undegraded":"2023-09-30T01:33:03.299303+0000","last_fullsized":"2023-09-30T01:33:03.299303+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T01:33:02.252599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T01:33:02.252599+0000","last_clean_scrub_stamp":"2023-09-30T01:33:02.252599+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T06:00:29.060679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":1057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":6}],"osd_stats":[{"osd":5,"up_from":11,"seq":47244640319,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5128,"kb_used_data":128,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":94366712,"statfs":{"total":96636764160,"available":96631513088,"internally_reserved":0,"allocated":131072,"data_stored":37781,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 01:38:05 2023","interfaces":[{"interface":"back","average":{"1min":0.76700000000000002,"5min":0.85799999999999998,"15min":0.79400000000000004},"min":{"1min":0.46999999999999997,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":1.028,"5min":8.109,"15min":8.109},"last":0.88500000000000001},{"interface":"front","average":{"1min":0.82199999999999995,"5min":0.88600000000000001,"15min":0.85999999999999999},"min":{"1min":0.57699999999999996,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":1.0009999999999999,"5min":8.3780000000000001,"15min":8.3780000000000001},"last":0.91500000000000004}]},{"osd":1,"last update":"Sat Sep 30 01:38:05 2023","interfaces":[{"interface":"back","average":{"1min":0.751,"5min":0.78300000000000003,"15min":0.77700000000000002},"min":{"1min":0.628,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":0.96999999999999997,"5min":1.087,"15min":1.087},"last":0.85499999999999998},{"interface":"front","average":{"1min":0.82899999999999996,"5min":0.81399999999999995,"15min":0.81699999999999995},"min":{"1min":0.67100000000000004,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.97899999999999998,"5min":1.5289999999999999,"15min":1.5289999999999999},"last":0.73999999999999999}]},{"osd":2,"last update":"Sat Sep 30 01:38:05 2023","interfaces":[{"interface":"back","average":{"1min":0.83799999999999997,"5min":0.878,"15min":0.86799999999999999},"min":{"1min":0.67200000000000004,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":1.0820000000000001,"5min":4.0759999999999996,"15min":4.0759999999999996},"last":0.81100000000000005},{"interface":"front","average":{"1min":0.872,"5min":0.91500000000000004,"15min":0.86799999999999999},"min":{"1min":0.65400000000000003,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":1.0629999999999999,"5min":8.3109999999999999,"15min":8.3109999999999999},"last":0.59599999999999997}]},{"osd":3,"last update":"Sat Sep 30 01:38:05 2023","interfaces":[{"interface":"back","average":{"1min":0.751,"5min":0.751,"15min":0.78700000000000003},"min":{"1min":0.51600000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.95699999999999996,"5min":1.446,"15min":1.446},"last":0.67200000000000004},{"interface":"front","average":{"1min":0.98899999999999999,"5min":0.94299999999999995,"15min":0.95899999999999996},"min":{"1min":0.84299999999999997,"5min":0.53900000000000003,"15min":0.53900000000000003},"max":{"1min":1.1299999999999999,"5min":1.4790000000000001,"15min":1.4790000000000001},"last":0.94199999999999995}]},{"osd":4,"last update":"Sat Sep 30 01:38:05 2023","interfaces":[{"interface":"back","average":{"1min":0.85299999999999998,"5min":0.88400000000000001,"15min":0.84199999999999997},"min":{"1min":0.626,"5min":0.441,"15min":0.441},"max":{"1min":1.3109999999999999,"5min":8.2289999999999992,"15min":8.2289999999999992},"last":0.70499999999999996},{"interface":"front","average":{"1min":0.85199999999999998,"5min":0.82199999999999995,"15min":0.83399999999999996},"min":{"1min":0.58999999999999997,"5min":0.52100000000000002,"15min":0.52100000000000002},"max":{"1min":1,"5min":1.2310000000000001,"15min":1.2310000000000001},"last":0.76800000000000002}]}]},{"osd":4,"up_from":11,"seq":47244640319,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5128,"kb_used_data":128,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":94366712,"statfs":{"total":96636764160,"available":96631513088,"internally_reserved":0,"allocated":131072,"data_stored":37781,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 01:37:10 2023","interfaces":[{"interface":"back","average":{"1min":0.82399999999999995,"5min":0.77300000000000002,"15min":0.76900000000000002},"min":{"1min":0.59299999999999997,"5min":0.52500000000000002,"15min":0.52500000000000002},"max":{"1min":1.0169999999999999,"5min":1.0169999999999999,"15min":1.0169999999999999},"last":0.68500000000000005},{"interface":"front","average":{"1min":0.76500000000000001,"5min":0.80000000000000004,"15min":0.80000000000000004},"min":{"1min":0.58899999999999997,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":1.022,"5min":1.6439999999999999,"15min":1.6439999999999999},"last":0.878}]},{"osd":1,"last update":"Sat Sep 30 01:37:10 2023","interfaces":[{"interface":"back","average":{"1min":0.77100000000000002,"5min":0.78700000000000003,"15min":0.78100000000000003},"min":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"max":{"1min":1.028,"5min":1.028,"15min":1.028},"last":0.58699999999999997},{"interface":"front","average":{"1min":0.81200000000000006,"5min":0.79400000000000004,"15min":0.78500000000000003},"min":{"1min":0.59799999999999998,"5min":0.55800000000000005,"15min":0.55800000000000005},"max":{"1min":0.95999999999999996,"5min":1.548,"15min":1.548},"last":0.76300000000000001}]},{"osd":2,"last update":"Sat Sep 30 01:37:10 2023","interfaces":[{"interface":"back","average":{"1min":0.84499999999999997,"5min":0.84899999999999998,"15min":0.86499999999999999},"min":{"1min":0.71299999999999997,"5min":0.625,"15min":0.625},"max":{"1min":1.0449999999999999,"5min":1.1950000000000001,"15min":1.1950000000000001},"last":0.85099999999999998},{"interface":"front","average":{"1min":0.90200000000000002,"5min":0.92000000000000004,"15min":0.93400000000000005},"min":{"1min":0.67500000000000004,"5min":0.64800000000000002,"15min":0.64800000000000002},"max":{"1min":1.0469999999999999,"5min":1.165,"15min":1.165},"last":0.92100000000000004}]},{"osd":3,"last update":"Sat Sep 30 01:37:10 2023","interfaces":[{"interface":"back","average":{"1min":0.72799999999999998,"5min":0.73199999999999998,"15min":0.73799999999999999},"min":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"max":{"1min":0.996,"5min":1.5,"15min":1.5},"last":0.55300000000000005},{"interface":"front","average":{"1min":0.80800000000000005,"5min":0.80100000000000005,"15min":0.81699999999999995},"min":{"1min":0.61099999999999999,"5min":0.54700000000000004,"15min":0.54700000000000004},"max":{"1min":1.1020000000000001,"5min":1.405,"15min":1.405},"last":0.80800000000000005}]},{"osd":5,"last update":"Sat Sep 30 01:37:10 2023","interfaces":[{"interface":"back","average":{"1min":0.80900000000000005,"5min":0.79000000000000004,"15min":0.77900000000000003},"min":{"1min":0.56000000000000005,"5min":0.54800000000000004,"15min":0.54800000000000004},"max":{"1min":1.0860000000000001,"5min":1.7270000000000001,"15min":1.7270000000000001},"last":0.626},{"interface":"front","average":{"1min":0.90600000000000003,"5min":0.90300000000000002,"15min":0.90500000000000003},"min":{"1min":0.80900000000000005,"5min":0.66800000000000004,"15min":0.66800000000000004},"max":{"1min":1.089,"5min":1.0960000000000001,"15min":1.0960000000000001},"last":0.72699999999999998}]}]},{"osd":3,"up_from":11,"seq":47244640320,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6096,"kb_used_data":648,"kb_used_omap":0,"kb_used_meta":5440,"kb_avail":94365744,"statfs":{"total":96636764160,"available":96630521856,"internally_reserved":0,"allocated":663552,"data_stored":566709,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5570560},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 01:37:03 2023","interfaces":[{"interface":"back","average":{"1min":0.83499999999999996,"5min":0.78300000000000003,"15min":0.77300000000000002},"min":{"1min":0.48299999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":1.052,"5min":1.052,"15min":1.052},"last":0.92500000000000004},{"interface":"front","average":{"1min":0.751,"5min":0.75,"15min":0.73499999999999999},"min":{"1min":0.61199999999999999,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":0.96799999999999997,"5min":1.206,"15min":1.206},"last":0.627}]},{"osd":1,"last update":"Sat Sep 30 01:37:03 2023","interfaces":[{"interface":"back","average":{"1min":0.84099999999999997,"5min":0.83099999999999996,"15min":0.82899999999999996},"min":{"1min":0.59799999999999998,"5min":0.56599999999999995,"15min":0.56599999999999995},"max":{"1min":0.96899999999999997,"5min":1.159,"15min":1.159},"last":0.876},{"interface":"front","average":{"1min":0.80000000000000004,"5min":0.77700000000000002,"15min":0.76100000000000001},"min":{"1min":0.56799999999999995,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":1.016,"5min":1.137,"15min":1.137},"last":0.84399999999999997}]},{"osd":2,"last update":"Sat Sep 30 01:37:03 2023","interfaces":[{"interface":"back","average":{"1min":0.86099999999999999,"5min":0.90700000000000003,"15min":0.93999999999999995},"min":{"1min":0.67700000000000005,"5min":0.60599999999999998,"15min":0.60599999999999998},"max":{"1min":1.0269999999999999,"5min":1.296,"15min":1.296},"last":0.98199999999999998},{"interface":"front","average":{"1min":0.88300000000000001,"5min":0.90700000000000003,"15min":0.92500000000000004},"min":{"1min":0.76300000000000001,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":1.0009999999999999,"5min":1.46,"15min":1.46},"last":0.69399999999999995}]},{"osd":4,"last update":"Sat Sep 30 01:37:03 2023","interfaces":[{"interface":"back","average":{"1min":0.749,"5min":0.73699999999999999,"15min":0.73499999999999999},"min":{"1min":0.57399999999999995,"5min":0.51300000000000001,"15min":0.51300000000000001},"max":{"1min":1.0389999999999999,"5min":1.3220000000000001,"15min":1.3220000000000001},"last":0.56399999999999995},{"interface":"front","average":{"1min":0.79200000000000004,"5min":0.84199999999999997,"15min":0.876},"min":{"1min":0.61799999999999999,"5min":0.56299999999999994,"15min":0.56299999999999994},"max":{"1min":1.046,"5min":1.0800000000000001,"15min":1.0800000000000001},"last":0.71599999999999997}]},{"osd":5,"last update":"Sat Sep 30 01:37:03 2023","interfaces":[{"interface":"back","average":{"1min":0.81899999999999995,"5min":0.81499999999999995,"15min":0.82799999999999996},"min":{"1min":0.64400000000000002,"5min":0.61099999999999999,"15min":0.61099999999999999},"max":{"1min":0.95499999999999996,"5min":1.3320000000000001,"15min":1.3320000000000001},"last":0.76500000000000001},{"interface":"front","average":{"1min":0.79700000000000004,"5min":0.84699999999999998,"15min":0.871},"min":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"max":{"1min":1.0740000000000001,"5min":1.387,"15min":1.387},"last":0.80500000000000005}]}]},{"osd":2,"up_from":11,"seq":47244640321,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5128,"kb_used_data":128,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":94366712,"statfs":{"total":96636764160,"available":96631513088,"internally_reserved":0,"allocated":131072,"data_stored":37781,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 01:37:06 2023","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.57599999999999996,"15min":0.56399999999999995},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.95099999999999996,"5min":0.95099999999999996,"15min":0.95099999999999996},"last":0.49299999999999999},{"interface":"front","average":{"1min":0.70899999999999996,"5min":0.67600000000000005,"15min":0.65700000000000003},"min":{"1min":0.46500000000000002,"5min":0.439,"15min":0.439},"max":{"1min":2.2029999999999998,"5min":2.2029999999999998,"15min":2.2029999999999998},"last":0.57599999999999996}]},{"osd":1,"last update":"Sat Sep 30 01:37:06 2023","interfaces":[{"interface":"back","average":{"1min":0.71799999999999997,"5min":0.70399999999999996,"15min":0.70799999999999996},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":2.2490000000000001,"5min":2.2490000000000001,"15min":2.2490000000000001},"last":0.69899999999999995},{"interface":"front","average":{"1min":0.64300000000000002,"5min":0.64600000000000002,"15min":0.65500000000000003},"min":{"1min":0.46899999999999997,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.85799999999999998,"5min":1.0409999999999999,"15min":1.0409999999999999},"last":0.73099999999999998}]},{"osd":3,"last update":"Sat Sep 30 01:37:06 2023","interfaces":[{"interface":"back","average":{"1min":0.77100000000000002,"5min":0.82899999999999996,"15min":0.83599999999999997},"min":{"1min":0.61199999999999999,"5min":0.61199999999999999,"15min":0.61199999999999999},"max":{"1min":1.0660000000000001,"5min":1.343,"15min":1.343},"last":0.66700000000000004},{"interface":"front","average":{"1min":0.96299999999999997,"5min":0.92200000000000004,"15min":0.92500000000000004},"min":{"1min":0.76700000000000002,"5min":0.60499999999999998,"15min":0.60499999999999998},"max":{"1min":2.4390000000000001,"5min":2.4390000000000001,"15min":2.4390000000000001},"last":0.93999999999999995}]},{"osd":4,"last update":"Sat Sep 30 01:37:06 2023","interfaces":[{"interface":"back","average":{"1min":0.94699999999999995,"5min":0.90000000000000002,"15min":0.90800000000000003},"min":{"1min":0.72699999999999998,"5min":0.64300000000000002,"15min":0.64300000000000002},"max":{"1min":2.3679999999999999,"5min":2.3679999999999999,"15min":2.3679999999999999},"last":0.83099999999999996},{"interface":"front","average":{"1min":0.85699999999999998,"5min":0.84799999999999998,"15min":0.84799999999999998},"min":{"1min":0.68500000000000005,"5min":0.505,"15min":0.505},"max":{"1min":1.401,"5min":1.401,"15min":1.401},"last":0.85299999999999998}]},{"osd":5,"last update":"Sat Sep 30 01:37:06 2023","interfaces":[{"interface":"back","average":{"1min":0.85199999999999998,"5min":0.86199999999999999,"15min":0.86599999999999999},"min":{"1min":0.628,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":1.0629999999999999,"5min":1.3959999999999999,"15min":1.3959999999999999},"last":0.77600000000000002},{"interface":"front","average":{"1min":0.88900000000000001,"5min":0.91500000000000004,"15min":0.92700000000000005},"min":{"1min":0.71099999999999997,"5min":0.61799999999999999,"15min":0.61799999999999999},"max":{"1min":1.234,"5min":1.625,"15min":1.625},"last":0.90700000000000003}]}]},{"osd":1,"up_from":11,"seq":47244640320,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6096,"kb_used_data":648,"kb_used_omap":0,"kb_used_meta":5440,"kb_avail":94365744,"statfs":{"total":96636764160,"available":96630521856,"internally_reserved":0,"allocated":663552,"data_stored":566709,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5570560},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 01:37:07 2023","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.57299999999999995,"15min":0.57199999999999995},"min":{"1min":0.48999999999999999,"5min":0.317,"15min":0.317},"max":{"1min":0.86299999999999999,"5min":0.92400000000000004,"15min":0.92400000000000004},"last":0.54300000000000004},{"interface":"front","average":{"1min":0.76800000000000002,"5min":0.65300000000000002,"15min":0.65200000000000002},"min":{"1min":0.47499999999999998,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":2.085,"5min":2.085,"15min":2.085},"last":0.71599999999999997}]},{"osd":2,"last update":"Sat Sep 30 01:37:07 2023","interfaces":[{"interface":"back","average":{"1min":0.64800000000000002,"5min":0.64400000000000002,"15min":0.66900000000000004},"min":{"1min":0.442,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.80900000000000005,"5min":1.159,"15min":1.159},"last":0.58199999999999996},{"interface":"front","average":{"1min":0.749,"5min":0.69899999999999995,"15min":0.70099999999999996},"min":{"1min":0.58799999999999997,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":1.9930000000000001,"5min":1.9930000000000001,"15min":1.9930000000000001},"last":0.66800000000000004}]},{"osd":3,"last update":"Sat Sep 30 01:37:07 2023","interfaces":[{"interface":"back","average":{"1min":0.81599999999999995,"5min":0.82199999999999995,"15min":0.82499999999999996},"min":{"1min":0.54500000000000004,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":0.98899999999999999,"5min":1.1499999999999999,"15min":1.1499999999999999},"last":0.80200000000000005},{"interface":"front","average":{"1min":0.83599999999999997,"5min":0.83299999999999996,"15min":0.84099999999999997},"min":{"1min":0.69999999999999996,"5min":0.61699999999999999,"15min":0.61699999999999999},"max":{"1min":1.024,"5min":1.2929999999999999,"15min":1.2929999999999999},"last":0.88800000000000001}]},{"osd":4,"last update":"Sat Sep 30 01:37:07 2023","interfaces":[{"interface":"back","average":{"1min":0.84899999999999998,"5min":0.86399999999999999,"15min":0.88100000000000001},"min":{"1min":0.60399999999999998,"5min":0.53600000000000003,"15min":0.53600000000000003},"max":{"1min":0.97499999999999998,"5min":1.415,"15min":1.415},"last":0.83199999999999996},{"interface":"front","average":{"1min":0.878,"5min":0.876,"15min":0.88700000000000001},"min":{"1min":0.59299999999999997,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":1.004,"5min":1.381,"15min":1.381},"last":0.94499999999999995}]},{"osd":5,"last update":"Sat Sep 30 01:37:07 2023","interfaces":[{"interface":"back","average":{"1min":0.93300000000000005,"5min":0.90600000000000003,"15min":0.90500000000000003},"min":{"1min":0.77400000000000002,"5min":0.66200000000000003,"15min":0.66200000000000003},"max":{"1min":1.294,"5min":1.395,"15min":1.395},"last":1.0109999999999999},{"interface":"front","average":{"1min":0.98199999999999998,"5min":0.95799999999999996,"15min":0.96299999999999997},"min":{"1min":0.753,"5min":0.56299999999999994,"15min":0.56299999999999994},"max":{"1min":1.349,"5min":1.4179999999999999,"15min":1.4179999999999999},"last":1.077}]}]},{"osd":0,"up_from":11,"seq":47244640321,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5128,"kb_used_data":128,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":94366712,"statfs":{"total":96636764160,"available":96631513088,"internally_reserved":0,"allocated":131072,"data_stored":37781,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Sep 30 01:37:05 2023","interfaces":[{"interface":"back","average":{"1min":0.57399999999999995,"5min":0.58299999999999996,"15min":0.59299999999999997},"min":{"1min":0.48899999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.69099999999999995,"5min":0.92900000000000005,"15min":0.92900000000000005},"last":0.66700000000000004},{"interface":"front","average":{"1min":0.67400000000000004,"5min":0.64300000000000002,"15min":0.64100000000000001},"min":{"1min":0.42599999999999999,"5min":0.314,"15min":0.314},"max":{"1min":0.98599999999999999,"5min":0.98599999999999999,"15min":0.98599999999999999},"last":0.54000000000000004}]},{"osd":2,"last update":"Sat Sep 30 01:37:05 2023","interfaces":[{"interface":"back","average":{"1min":0.67700000000000005,"5min":0.628,"15min":0.61899999999999999},"min":{"1min":0.53000000000000003,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.93799999999999994,"5min":0.93799999999999994,"15min":0.93799999999999994},"last":0.86299999999999999},{"interface":"front","average":{"1min":0.754,"5min":0.72199999999999998,"15min":0.70799999999999996},"min":{"1min":0.56499999999999995,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":1.2669999999999999,"5min":2.069,"15min":2.069},"last":0.63700000000000001}]},{"osd":3,"last update":"Sat Sep 30 01:37:05 2023","interfaces":[{"interface":"back","average":{"1min":0.80200000000000005,"5min":0.84299999999999997,"15min":0.85599999999999998},"min":{"1min":0.67600000000000005,"5min":0.60299999999999998,"15min":0.60299999999999998},"max":{"1min":0.89800000000000002,"5min":1.1319999999999999,"15min":1.1319999999999999},"last":0.84099999999999997},{"interface":"front","average":{"1min":0.88600000000000001,"5min":0.88300000000000001,"15min":0.88},"min":{"1min":0.68400000000000005,"5min":0.62,"15min":0.62},"max":{"1min":1.0649999999999999,"5min":2.4870000000000001,"15min":2.4870000000000001},"last":1.024}]},{"osd":4,"last update":"Sat Sep 30 01:37:05 2023","interfaces":[{"interface":"back","average":{"1min":0.92600000000000005,"5min":0.92400000000000004,"15min":0.92600000000000005},"min":{"1min":0.74299999999999999,"5min":0.66300000000000003,"15min":0.66300000000000003},"max":{"1min":1.3,"5min":2.589,"15min":2.589},"last":0.78700000000000003},{"interface":"front","average":{"1min":0.86799999999999999,"5min":0.878,"15min":0.90800000000000003},"min":{"1min":0.70099999999999996,"5min":0.54900000000000004,"15min":0.54900000000000004},"max":{"1min":1.2170000000000001,"5min":1.413,"15min":1.413},"last":0.93999999999999995}]},{"osd":5,"last update":"Sat Sep 30 01:37:05 2023","interfaces":[{"interface":"back","average":{"1min":0.92800000000000005,"5min":0.90900000000000003,"15min":0.92900000000000005},"min":{"1min":0.68000000000000005,"5min":0.58799999999999997,"15min":0.58799999999999997},"max":{"1min":1.1599999999999999,"5min":1.3200000000000001,"15min":1.3200000000000001},"last":0.81000000000000005},{"interface":"front","average":{"1min":0.97299999999999998,"5min":0.96399999999999997,"15min":0.96899999999999997},"min":{"1min":0.73999999999999999,"5min":0.73999999999999999,"15min":0.73999999999999999},"max":{"1min":1.3879999999999999,"5min":2.4279999999999999,"15min":2.4279999999999999},"last":0.99199999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-09-30T01:38:09.860 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-09-30T01:38:09.964 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.963+0000 7fd625ef4700 10 monclient: get_monmap_and_config 2023-09-30T01:38:09.964 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.963+0000 7fd625ef4700 10 monclient: build_initial_monmap 2023-09-30T01:38:09.964 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.963+0000 7fd625ef4700 10 monclient: monmap: 2023-09-30T01:38:09.965 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2023-09-30T01:38:09.965 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:38:09.965 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:38:09.963994+0000 2023-09-30T01:38:09.965 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:38:09.963994+0000 2023-09-30T01:38:09.965 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2023-09-30T01:38:09.965 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:38:09.965 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a-legacy 2023-09-30T01:38:09.965 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.107:6789/0 mon.noname-f-legacy 2023-09-30T01:38:09.965 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.noname-b 2023-09-30T01:38:09.965 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.noname-g 2023-09-30T01:38:09.965 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.noname-c 2023-09-30T01:38:09.965 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.noname-h 2023-09-30T01:38:09.966 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.noname-d 2023-09-30T01:38:09.966 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.noname-i 2023-09-30T01:38:09.966 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.noname-e 2023-09-30T01:38:09.966 INFO:teuthology.orchestra.run.smithi088.stderr:9: v2:172.21.15.88:3300/0 mon.noname-a 2023-09-30T01:38:09.966 INFO:teuthology.orchestra.run.smithi088.stderr:10: v2:172.21.15.107:3300/0 mon.noname-f 2023-09-30T01:38:09.966 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:38:09.966 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.963+0000 7fd625ef4700 1 Processor -- start 2023-09-30T01:38:09.966 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.963+0000 7fd625ef4700 1 -- start start 2023-09-30T01:38:09.966 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.963+0000 7fd625ef4700 10 monclient: init 2023-09-30T01:38:09.974 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.963+0000 7fd625ef4700 10 monclient: _reopen_session rank -1 2023-09-30T01:38:09.974 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.963+0000 7fd625ef4700 10 monclient: _add_conns ranks=[6,10,7,9,3,5,0,1,4,8,2] 2023-09-30T01:38:09.974 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.963+0000 7fd625ef4700 1 --2- >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7fd620097a40 0x7fd620132700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:38:09.974 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.963+0000 7fd625ef4700 10 monclient(hunting): picked mon.noname-d con 0x7fd620097a40 addr [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] 2023-09-30T01:38:09.974 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.963+0000 7fd625ef4700 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fd6201311c0 0x7fd620132c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:38:09.974 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.963+0000 7fd625ef4700 10 monclient(hunting): picked mon.noname-f con 0x7fd6201311c0 addr v2:172.21.15.107:3300/0 2023-09-30T01:38:09.975 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.963+0000 7fd625ef4700 1 --2- >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7fd6201332e0 0x7fd620133740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:38:09.975 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.963+0000 7fd625ef4700 10 monclient(hunting): picked mon.noname-i con 0x7fd6201332e0 addr [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] 2023-09-30T01:38:09.975 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.963+0000 7fd625ef4700 10 monclient(hunting): start opening mon connection 2023-09-30T01:38:09.975 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.963+0000 7fd625ef4700 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd620131590 con 0x7fd6201311c0 2023-09-30T01:38:09.975 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.963+0000 7fd625ef4700 10 monclient(hunting): start opening mon connection 2023-09-30T01:38:09.975 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.963+0000 7fd625ef4700 1 -- --> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] -- mon_getmap magic: 0 v1 -- 0x7fd62009a5d0 con 0x7fd620097a40 2023-09-30T01:38:09.975 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.963+0000 7fd625ef4700 10 monclient(hunting): start opening mon connection 2023-09-30T01:38:09.975 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.963+0000 7fd625ef4700 1 -- --> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] -- mon_getmap magic: 0 v1 -- 0x7fd620097e10 con 0x7fd6201332e0 2023-09-30T01:38:09.975 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.963+0000 7fd625ef4700 10 monclient(hunting): _renew_subs 2023-09-30T01:38:09.975 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.963+0000 7fd625ef4700 10 monclient(hunting): authenticate will time out at 1162.001414s 2023-09-30T01:38:09.975 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.963+0000 7fd624ef2700 1 --2- >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7fd620097a40 0x7fd620132700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:38:09.975 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.963+0000 7fd624ef2700 1 --2- >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7fd620097a40 0x7fd620132700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3303/0 says I am v2:172.21.15.88:49756/0 (socket says 172.21.15.88:49756) 2023-09-30T01:38:09.976 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.963+0000 7fd624ef2700 1 -- 172.21.15.88:0/2432767395 learned_addr learned my addr 172.21.15.88:0/2432767395 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-09-30T01:38:09.976 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.963+0000 7fd624ef2700 10 monclient(hunting): get_auth_request con 0x7fd620097a40 auth_method 0 2023-09-30T01:38:09.976 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.963+0000 7fd624ef2700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:38:09.976 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.963+0000 7fd624ef2700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:38:09.976 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.963+0000 7fd624ef2700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:38:09.976 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.963+0000 7fd624ef2700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:38:09.976 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.963+0000 7fd624ef2700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:38:09.976 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.963+0000 7fd624ef2700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:38:09.976 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.963+0000 7fd61ffff700 1 --2- 172.21.15.88:0/2432767395 >> v2:172.21.15.107:3300/0 conn(0x7fd6201311c0 0x7fd620132c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:38:09.976 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.963+0000 7fd6256f3700 1 --2- 172.21.15.88:0/2432767395 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7fd6201332e0 0x7fd620133740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:38:09.976 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.963+0000 7fd61ffff700 10 monclient(hunting): get_auth_request con 0x7fd6201311c0 auth_method 0 2023-09-30T01:38:09.976 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.963+0000 7fd61ffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:38:09.976 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.963+0000 7fd61ffff700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:38:09.977 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.963+0000 7fd61ffff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:38:09.977 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.963+0000 7fd6256f3700 10 monclient(hunting): get_auth_request con 0x7fd6201332e0 auth_method 0 2023-09-30T01:38:09.977 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.963+0000 7fd6256f3700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:38:09.977 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.963+0000 7fd6256f3700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:38:09.977 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.963+0000 7fd6256f3700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:38:09.977 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.963+0000 7fd61ffff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:38:09.977 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.963+0000 7fd61ffff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:38:09.977 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.963+0000 7fd61ffff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:38:09.977 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.963+0000 7fd624ef2700 10 monclient(hunting): handle_auth_done global_id 4785 payload 274 2023-09-30T01:38:09.977 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.963+0000 7fd624ef2700 1 -- 172.21.15.88:0/2432767395 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7fd6201332e0 msgr2=0x7fd620133740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:38:09.977 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.963+0000 7fd624ef2700 1 --2- 172.21.15.88:0/2432767395 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7fd6201332e0 0x7fd620133740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:38:09.977 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.963+0000 7fd624ef2700 1 -- 172.21.15.88:0/2432767395 >> v2:172.21.15.107:3300/0 conn(0x7fd6201311c0 msgr2=0x7fd620132c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:38:09.978 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.963+0000 7fd624ef2700 1 --2- 172.21.15.88:0/2432767395 >> v2:172.21.15.107:3300/0 conn(0x7fd6201311c0 0x7fd620132c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:38:09.978 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.963+0000 7fd624ef2700 10 monclient: _finish_hunting 0 2023-09-30T01:38:09.978 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.963+0000 7fd624ef2700 1 monclient: found mon.noname-d 2023-09-30T01:38:09.978 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.963+0000 7fd624ef2700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:38:09.978 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.963+0000 7fd624ef2700 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.88:3303/0 2023-09-30T01:38:09.978 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.963+0000 7fd624ef2700 1 -- 172.21.15.88:0/2432767395 --> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd62004bb20 con 0x7fd620097a40 2023-09-30T01:38:09.978 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.963+0000 7fd624ef2700 1 --2- 172.21.15.88:0/2432767395 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7fd620097a40 0x7fd620132700 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fd618009f20 tx=0x7fd618009840 comp rx=0 tx=0).ready entity=mon.6 client_cookie=dcdb23717a7ec657 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:38:09.978 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.963+0000 7fd61f7fe700 1 -- 172.21.15.88:0/2432767395 <== mon.6 v2:172.21.15.88:3303/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7fd61800e070 con 0x7fd620097a40 2023-09-30T01:38:09.979 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.963+0000 7fd61f7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:38:09.979 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.963+0000 7fd61f7fe700 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2023-09-30T01:38:09.979 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.963+0000 7fd61f7fe700 10 monclient: dump: 2023-09-30T01:38:09.979 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:38:09.979 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:38:09.979 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:38:09.979 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:38:09.979 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:38:09.979 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:38:09.979 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:38:09.979 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:38:09.979 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:38:09.980 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:38:09.980 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:38:09.980 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:38:09.980 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:38:09.980 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:38:09.980 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:38:09.980 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:38:09.980 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.963+0000 7fd61f7fe700 10 monclient: _finish_auth 0 2023-09-30T01:38:09.980 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.963+0000 7fd61f7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:38:09.980 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.963+0000 7fd625ef4700 5 monclient: authenticate success, global_id 4785 2023-09-30T01:38:09.980 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.963+0000 7fd625ef4700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:38:09.980 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.963+0000 7fd61f7fe700 1 -- 172.21.15.88:0/2432767395 <== mon.6 v2:172.21.15.88:3303/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd61800be50 con 0x7fd620097a40 2023-09-30T01:38:09.980 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.963+0000 7fd61f7fe700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:38:09.981 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.963+0000 7fd625ef4700 10 monclient: get_monmap_and_config success 2023-09-30T01:38:09.981 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.963+0000 7fd625ef4700 4 set_mon_vals no callback set 2023-09-30T01:38:09.981 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.963+0000 7fd61f7fe700 1 -- 172.21.15.88:0/2432767395 <== mon.6 v2:172.21.15.88:3303/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7fd6180092e0 con 0x7fd620097a40 2023-09-30T01:38:09.981 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.963+0000 7fd61f7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:38:09.981 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.963+0000 7fd61f7fe700 10 monclient: got monmap 1 from mon.d (according to old e1) 2023-09-30T01:38:09.981 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.963+0000 7fd61f7fe700 10 monclient: dump: 2023-09-30T01:38:09.981 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:38:09.981 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:38:09.981 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:38:09.981 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:38:09.981 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:38:09.981 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:38:09.982 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:38:09.982 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:38:09.982 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:38:09.982 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:38:09.982 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:38:09.982 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:38:09.982 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:38:09.982 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:38:09.982 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:38:09.982 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:38:09.982 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.963+0000 7fd625ef4700 10 monclient: shutdown 2023-09-30T01:38:09.982 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.963+0000 7fd625ef4700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:38:09.983 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.963+0000 7fd625ef4700 1 -- 172.21.15.88:0/2432767395 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7fd620097a40 msgr2=0x7fd620132700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:38:09.983 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.963+0000 7fd625ef4700 1 --2- 172.21.15.88:0/2432767395 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7fd620097a40 0x7fd620132700 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fd618009f20 tx=0x7fd618009840 comp rx=0 tx=0).stop 2023-09-30T01:38:09.983 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.963+0000 7fd625ef4700 1 -- 172.21.15.88:0/2432767395 shutdown_connections 2023-09-30T01:38:09.983 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.963+0000 7fd625ef4700 1 --2- 172.21.15.88:0/2432767395 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7fd6201332e0 0x7fd620133740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:38:09.983 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.963+0000 7fd625ef4700 1 --2- 172.21.15.88:0/2432767395 >> v2:172.21.15.107:3300/0 conn(0x7fd6201311c0 0x7fd620132c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:38:09.983 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.963+0000 7fd625ef4700 1 --2- 172.21.15.88:0/2432767395 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7fd620097a40 0x7fd620132700 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:38:09.983 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.963+0000 7fd625ef4700 1 -- 172.21.15.88:0/2432767395 >> 172.21.15.88:0/2432767395 conn(0x7fd62006e5e0 msgr2=0x7fd62006e9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:38:09.983 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.963+0000 7fd625ef4700 1 -- 172.21.15.88:0/2432767395 shutdown_connections 2023-09-30T01:38:09.983 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.963+0000 7fd625ef4700 1 -- 172.21.15.88:0/2432767395 wait complete. 2023-09-30T01:38:09.983 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.967+0000 7fd625ef4700 10 monclient: build_initial_monmap 2023-09-30T01:38:09.983 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.967+0000 7fd625ef4700 10 monclient: monmap: 2023-09-30T01:38:09.983 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2023-09-30T01:38:09.984 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:38:09.984 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 0.000000 2023-09-30T01:38:09.984 INFO:teuthology.orchestra.run.smithi088.stderr:created 0.000000 2023-09-30T01:38:09.984 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2023-09-30T01:38:09.984 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 1 2023-09-30T01:38:09.984 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.noname-a 2023-09-30T01:38:09.984 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.noname-f 2023-09-30T01:38:09.984 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.noname-b 2023-09-30T01:38:09.984 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.noname-g 2023-09-30T01:38:09.984 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.noname-c 2023-09-30T01:38:09.984 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.noname-h 2023-09-30T01:38:09.984 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.noname-d 2023-09-30T01:38:09.985 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.noname-i 2023-09-30T01:38:09.985 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.noname-e 2023-09-30T01:38:09.985 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:38:09.985 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.967+0000 7fd625ef4700 1 Processor -- start 2023-09-30T01:38:09.985 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.967+0000 7fd625ef4700 1 -- start start 2023-09-30T01:38:09.985 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.967+0000 7fd625ef4700 10 monclient: init 2023-09-30T01:38:09.985 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.967+0000 7fd625ef4700 10 monclient: _reopen_session rank -1 2023-09-30T01:38:09.985 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.967+0000 7fd625ef4700 10 monclient: _add_conns ranks=[6,4,1,3,5,0,2,7,8] 2023-09-30T01:38:09.985 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.967+0000 7fd625ef4700 1 --2- >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7fd620097a40 0x7fd62011d870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:38:09.985 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.967+0000 7fd625ef4700 10 monclient(hunting): picked mon.noname-d con 0x7fd620097a40 addr [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] 2023-09-30T01:38:09.985 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.967+0000 7fd625ef4700 1 --2- >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7fd6201311c0 0x7fd62011ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:38:09.985 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.967+0000 7fd625ef4700 10 monclient(hunting): picked mon.noname-c con 0x7fd6201311c0 addr [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] 2023-09-30T01:38:09.985 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.967+0000 7fd625ef4700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd6201332e0 0x7fd62011e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:38:09.986 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.967+0000 7fd625ef4700 10 monclient(hunting): picked mon.noname-f con 0x7fd6201332e0 addr [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] 2023-09-30T01:38:09.986 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.967+0000 7fd625ef4700 10 monclient(hunting): start opening mon connection 2023-09-30T01:38:09.986 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.967+0000 7fd625ef4700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd620130300 con 0x7fd6201332e0 2023-09-30T01:38:09.986 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.967+0000 7fd625ef4700 10 monclient(hunting): start opening mon connection 2023-09-30T01:38:09.986 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.967+0000 7fd625ef4700 1 -- --> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] -- mon_getmap magic: 0 v1 -- 0x7fd620099c30 con 0x7fd6201311c0 2023-09-30T01:38:09.986 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.967+0000 7fd625ef4700 10 monclient(hunting): start opening mon connection 2023-09-30T01:38:09.986 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.967+0000 7fd625ef4700 1 -- --> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] -- mon_getmap magic: 0 v1 -- 0x7fd6200996b0 con 0x7fd620097a40 2023-09-30T01:38:09.986 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.967+0000 7fd625ef4700 10 monclient(hunting): _renew_subs 2023-09-30T01:38:09.986 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.967+0000 7fd625ef4700 10 monclient(hunting): authenticate will time out at 1162.005794s 2023-09-30T01:38:09.986 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.967+0000 7fd624ef2700 1 --2- >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7fd620097a40 0x7fd62011d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:38:09.986 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.967+0000 7fd624ef2700 1 --2- >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7fd620097a40 0x7fd62011d870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3303/0 says I am v2:172.21.15.88:49760/0 (socket says 172.21.15.88:49760) 2023-09-30T01:38:09.986 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.967+0000 7fd624ef2700 1 -- 172.21.15.88:0/2175160693 learned_addr learned my addr 172.21.15.88:0/2175160693 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-09-30T01:38:09.987 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.967+0000 7fd624ef2700 10 monclient(hunting): get_auth_request con 0x7fd620097a40 auth_method 0 2023-09-30T01:38:09.987 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.967+0000 7fd624ef2700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:38:09.987 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.967+0000 7fd624ef2700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:38:09.987 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.967+0000 7fd624ef2700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:38:09.987 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.967+0000 7fd624ef2700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:38:09.987 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.967+0000 7fd624ef2700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:38:09.987 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.967+0000 7fd624ef2700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:38:09.987 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.967+0000 7fd61ffff700 1 --2- 172.21.15.88:0/2175160693 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7fd6201311c0 0x7fd62011ddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:38:09.987 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.967+0000 7fd61ffff700 10 monclient(hunting): get_auth_request con 0x7fd6201311c0 auth_method 0 2023-09-30T01:38:09.987 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.967+0000 7fd61ffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:38:09.987 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.967+0000 7fd61ffff700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:38:09.987 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.967+0000 7fd61ffff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:38:09.988 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.967+0000 7fd624ef2700 10 monclient(hunting): handle_auth_done global_id 4794 payload 824 2023-09-30T01:38:09.988 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.967+0000 7fd6256f3700 1 --2- 172.21.15.88:0/2175160693 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd6201332e0 0x7fd62011e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:38:09.988 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.967+0000 7fd624ef2700 1 -- 172.21.15.88:0/2175160693 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7fd6201311c0 msgr2=0x7fd62011ddb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:38:09.988 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.967+0000 7fd624ef2700 1 --2- 172.21.15.88:0/2175160693 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7fd6201311c0 0x7fd62011ddb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:38:09.988 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.967+0000 7fd624ef2700 1 -- 172.21.15.88:0/2175160693 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd6201332e0 msgr2=0x7fd62011e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:38:09.988 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.967+0000 7fd624ef2700 1 --2- 172.21.15.88:0/2175160693 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd6201332e0 0x7fd62011e2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:38:09.988 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.967+0000 7fd624ef2700 10 monclient: _finish_hunting 0 2023-09-30T01:38:09.988 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.967+0000 7fd624ef2700 1 monclient: found mon.noname-d 2023-09-30T01:38:09.988 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.967+0000 7fd624ef2700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:38:09.989 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.967+0000 7fd624ef2700 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.88:3303/0 2023-09-30T01:38:09.989 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.967+0000 7fd624ef2700 1 -- 172.21.15.88:0/2175160693 --> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd618009ae0 con 0x7fd620097a40 2023-09-30T01:38:09.989 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.967+0000 7fd61ffff700 1 --2- 172.21.15.88:0/2175160693 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7fd6201311c0 0x7fd62011ddb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-30T01:38:09.989 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.967+0000 7fd624ef2700 1 --2- 172.21.15.88:0/2175160693 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7fd620097a40 0x7fd62011d870 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fd62009a710 tx=0x7fd618018000 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:38:09.989 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.967+0000 7fd61cff9700 1 -- 172.21.15.88:0/2175160693 <== mon.6 v2:172.21.15.88:3303/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7fd618003300 con 0x7fd620097a40 2023-09-30T01:38:09.989 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.967+0000 7fd61cff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:38:09.989 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.967+0000 7fd61cff9700 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2023-09-30T01:38:09.989 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.967+0000 7fd61cff9700 10 monclient: dump: 2023-09-30T01:38:09.989 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:38:09.989 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:38:09.989 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:38:09.989 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:38:09.990 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:38:09.990 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:38:09.990 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:38:09.990 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:38:09.990 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:38:09.990 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:38:09.990 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:38:09.990 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:38:09.990 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:38:09.990 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:38:09.990 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:38:09.990 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:38:09.991 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.967+0000 7fd61cff9700 10 monclient: _finish_auth 0 2023-09-30T01:38:09.991 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.967+0000 7fd61cff9700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:38:09.991 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.967+0000 7fd625ef4700 5 monclient: authenticate success, global_id 4794 2023-09-30T01:38:09.991 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.967+0000 7fd61cff9700 1 -- 172.21.15.88:0/2175160693 <== mon.6 v2:172.21.15.88:3303/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd6180092e0 con 0x7fd620097a40 2023-09-30T01:38:09.991 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.967+0000 7fd625ef4700 10 monclient: _renew_subs 2023-09-30T01:38:09.991 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.967+0000 7fd625ef4700 10 monclient: _send_mon_message to mon.d at v2:172.21.15.88:3303/0 2023-09-30T01:38:09.991 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.967+0000 7fd625ef4700 1 -- 172.21.15.88:0/2175160693 --> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd620120fd0 con 0x7fd620097a40 2023-09-30T01:38:09.991 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.967+0000 7fd61cff9700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:38:09.991 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.967+0000 7fd61cff9700 1 -- 172.21.15.88:0/2175160693 <== mon.6 v2:172.21.15.88:3303/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7fd61800bca0 con 0x7fd620097a40 2023-09-30T01:38:09.991 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.967+0000 7fd61cff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:38:09.991 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.967+0000 7fd61effd700 4 set_mon_vals no callback set 2023-09-30T01:38:09.991 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.967+0000 7fd61cff9700 10 monclient: got monmap 1 from mon.d (according to old e1) 2023-09-30T01:38:09.991 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.967+0000 7fd61cff9700 10 monclient: dump: 2023-09-30T01:38:09.992 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:38:09.992 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:38:09.992 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:38:09.992 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:38:09.992 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:38:09.992 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:38:09.992 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:38:09.992 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:38:09.992 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:38:09.992 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:38:09.992 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:38:09.992 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:38:09.993 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:38:09.993 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:38:09.993 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:38:09.993 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:38:09.993 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.967+0000 7fd625ef4700 10 monclient: _renew_subs 2023-09-30T01:38:09.993 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.967+0000 7fd625ef4700 10 monclient: _send_mon_message to mon.d at v2:172.21.15.88:3303/0 2023-09-30T01:38:09.993 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.967+0000 7fd625ef4700 1 -- 172.21.15.88:0/2175160693 --> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd62011f1c0 con 0x7fd620097a40 2023-09-30T01:38:09.993 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.971+0000 7fd625ef4700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:38:09.993 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.971+0000 7fd625ef4700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:38:09.993 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.971+0000 7fd625ef4700 10 monclient: _send_mon_message to mon.d at v2:172.21.15.88:3303/0 2023-09-30T01:38:09.993 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.971+0000 7fd625ef4700 1 -- 172.21.15.88:0/2175160693 --> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd62009f120 con 0x7fd620097a40 2023-09-30T01:38:09.993 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.971+0000 7fd61cff9700 1 -- 172.21.15.88:0/2175160693 <== mon.6 v2:172.21.15.88:3303/0 4 ==== mgrmap(e 5) v1 ==== 40144+0+0 (secure 0 0 0) 0x7fd618003050 con 0x7fd620097a40 2023-09-30T01:38:09.994 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.971+0000 7fd61cff9700 1 --2- 172.21.15.88:0/2175160693 >> v2:172.21.15.107:6812/29926 conn(0x7fd600033360 0x7fd600035810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:38:09.994 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.971+0000 7fd61cff9700 1 -- 172.21.15.88:0/2175160693 <== mon.6 v2:172.21.15.88:3303/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 3502+0+0 (secure 0 0 0) 0x7fd618044e80 con 0x7fd620097a40 2023-09-30T01:38:09.994 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.971+0000 7fd61ffff700 1 --2- 172.21.15.88:0/2175160693 >> v2:172.21.15.107:6812/29926 conn(0x7fd600033360 0x7fd600035810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:38:09.994 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.971+0000 7fd61ffff700 10 monclient: get_auth_request con 0x7fd600033360 auth_method 0 2023-09-30T01:38:09.994 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.971+0000 7fd61ffff700 1 --2- 172.21.15.88:0/2175160693 >> v2:172.21.15.107:6812/29926 conn(0x7fd600033360 0x7fd600035810 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fd608009d80 tx=0x7fd608008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:38:09.994 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.975+0000 7fd61cff9700 1 -- 172.21.15.88:0/2175160693 <== mon.6 v2:172.21.15.88:3303/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fd618007ca0 con 0x7fd620097a40 2023-09-30T01:38:09.994 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.975+0000 7fd61cff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:38:09.995 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:09.975+0000 7fd61cff9700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:38:10.140 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.139+0000 7fd625ef4700 1 -- 172.21.15.88:0/2175160693 --> v2:172.21.15.107:6812/29926 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fd6200a7050 con 0x7fd600033360 2023-09-30T01:38:10.145 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.143+0000 7fd61cff9700 1 -- 172.21.15.88:0/2175160693 <== mgr.4108 v2:172.21.15.107:6812/29926 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+53337 (secure 0 0 0) 0x7fd6200a7050 con 0x7fd600033360 2023-09-30T01:38:10.146 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-09-30T01:38:10.148 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.143+0000 7fd625ef4700 1 -- 172.21.15.88:0/2175160693 >> v2:172.21.15.107:6812/29926 conn(0x7fd600033360 msgr2=0x7fd600035810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:38:10.148 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.143+0000 7fd625ef4700 1 --2- 172.21.15.88:0/2175160693 >> v2:172.21.15.107:6812/29926 conn(0x7fd600033360 0x7fd600035810 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fd608009d80 tx=0x7fd608008040 comp rx=0 tx=0).stop 2023-09-30T01:38:10.148 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.143+0000 7fd625ef4700 10 monclient: shutdown 2023-09-30T01:38:10.148 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.143+0000 7fd625ef4700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:38:10.148 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.143+0000 7fd625ef4700 1 -- 172.21.15.88:0/2175160693 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7fd620097a40 msgr2=0x7fd62011d870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:38:10.148 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.143+0000 7fd625ef4700 1 --2- 172.21.15.88:0/2175160693 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7fd620097a40 0x7fd62011d870 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fd62009a710 tx=0x7fd618018000 comp rx=0 tx=0).stop 2023-09-30T01:38:10.148 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.147+0000 7fd625ef4700 1 -- 172.21.15.88:0/2175160693 shutdown_connections 2023-09-30T01:38:10.148 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.147+0000 7fd625ef4700 1 --2- 172.21.15.88:0/2175160693 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd6201332e0 0x7fd62011e2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:38:10.148 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.147+0000 7fd625ef4700 1 --2- 172.21.15.88:0/2175160693 >> v2:172.21.15.107:6812/29926 conn(0x7fd600033360 0x7fd600035810 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:38:10.149 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.147+0000 7fd625ef4700 1 --2- 172.21.15.88:0/2175160693 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7fd6201311c0 0x7fd62011ddb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:38:10.149 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.147+0000 7fd625ef4700 1 --2- 172.21.15.88:0/2175160693 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7fd620097a40 0x7fd62011d870 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:38:10.149 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.147+0000 7fd625ef4700 1 -- 172.21.15.88:0/2175160693 >> 172.21.15.88:0/2175160693 conn(0x7fd62006e5e0 msgr2=0x7fd62009b040 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:38:10.149 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.147+0000 7fd625ef4700 1 -- 172.21.15.88:0/2175160693 shutdown_connections 2023-09-30T01:38:10.149 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.147+0000 7fd625ef4700 1 -- 172.21.15.88:0/2175160693 wait complete. 2023-09-30T01:38:10.149 INFO:teuthology.orchestra.run.smithi088.stderr:dumped all 2023-09-30T01:38:10.160 INFO:teuthology.orchestra.run.smithi088.stdout:{"pg_ready":true,"pg_map":{"version":165,"stamp":"2023-09-30T01:38:09.743539+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":32704,"kb_used_data":1808,"kb_used_omap":0,"kb_used_meta":30848,"kb_avail":566198336,"statfs":{"total":579820584960,"available":579787096064,"internally_reserved":0,"allocated":1851392,"data_stored":1284542,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":31588352},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003602"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-09-30T01:33:03.296691+0000","last_change":"2023-09-30T01:33:03.292315+0000","last_active":"2023-09-30T01:33:03.296691+0000","last_peered":"2023-09-30T01:33:03.296691+0000","last_clean":"2023-09-30T01:33:03.296691+0000","last_became_active":"2023-09-30T01:33:03.291889+0000","last_became_peered":"2023-09-30T01:33:03.291889+0000","last_unstale":"2023-09-30T01:33:03.296691+0000","last_undegraded":"2023-09-30T01:33:03.296691+0000","last_fullsized":"2023-09-30T01:33:03.296691+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T01:33:02.252599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T01:33:02.252599+0000","last_clean_scrub_stamp":"2023-09-30T01:33:02.252599+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T11:44:00.574379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-09-30T01:33:03.297822+0000","last_change":"2023-09-30T01:33:03.295081+0000","last_active":"2023-09-30T01:33:03.297822+0000","last_peered":"2023-09-30T01:33:03.297822+0000","last_clean":"2023-09-30T01:33:03.297822+0000","last_became_active":"2023-09-30T01:33:03.294535+0000","last_became_peered":"2023-09-30T01:33:03.294535+0000","last_unstale":"2023-09-30T01:33:03.297822+0000","last_undegraded":"2023-09-30T01:33:03.297822+0000","last_fullsized":"2023-09-30T01:33:03.297822+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T01:33:02.252599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T01:33:02.252599+0000","last_clean_scrub_stamp":"2023-09-30T01:33:02.252599+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T05:49:49.642723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-09-30T01:33:03.300117+0000","last_change":"2023-09-30T01:33:03.297392+0000","last_active":"2023-09-30T01:33:03.300117+0000","last_peered":"2023-09-30T01:33:03.300117+0000","last_clean":"2023-09-30T01:33:03.300117+0000","last_became_active":"2023-09-30T01:33:03.296959+0000","last_became_peered":"2023-09-30T01:33:03.296959+0000","last_unstale":"2023-09-30T01:33:03.300117+0000","last_undegraded":"2023-09-30T01:33:03.300117+0000","last_fullsized":"2023-09-30T01:33:03.300117+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T01:33:02.252599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T01:33:02.252599+0000","last_clean_scrub_stamp":"2023-09-30T01:33:02.252599+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T11:57:13.112607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-09-30T01:33:03.296932+0000","last_change":"2023-09-30T01:33:03.295478+0000","last_active":"2023-09-30T01:33:03.296932+0000","last_peered":"2023-09-30T01:33:03.296932+0000","last_clean":"2023-09-30T01:33:03.296932+0000","last_became_active":"2023-09-30T01:33:03.295106+0000","last_became_peered":"2023-09-30T01:33:03.295106+0000","last_unstale":"2023-09-30T01:33:03.296932+0000","last_undegraded":"2023-09-30T01:33:03.296932+0000","last_fullsized":"2023-09-30T01:33:03.296932+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T01:33:02.252599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T01:33:02.252599+0000","last_clean_scrub_stamp":"2023-09-30T01:33:02.252599+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T07:34:36.123333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"15'117","reported_seq":159,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-09-30T01:33:05.391246+0000","last_change":"2023-09-30T01:33:03.296872+0000","last_active":"2023-09-30T01:33:05.391246+0000","last_peered":"2023-09-30T01:33:05.391246+0000","last_clean":"2023-09-30T01:33:05.391246+0000","last_became_active":"2023-09-30T01:33:03.295200+0000","last_became_peered":"2023-09-30T01:33:03.295200+0000","last_unstale":"2023-09-30T01:33:05.391246+0000","last_undegraded":"2023-09-30T01:33:05.391246+0000","last_fullsized":"2023-09-30T01:33:05.391246+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T01:33:02.252599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T01:33:02.252599+0000","last_clean_scrub_stamp":"2023-09-30T01:33:02.252599+0000","objects_scrubbed":0,"log_size":117,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T01:51:03.655484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-09-30T01:33:03.296258+0000","last_change":"2023-09-30T01:33:03.293385+0000","last_active":"2023-09-30T01:33:03.296258+0000","last_peered":"2023-09-30T01:33:03.296258+0000","last_clean":"2023-09-30T01:33:03.296258+0000","last_became_active":"2023-09-30T01:33:03.293004+0000","last_became_peered":"2023-09-30T01:33:03.293004+0000","last_unstale":"2023-09-30T01:33:03.296258+0000","last_undegraded":"2023-09-30T01:33:03.296258+0000","last_fullsized":"2023-09-30T01:33:03.296258+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T01:33:02.252599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T01:33:02.252599+0000","last_clean_scrub_stamp":"2023-09-30T01:33:02.252599+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T03:32:32.404638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-09-30T01:33:03.299431+0000","last_change":"2023-09-30T01:33:03.296945+0000","last_active":"2023-09-30T01:33:03.299431+0000","last_peered":"2023-09-30T01:33:03.299431+0000","last_clean":"2023-09-30T01:33:03.299431+0000","last_became_active":"2023-09-30T01:33:03.296465+0000","last_became_peered":"2023-09-30T01:33:03.296465+0000","last_unstale":"2023-09-30T01:33:03.299431+0000","last_undegraded":"2023-09-30T01:33:03.299431+0000","last_fullsized":"2023-09-30T01:33:03.299431+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T01:33:02.252599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T01:33:02.252599+0000","last_clean_scrub_stamp":"2023-09-30T01:33:02.252599+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T05:32:13.855128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-09-30T01:33:03.296569+0000","last_change":"2023-09-30T01:33:03.291527+0000","last_active":"2023-09-30T01:33:03.296569+0000","last_peered":"2023-09-30T01:33:03.296569+0000","last_clean":"2023-09-30T01:33:03.296569+0000","last_became_active":"2023-09-30T01:33:03.291084+0000","last_became_peered":"2023-09-30T01:33:03.291084+0000","last_unstale":"2023-09-30T01:33:03.296569+0000","last_undegraded":"2023-09-30T01:33:03.296569+0000","last_fullsized":"2023-09-30T01:33:03.296569+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T01:33:02.252599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T01:33:02.252599+0000","last_clean_scrub_stamp":"2023-09-30T01:33:02.252599+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T12:10:57.124090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-09-30T01:33:03.299303+0000","last_change":"2023-09-30T01:33:03.296313+0000","last_active":"2023-09-30T01:33:03.299303+0000","last_peered":"2023-09-30T01:33:03.299303+0000","last_clean":"2023-09-30T01:33:03.299303+0000","last_became_active":"2023-09-30T01:33:03.294604+0000","last_became_peered":"2023-09-30T01:33:03.294604+0000","last_unstale":"2023-09-30T01:33:03.299303+0000","last_undegraded":"2023-09-30T01:33:03.299303+0000","last_fullsized":"2023-09-30T01:33:03.299303+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T01:33:02.252599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T01:33:02.252599+0000","last_clean_scrub_stamp":"2023-09-30T01:33:02.252599+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T06:00:29.060679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":1057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":6}],"osd_stats":[{"osd":5,"up_from":11,"seq":47244640319,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5128,"kb_used_data":128,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":94366712,"statfs":{"total":96636764160,"available":96631513088,"internally_reserved":0,"allocated":131072,"data_stored":37781,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 01:38:05 2023","interfaces":[{"interface":"back","average":{"1min":0.76700000000000002,"5min":0.85799999999999998,"15min":0.79400000000000004},"min":{"1min":0.46999999999999997,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":1.028,"5min":8.109,"15min":8.109},"last":0.88500000000000001},{"interface":"front","average":{"1min":0.82199999999999995,"5min":0.88600000000000001,"15min":0.85999999999999999},"min":{"1min":0.57699999999999996,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":1.0009999999999999,"5min":8.3780000000000001,"15min":8.3780000000000001},"last":0.91500000000000004}]},{"osd":1,"last update":"Sat Sep 30 01:38:05 2023","interfaces":[{"interface":"back","average":{"1min":0.751,"5min":0.78300000000000003,"15min":0.77700000000000002},"min":{"1min":0.628,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":0.96999999999999997,"5min":1.087,"15min":1.087},"last":0.85499999999999998},{"interface":"front","average":{"1min":0.82899999999999996,"5min":0.81399999999999995,"15min":0.81699999999999995},"min":{"1min":0.67100000000000004,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.97899999999999998,"5min":1.5289999999999999,"15min":1.5289999999999999},"last":0.73999999999999999}]},{"osd":2,"last update":"Sat Sep 30 01:38:05 2023","interfaces":[{"interface":"back","average":{"1min":0.83799999999999997,"5min":0.878,"15min":0.86799999999999999},"min":{"1min":0.67200000000000004,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":1.0820000000000001,"5min":4.0759999999999996,"15min":4.0759999999999996},"last":0.81100000000000005},{"interface":"front","average":{"1min":0.872,"5min":0.91500000000000004,"15min":0.86799999999999999},"min":{"1min":0.65400000000000003,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":1.0629999999999999,"5min":8.3109999999999999,"15min":8.3109999999999999},"last":0.59599999999999997}]},{"osd":3,"last update":"Sat Sep 30 01:38:05 2023","interfaces":[{"interface":"back","average":{"1min":0.751,"5min":0.751,"15min":0.78700000000000003},"min":{"1min":0.51600000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.95699999999999996,"5min":1.446,"15min":1.446},"last":0.67200000000000004},{"interface":"front","average":{"1min":0.98899999999999999,"5min":0.94299999999999995,"15min":0.95899999999999996},"min":{"1min":0.84299999999999997,"5min":0.53900000000000003,"15min":0.53900000000000003},"max":{"1min":1.1299999999999999,"5min":1.4790000000000001,"15min":1.4790000000000001},"last":0.94199999999999995}]},{"osd":4,"last update":"Sat Sep 30 01:38:05 2023","interfaces":[{"interface":"back","average":{"1min":0.85299999999999998,"5min":0.88400000000000001,"15min":0.84199999999999997},"min":{"1min":0.626,"5min":0.441,"15min":0.441},"max":{"1min":1.3109999999999999,"5min":8.2289999999999992,"15min":8.2289999999999992},"last":0.70499999999999996},{"interface":"front","average":{"1min":0.85199999999999998,"5min":0.82199999999999995,"15min":0.83399999999999996},"min":{"1min":0.58999999999999997,"5min":0.52100000000000002,"15min":0.52100000000000002},"max":{"1min":1,"5min":1.2310000000000001,"15min":1.2310000000000001},"last":0.76800000000000002}]}]},{"osd":4,"up_from":11,"seq":47244640319,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5128,"kb_used_data":128,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":94366712,"statfs":{"total":96636764160,"available":96631513088,"internally_reserved":0,"allocated":131072,"data_stored":37781,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 01:37:10 2023","interfaces":[{"interface":"back","average":{"1min":0.82399999999999995,"5min":0.77300000000000002,"15min":0.76900000000000002},"min":{"1min":0.59299999999999997,"5min":0.52500000000000002,"15min":0.52500000000000002},"max":{"1min":1.0169999999999999,"5min":1.0169999999999999,"15min":1.0169999999999999},"last":0.68500000000000005},{"interface":"front","average":{"1min":0.76500000000000001,"5min":0.80000000000000004,"15min":0.80000000000000004},"min":{"1min":0.58899999999999997,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":1.022,"5min":1.6439999999999999,"15min":1.6439999999999999},"last":0.878}]},{"osd":1,"last update":"Sat Sep 30 01:37:10 2023","interfaces":[{"interface":"back","average":{"1min":0.77100000000000002,"5min":0.78700000000000003,"15min":0.78100000000000003},"min":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"max":{"1min":1.028,"5min":1.028,"15min":1.028},"last":0.58699999999999997},{"interface":"front","average":{"1min":0.81200000000000006,"5min":0.79400000000000004,"15min":0.78500000000000003},"min":{"1min":0.59799999999999998,"5min":0.55800000000000005,"15min":0.55800000000000005},"max":{"1min":0.95999999999999996,"5min":1.548,"15min":1.548},"last":0.76300000000000001}]},{"osd":2,"last update":"Sat Sep 30 01:37:10 2023","interfaces":[{"interface":"back","average":{"1min":0.84499999999999997,"5min":0.84899999999999998,"15min":0.86499999999999999},"min":{"1min":0.71299999999999997,"5min":0.625,"15min":0.625},"max":{"1min":1.0449999999999999,"5min":1.1950000000000001,"15min":1.1950000000000001},"last":0.85099999999999998},{"interface":"front","average":{"1min":0.90200000000000002,"5min":0.92000000000000004,"15min":0.93400000000000005},"min":{"1min":0.67500000000000004,"5min":0.64800000000000002,"15min":0.64800000000000002},"max":{"1min":1.0469999999999999,"5min":1.165,"15min":1.165},"last":0.92100000000000004}]},{"osd":3,"last update":"Sat Sep 30 01:37:10 2023","interfaces":[{"interface":"back","average":{"1min":0.72799999999999998,"5min":0.73199999999999998,"15min":0.73799999999999999},"min":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"max":{"1min":0.996,"5min":1.5,"15min":1.5},"last":0.55300000000000005},{"interface":"front","average":{"1min":0.80800000000000005,"5min":0.80100000000000005,"15min":0.81699999999999995},"min":{"1min":0.61099999999999999,"5min":0.54700000000000004,"15min":0.54700000000000004},"max":{"1min":1.1020000000000001,"5min":1.405,"15min":1.405},"last":0.80800000000000005}]},{"osd":5,"last update":"Sat Sep 30 01:37:10 2023","interfaces":[{"interface":"back","average":{"1min":0.80900000000000005,"5min":0.79000000000000004,"15min":0.77900000000000003},"min":{"1min":0.56000000000000005,"5min":0.54800000000000004,"15min":0.54800000000000004},"max":{"1min":1.0860000000000001,"5min":1.7270000000000001,"15min":1.7270000000000001},"last":0.626},{"interface":"front","average":{"1min":0.90600000000000003,"5min":0.90300000000000002,"15min":0.90500000000000003},"min":{"1min":0.80900000000000005,"5min":0.66800000000000004,"15min":0.66800000000000004},"max":{"1min":1.089,"5min":1.0960000000000001,"15min":1.0960000000000001},"last":0.72699999999999998}]}]},{"osd":3,"up_from":11,"seq":47244640320,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6096,"kb_used_data":648,"kb_used_omap":0,"kb_used_meta":5440,"kb_avail":94365744,"statfs":{"total":96636764160,"available":96630521856,"internally_reserved":0,"allocated":663552,"data_stored":566709,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5570560},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 01:37:03 2023","interfaces":[{"interface":"back","average":{"1min":0.83499999999999996,"5min":0.78300000000000003,"15min":0.77300000000000002},"min":{"1min":0.48299999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":1.052,"5min":1.052,"15min":1.052},"last":0.92500000000000004},{"interface":"front","average":{"1min":0.751,"5min":0.75,"15min":0.73499999999999999},"min":{"1min":0.61199999999999999,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":0.96799999999999997,"5min":1.206,"15min":1.206},"last":0.627}]},{"osd":1,"last update":"Sat Sep 30 01:37:03 2023","interfaces":[{"interface":"back","average":{"1min":0.84099999999999997,"5min":0.83099999999999996,"15min":0.82899999999999996},"min":{"1min":0.59799999999999998,"5min":0.56599999999999995,"15min":0.56599999999999995},"max":{"1min":0.96899999999999997,"5min":1.159,"15min":1.159},"last":0.876},{"interface":"front","average":{"1min":0.80000000000000004,"5min":0.77700000000000002,"15min":0.76100000000000001},"min":{"1min":0.56799999999999995,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":1.016,"5min":1.137,"15min":1.137},"last":0.84399999999999997}]},{"osd":2,"last update":"Sat Sep 30 01:37:03 2023","interfaces":[{"interface":"back","average":{"1min":0.86099999999999999,"5min":0.90700000000000003,"15min":0.93999999999999995},"min":{"1min":0.67700000000000005,"5min":0.60599999999999998,"15min":0.60599999999999998},"max":{"1min":1.0269999999999999,"5min":1.296,"15min":1.296},"last":0.98199999999999998},{"interface":"front","average":{"1min":0.88300000000000001,"5min":0.90700000000000003,"15min":0.92500000000000004},"min":{"1min":0.76300000000000001,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":1.0009999999999999,"5min":1.46,"15min":1.46},"last":0.69399999999999995}]},{"osd":4,"last update":"Sat Sep 30 01:37:03 2023","interfaces":[{"interface":"back","average":{"1min":0.749,"5min":0.73699999999999999,"15min":0.73499999999999999},"min":{"1min":0.57399999999999995,"5min":0.51300000000000001,"15min":0.51300000000000001},"max":{"1min":1.0389999999999999,"5min":1.3220000000000001,"15min":1.3220000000000001},"last":0.56399999999999995},{"interface":"front","average":{"1min":0.79200000000000004,"5min":0.84199999999999997,"15min":0.876},"min":{"1min":0.61799999999999999,"5min":0.56299999999999994,"15min":0.56299999999999994},"max":{"1min":1.046,"5min":1.0800000000000001,"15min":1.0800000000000001},"last":0.71599999999999997}]},{"osd":5,"last update":"Sat Sep 30 01:37:03 2023","interfaces":[{"interface":"back","average":{"1min":0.81899999999999995,"5min":0.81499999999999995,"15min":0.82799999999999996},"min":{"1min":0.64400000000000002,"5min":0.61099999999999999,"15min":0.61099999999999999},"max":{"1min":0.95499999999999996,"5min":1.3320000000000001,"15min":1.3320000000000001},"last":0.76500000000000001},{"interface":"front","average":{"1min":0.79700000000000004,"5min":0.84699999999999998,"15min":0.871},"min":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"max":{"1min":1.0740000000000001,"5min":1.387,"15min":1.387},"last":0.80500000000000005}]}]},{"osd":2,"up_from":11,"seq":47244640321,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5128,"kb_used_data":128,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":94366712,"statfs":{"total":96636764160,"available":96631513088,"internally_reserved":0,"allocated":131072,"data_stored":37781,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 01:37:06 2023","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.57599999999999996,"15min":0.56399999999999995},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.95099999999999996,"5min":0.95099999999999996,"15min":0.95099999999999996},"last":0.49299999999999999},{"interface":"front","average":{"1min":0.70899999999999996,"5min":0.67600000000000005,"15min":0.65700000000000003},"min":{"1min":0.46500000000000002,"5min":0.439,"15min":0.439},"max":{"1min":2.2029999999999998,"5min":2.2029999999999998,"15min":2.2029999999999998},"last":0.57599999999999996}]},{"osd":1,"last update":"Sat Sep 30 01:37:06 2023","interfaces":[{"interface":"back","average":{"1min":0.71799999999999997,"5min":0.70399999999999996,"15min":0.70799999999999996},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":2.2490000000000001,"5min":2.2490000000000001,"15min":2.2490000000000001},"last":0.69899999999999995},{"interface":"front","average":{"1min":0.64300000000000002,"5min":0.64600000000000002,"15min":0.65500000000000003},"min":{"1min":0.46899999999999997,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.85799999999999998,"5min":1.0409999999999999,"15min":1.0409999999999999},"last":0.73099999999999998}]},{"osd":3,"last update":"Sat Sep 30 01:37:06 2023","interfaces":[{"interface":"back","average":{"1min":0.77100000000000002,"5min":0.82899999999999996,"15min":0.83599999999999997},"min":{"1min":0.61199999999999999,"5min":0.61199999999999999,"15min":0.61199999999999999},"max":{"1min":1.0660000000000001,"5min":1.343,"15min":1.343},"last":0.66700000000000004},{"interface":"front","average":{"1min":0.96299999999999997,"5min":0.92200000000000004,"15min":0.92500000000000004},"min":{"1min":0.76700000000000002,"5min":0.60499999999999998,"15min":0.60499999999999998},"max":{"1min":2.4390000000000001,"5min":2.4390000000000001,"15min":2.4390000000000001},"last":0.93999999999999995}]},{"osd":4,"last update":"Sat Sep 30 01:37:06 2023","interfaces":[{"interface":"back","average":{"1min":0.94699999999999995,"5min":0.90000000000000002,"15min":0.90800000000000003},"min":{"1min":0.72699999999999998,"5min":0.64300000000000002,"15min":0.64300000000000002},"max":{"1min":2.3679999999999999,"5min":2.3679999999999999,"15min":2.3679999999999999},"last":0.83099999999999996},{"interface":"front","average":{"1min":0.85699999999999998,"5min":0.84799999999999998,"15min":0.84799999999999998},"min":{"1min":0.68500000000000005,"5min":0.505,"15min":0.505},"max":{"1min":1.401,"5min":1.401,"15min":1.401},"last":0.85299999999999998}]},{"osd":5,"last update":"Sat Sep 30 01:37:06 2023","interfaces":[{"interface":"back","average":{"1min":0.85199999999999998,"5min":0.86199999999999999,"15min":0.86599999999999999},"min":{"1min":0.628,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":1.0629999999999999,"5min":1.3959999999999999,"15min":1.3959999999999999},"last":0.77600000000000002},{"interface":"front","average":{"1min":0.88900000000000001,"5min":0.91500000000000004,"15min":0.92700000000000005},"min":{"1min":0.71099999999999997,"5min":0.61799999999999999,"15min":0.61799999999999999},"max":{"1min":1.234,"5min":1.625,"15min":1.625},"last":0.90700000000000003}]}]},{"osd":1,"up_from":11,"seq":47244640320,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6096,"kb_used_data":648,"kb_used_omap":0,"kb_used_meta":5440,"kb_avail":94365744,"statfs":{"total":96636764160,"available":96630521856,"internally_reserved":0,"allocated":663552,"data_stored":566709,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5570560},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 01:37:07 2023","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.57299999999999995,"15min":0.57199999999999995},"min":{"1min":0.48999999999999999,"5min":0.317,"15min":0.317},"max":{"1min":0.86299999999999999,"5min":0.92400000000000004,"15min":0.92400000000000004},"last":0.54300000000000004},{"interface":"front","average":{"1min":0.76800000000000002,"5min":0.65300000000000002,"15min":0.65200000000000002},"min":{"1min":0.47499999999999998,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":2.085,"5min":2.085,"15min":2.085},"last":0.71599999999999997}]},{"osd":2,"last update":"Sat Sep 30 01:37:07 2023","interfaces":[{"interface":"back","average":{"1min":0.64800000000000002,"5min":0.64400000000000002,"15min":0.66900000000000004},"min":{"1min":0.442,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.80900000000000005,"5min":1.159,"15min":1.159},"last":0.58199999999999996},{"interface":"front","average":{"1min":0.749,"5min":0.69899999999999995,"15min":0.70099999999999996},"min":{"1min":0.58799999999999997,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":1.9930000000000001,"5min":1.9930000000000001,"15min":1.9930000000000001},"last":0.66800000000000004}]},{"osd":3,"last update":"Sat Sep 30 01:37:07 2023","interfaces":[{"interface":"back","average":{"1min":0.81599999999999995,"5min":0.82199999999999995,"15min":0.82499999999999996},"min":{"1min":0.54500000000000004,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":0.98899999999999999,"5min":1.1499999999999999,"15min":1.1499999999999999},"last":0.80200000000000005},{"interface":"front","average":{"1min":0.83599999999999997,"5min":0.83299999999999996,"15min":0.84099999999999997},"min":{"1min":0.69999999999999996,"5min":0.61699999999999999,"15min":0.61699999999999999},"max":{"1min":1.024,"5min":1.2929999999999999,"15min":1.2929999999999999},"last":0.88800000000000001}]},{"osd":4,"last update":"Sat Sep 30 01:37:07 2023","interfaces":[{"interface":"back","average":{"1min":0.84899999999999998,"5min":0.86399999999999999,"15min":0.88100000000000001},"min":{"1min":0.60399999999999998,"5min":0.53600000000000003,"15min":0.53600000000000003},"max":{"1min":0.97499999999999998,"5min":1.415,"15min":1.415},"last":0.83199999999999996},{"interface":"front","average":{"1min":0.878,"5min":0.876,"15min":0.88700000000000001},"min":{"1min":0.59299999999999997,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":1.004,"5min":1.381,"15min":1.381},"last":0.94499999999999995}]},{"osd":5,"last update":"Sat Sep 30 01:37:07 2023","interfaces":[{"interface":"back","average":{"1min":0.93300000000000005,"5min":0.90600000000000003,"15min":0.90500000000000003},"min":{"1min":0.77400000000000002,"5min":0.66200000000000003,"15min":0.66200000000000003},"max":{"1min":1.294,"5min":1.395,"15min":1.395},"last":1.0109999999999999},{"interface":"front","average":{"1min":0.98199999999999998,"5min":0.95799999999999996,"15min":0.96299999999999997},"min":{"1min":0.753,"5min":0.56299999999999994,"15min":0.56299999999999994},"max":{"1min":1.349,"5min":1.4179999999999999,"15min":1.4179999999999999},"last":1.077}]}]},{"osd":0,"up_from":11,"seq":47244640321,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5128,"kb_used_data":128,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":94366712,"statfs":{"total":96636764160,"available":96631513088,"internally_reserved":0,"allocated":131072,"data_stored":37781,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Sep 30 01:37:05 2023","interfaces":[{"interface":"back","average":{"1min":0.57399999999999995,"5min":0.58299999999999996,"15min":0.59299999999999997},"min":{"1min":0.48899999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.69099999999999995,"5min":0.92900000000000005,"15min":0.92900000000000005},"last":0.66700000000000004},{"interface":"front","average":{"1min":0.67400000000000004,"5min":0.64300000000000002,"15min":0.64100000000000001},"min":{"1min":0.42599999999999999,"5min":0.314,"15min":0.314},"max":{"1min":0.98599999999999999,"5min":0.98599999999999999,"15min":0.98599999999999999},"last":0.54000000000000004}]},{"osd":2,"last update":"Sat Sep 30 01:37:05 2023","interfaces":[{"interface":"back","average":{"1min":0.67700000000000005,"5min":0.628,"15min":0.61899999999999999},"min":{"1min":0.53000000000000003,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.93799999999999994,"5min":0.93799999999999994,"15min":0.93799999999999994},"last":0.86299999999999999},{"interface":"front","average":{"1min":0.754,"5min":0.72199999999999998,"15min":0.70799999999999996},"min":{"1min":0.56499999999999995,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":1.2669999999999999,"5min":2.069,"15min":2.069},"last":0.63700000000000001}]},{"osd":3,"last update":"Sat Sep 30 01:37:05 2023","interfaces":[{"interface":"back","average":{"1min":0.80200000000000005,"5min":0.84299999999999997,"15min":0.85599999999999998},"min":{"1min":0.67600000000000005,"5min":0.60299999999999998,"15min":0.60299999999999998},"max":{"1min":0.89800000000000002,"5min":1.1319999999999999,"15min":1.1319999999999999},"last":0.84099999999999997},{"interface":"front","average":{"1min":0.88600000000000001,"5min":0.88300000000000001,"15min":0.88},"min":{"1min":0.68400000000000005,"5min":0.62,"15min":0.62},"max":{"1min":1.0649999999999999,"5min":2.4870000000000001,"15min":2.4870000000000001},"last":1.024}]},{"osd":4,"last update":"Sat Sep 30 01:37:05 2023","interfaces":[{"interface":"back","average":{"1min":0.92600000000000005,"5min":0.92400000000000004,"15min":0.92600000000000005},"min":{"1min":0.74299999999999999,"5min":0.66300000000000003,"15min":0.66300000000000003},"max":{"1min":1.3,"5min":2.589,"15min":2.589},"last":0.78700000000000003},{"interface":"front","average":{"1min":0.86799999999999999,"5min":0.878,"15min":0.90800000000000003},"min":{"1min":0.70099999999999996,"5min":0.54900000000000004,"15min":0.54900000000000004},"max":{"1min":1.2170000000000001,"5min":1.413,"15min":1.413},"last":0.93999999999999995}]},{"osd":5,"last update":"Sat Sep 30 01:37:05 2023","interfaces":[{"interface":"back","average":{"1min":0.92800000000000005,"5min":0.90900000000000003,"15min":0.92900000000000005},"min":{"1min":0.68000000000000005,"5min":0.58799999999999997,"15min":0.58799999999999997},"max":{"1min":1.1599999999999999,"5min":1.3200000000000001,"15min":1.3200000000000001},"last":0.81000000000000005},{"interface":"front","average":{"1min":0.97299999999999998,"5min":0.96399999999999997,"15min":0.96899999999999997},"min":{"1min":0.73999999999999999,"5min":0.73999999999999999,"15min":0.73999999999999999},"max":{"1min":1.3879999999999999,"5min":2.4279999999999999,"15min":2.4279999999999999},"last":0.99199999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-09-30T01:38:10.162 INFO:tasks.ceph.ceph_manager.ceph:clean! 2023-09-30T01:38:10.162 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-09-30T01:38:10.263 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.259+0000 7f2670e74700 10 monclient: get_monmap_and_config 2023-09-30T01:38:10.264 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.259+0000 7f2670e74700 10 monclient: build_initial_monmap 2023-09-30T01:38:10.264 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.259+0000 7f2670e74700 10 monclient: monmap: 2023-09-30T01:38:10.264 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2023-09-30T01:38:10.264 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:38:10.264 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:38:10.263704+0000 2023-09-30T01:38:10.264 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:38:10.263704+0000 2023-09-30T01:38:10.264 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2023-09-30T01:38:10.264 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:38:10.264 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a-legacy 2023-09-30T01:38:10.265 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.107:6789/0 mon.noname-f-legacy 2023-09-30T01:38:10.265 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.noname-b 2023-09-30T01:38:10.265 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.noname-g 2023-09-30T01:38:10.265 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.noname-c 2023-09-30T01:38:10.265 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.noname-h 2023-09-30T01:38:10.265 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.noname-d 2023-09-30T01:38:10.265 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.noname-i 2023-09-30T01:38:10.265 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.noname-e 2023-09-30T01:38:10.265 INFO:teuthology.orchestra.run.smithi088.stderr:9: v2:172.21.15.88:3300/0 mon.noname-a 2023-09-30T01:38:10.265 INFO:teuthology.orchestra.run.smithi088.stderr:10: v2:172.21.15.107:3300/0 mon.noname-f 2023-09-30T01:38:10.265 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:38:10.270 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.263+0000 7f2670e74700 1 Processor -- start 2023-09-30T01:38:10.270 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.263+0000 7f2670e74700 1 -- start start 2023-09-30T01:38:10.270 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.263+0000 7f2670e74700 10 monclient: init 2023-09-30T01:38:10.270 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.263+0000 7f2670e74700 10 monclient: _reopen_session rank -1 2023-09-30T01:38:10.270 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.263+0000 7f2670e74700 10 monclient: _add_conns ranks=[1,2,3,9,7,4,8,0,5,10,6] 2023-09-30T01:38:10.270 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.263+0000 7f2670e74700 10 monclient(hunting): picked mon.noname-f-legacy con 0x7f266c0a35f0 addr v1:172.21.15.107:6789/0 2023-09-30T01:38:10.270 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.263+0000 7f2670e74700 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f266c0a5700 0x7f266c133a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:38:10.270 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.263+0000 7f2670e74700 10 monclient(hunting): picked mon.noname-b con 0x7f266c0a5700 addr [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] 2023-09-30T01:38:10.270 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.263+0000 7f2670e74700 1 --2- >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7f266c1386a0 0x7f266c13aa80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:38:10.271 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.263+0000 7f2670e74700 10 monclient(hunting): picked mon.noname-g con 0x7f266c1386a0 addr [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] 2023-09-30T01:38:10.271 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.263+0000 7f2670e74700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f266c05ecb0 con 0x7f266c0a35f0 2023-09-30T01:38:10.271 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.263+0000 7f2670e74700 10 monclient(hunting): start opening mon connection 2023-09-30T01:38:10.271 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.263+0000 7f2670e74700 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f266c097ad0 con 0x7f266c0a5700 2023-09-30T01:38:10.271 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.263+0000 7f2670e74700 10 monclient(hunting): start opening mon connection 2023-09-30T01:38:10.271 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.263+0000 7f2670e74700 1 -- --> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f266c0a0940 con 0x7f266c1386a0 2023-09-30T01:38:10.271 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.263+0000 7f2670e74700 10 monclient(hunting): _renew_subs 2023-09-30T01:38:10.271 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.263+0000 7f2670e74700 10 monclient(hunting): authenticate will time out at 1162.301198s 2023-09-30T01:38:10.271 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.263+0000 7f2661d9b700 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f266c0a5700 0x7f266c133a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:38:10.271 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.263+0000 7f2661d9b700 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f266c0a5700 0x7f266c133a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3301/0 says I am v2:172.21.15.88:53460/0 (socket says 172.21.15.88:53460) 2023-09-30T01:38:10.271 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.263+0000 7f2661d9b700 1 -- 172.21.15.88:0/3289782612 learned_addr learned my addr 172.21.15.88:0/3289782612 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-09-30T01:38:10.271 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.263+0000 7f2661d9b700 10 monclient(hunting): get_auth_request con 0x7f266c0a5700 auth_method 0 2023-09-30T01:38:10.272 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.263+0000 7f2661d9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:38:10.272 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.263+0000 7f2661d9b700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:38:10.272 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.263+0000 7f2661d9b700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:38:10.272 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.263+0000 7f266ad9d700 1 --2- 172.21.15.88:0/3289782612 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7f266c1386a0 0x7f266c13aa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:38:10.272 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.263+0000 7f266ad9d700 10 monclient(hunting): get_auth_request con 0x7f266c1386a0 auth_method 0 2023-09-30T01:38:10.272 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.263+0000 7f266ad9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:38:10.272 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.263+0000 7f266ad9d700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:38:10.272 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.263+0000 7f266ad9d700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:38:10.272 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.263+0000 7f2661d9b700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:38:10.272 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.263+0000 7f2661d9b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:38:10.272 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.263+0000 7f2661d9b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:38:10.272 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.263+0000 7f266ad9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:38:10.273 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.263+0000 7f266ad9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:38:10.273 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.263+0000 7f266ad9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:38:10.273 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.263+0000 7f2669d9b700 1 -- 172.21.15.88:0/3289782612 <== mon.1 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2308809475 0 0) 0x7f266c05ecb0 con 0x7f266c0a35f0 2023-09-30T01:38:10.273 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.263+0000 7f2669d9b700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:38:10.273 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.263+0000 7f2669d9b700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:38:10.273 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.263+0000 7f2669d9b700 10 monclient(hunting): my global_id is 5221 2023-09-30T01:38:10.273 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.263+0000 7f2669d9b700 1 -- 172.21.15.88:0/3289782612 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2658003570 con 0x7f266c0a35f0 2023-09-30T01:38:10.273 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.263+0000 7f2661d9b700 10 monclient(hunting): handle_auth_done global_id 4808 payload 274 2023-09-30T01:38:10.273 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.263+0000 7f2661d9b700 1 -- 172.21.15.88:0/3289782612 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7f266c1386a0 msgr2=0x7f266c13aa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:38:10.273 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.263+0000 7f2661d9b700 1 --2- 172.21.15.88:0/3289782612 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7f266c1386a0 0x7f266c13aa80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:38:10.273 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.263+0000 7f2661d9b700 1 -- 172.21.15.88:0/3289782612 >> v1:172.21.15.107:6789/0 conn(0x7f266c0a35f0 legacy=0x7f266c133330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:38:10.273 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.263+0000 7f2661d9b700 10 monclient: _finish_hunting 0 2023-09-30T01:38:10.274 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.263+0000 7f2661d9b700 1 monclient: found mon.noname-b 2023-09-30T01:38:10.274 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.263+0000 7f2661d9b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:38:10.274 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.263+0000 7f2661d9b700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.88:3301/0 2023-09-30T01:38:10.274 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.263+0000 7f2661d9b700 1 -- 172.21.15.88:0/3289782612 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f266c0a39c0 con 0x7f266c0a5700 2023-09-30T01:38:10.274 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.263+0000 7f266ad9d700 1 --2- 172.21.15.88:0/3289782612 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7f266c1386a0 0x7f266c13aa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:38:10.274 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.263+0000 7f2661d9b700 1 --2- 172.21.15.88:0/3289782612 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f266c0a5700 0x7f266c133a40 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f26540099e0 tx=0x7f2654009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=27802a1d0099af33 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:38:10.274 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.263+0000 7f2669d9b700 1 -- 172.21.15.88:0/3289782612 <== mon.2 v2:172.21.15.88:3301/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f265400bb00 con 0x7f266c0a5700 2023-09-30T01:38:10.274 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.263+0000 7f2669d9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:38:10.274 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.263+0000 7f2669d9b700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T01:38:10.274 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.263+0000 7f2669d9b700 10 monclient: dump: 2023-09-30T01:38:10.274 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:38:10.274 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:38:10.275 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:38:10.275 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:38:10.275 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:38:10.275 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:38:10.275 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:38:10.275 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:38:10.275 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:38:10.275 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:38:10.275 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:38:10.275 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:38:10.275 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:38:10.275 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:38:10.275 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:38:10.276 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:38:10.276 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.263+0000 7f2669d9b700 10 monclient: _finish_auth 0 2023-09-30T01:38:10.276 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.263+0000 7f2669d9b700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:38:10.276 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.263+0000 7f2669d9b700 1 -- 172.21.15.88:0/3289782612 <== mon.2 v2:172.21.15.88:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f265400bc80 con 0x7f266c0a5700 2023-09-30T01:38:10.276 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.263+0000 7f2669d9b700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:38:10.276 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.263+0000 7f2669d9b700 1 -- 172.21.15.88:0/3289782612 <== mon.2 v2:172.21.15.88:3301/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f265400bdf0 con 0x7f266c0a5700 2023-09-30T01:38:10.276 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.263+0000 7f2670e74700 5 monclient: authenticate success, global_id 4808 2023-09-30T01:38:10.276 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.263+0000 7f2670e74700 10 monclient: get_monmap_and_config success 2023-09-30T01:38:10.276 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.263+0000 7f2670e74700 4 set_mon_vals no callback set 2023-09-30T01:38:10.276 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.263+0000 7f2669d9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:38:10.276 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.263+0000 7f2669d9b700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T01:38:10.276 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.263+0000 7f2669d9b700 10 monclient: dump: 2023-09-30T01:38:10.277 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:38:10.277 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:38:10.277 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:38:10.277 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:38:10.277 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:38:10.277 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:38:10.277 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:38:10.277 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:38:10.277 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:38:10.277 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:38:10.277 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:38:10.277 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:38:10.277 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:38:10.278 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:38:10.278 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:38:10.278 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:38:10.278 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.267+0000 7f2670e74700 10 monclient: shutdown 2023-09-30T01:38:10.278 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.267+0000 7f2670e74700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:38:10.278 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.267+0000 7f2670e74700 1 -- 172.21.15.88:0/3289782612 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f266c0a5700 msgr2=0x7f266c133a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:38:10.278 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.267+0000 7f2670e74700 1 --2- 172.21.15.88:0/3289782612 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f266c0a5700 0x7f266c133a40 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f26540099e0 tx=0x7f2654009250 comp rx=0 tx=0).stop 2023-09-30T01:38:10.278 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.267+0000 7f2670e74700 1 -- 172.21.15.88:0/3289782612 shutdown_connections 2023-09-30T01:38:10.278 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.267+0000 7f2670e74700 1 --2- 172.21.15.88:0/3289782612 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7f266c1386a0 0x7f266c13aa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:38:10.278 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.267+0000 7f2670e74700 1 --2- 172.21.15.88:0/3289782612 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f266c0a5700 0x7f266c133a40 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:38:10.278 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.267+0000 7f2670e74700 1 -- 172.21.15.88:0/3289782612 >> 172.21.15.88:0/3289782612 conn(0x7f266c06e5e0 msgr2=0x7f266c06e9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:38:10.278 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.267+0000 7f2670e74700 1 -- 172.21.15.88:0/3289782612 shutdown_connections 2023-09-30T01:38:10.279 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.267+0000 7f2670e74700 1 -- 172.21.15.88:0/3289782612 wait complete. 2023-09-30T01:38:10.279 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.267+0000 7f2670e74700 10 monclient: build_initial_monmap 2023-09-30T01:38:10.279 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.267+0000 7f2670e74700 10 monclient: monmap: 2023-09-30T01:38:10.279 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2023-09-30T01:38:10.279 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:38:10.279 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 0.000000 2023-09-30T01:38:10.279 INFO:teuthology.orchestra.run.smithi088.stderr:created 0.000000 2023-09-30T01:38:10.279 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2023-09-30T01:38:10.279 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 1 2023-09-30T01:38:10.279 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.noname-a 2023-09-30T01:38:10.280 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.noname-f 2023-09-30T01:38:10.280 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.noname-b 2023-09-30T01:38:10.280 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.noname-g 2023-09-30T01:38:10.280 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.noname-c 2023-09-30T01:38:10.280 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.noname-h 2023-09-30T01:38:10.280 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.noname-d 2023-09-30T01:38:10.280 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.noname-i 2023-09-30T01:38:10.280 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.noname-e 2023-09-30T01:38:10.280 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:38:10.280 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.267+0000 7f2670e74700 1 Processor -- start 2023-09-30T01:38:10.285 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.267+0000 7f2670e74700 1 -- start start 2023-09-30T01:38:10.285 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.267+0000 7f2670e74700 10 monclient: init 2023-09-30T01:38:10.285 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.267+0000 7f2670e74700 10 monclient: _reopen_session rank -1 2023-09-30T01:38:10.285 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.267+0000 7f2670e74700 10 monclient: _add_conns ranks=[2,4,6,7,8,3,1,0,5] 2023-09-30T01:38:10.285 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.267+0000 7f2670e74700 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f266c0a35f0 0x7f266c130bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:38:10.285 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.271+0000 7f2670e74700 10 monclient(hunting): picked mon.noname-b con 0x7f266c0a35f0 addr [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] 2023-09-30T01:38:10.285 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.271+0000 7f2670e74700 1 --2- >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f266c0a5700 0x7f266c09b710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:38:10.285 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.271+0000 7f2670e74700 10 monclient(hunting): picked mon.noname-c con 0x7f266c0a5700 addr [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] 2023-09-30T01:38:10.285 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.271+0000 7f2670e74700 1 --2- >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7f266c1386a0 0x7f266c09bc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:38:10.285 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.271+0000 7f2670e74700 10 monclient(hunting): picked mon.noname-d con 0x7f266c1386a0 addr [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] 2023-09-30T01:38:10.286 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.271+0000 7f2670e74700 10 monclient(hunting): start opening mon connection 2023-09-30T01:38:10.286 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.271+0000 7f2670e74700 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f266c09cbb0 con 0x7f266c0a35f0 2023-09-30T01:38:10.286 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.271+0000 7f2670e74700 10 monclient(hunting): start opening mon connection 2023-09-30T01:38:10.286 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.271+0000 7f2670e74700 1 -- --> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] -- mon_getmap magic: 0 v1 -- 0x7f266c0a0b70 con 0x7f266c0a5700 2023-09-30T01:38:10.286 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.271+0000 7f2670e74700 10 monclient(hunting): start opening mon connection 2023-09-30T01:38:10.286 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.271+0000 7f2670e74700 1 -- --> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] -- mon_getmap magic: 0 v1 -- 0x7f266c097d90 con 0x7f266c1386a0 2023-09-30T01:38:10.286 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.271+0000 7f2670e74700 10 monclient(hunting): _renew_subs 2023-09-30T01:38:10.286 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.271+0000 7f2670e74700 10 monclient(hunting): authenticate will time out at 1162.308185s 2023-09-30T01:38:10.286 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.271+0000 7f266a59c700 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f266c0a35f0 0x7f266c130bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:38:10.286 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.271+0000 7f2661d9b700 1 --2- >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f266c0a5700 0x7f266c09b710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:38:10.286 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.271+0000 7f266ad9d700 1 --2- >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7f266c1386a0 0x7f266c09bc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:38:10.286 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.271+0000 7f2661d9b700 1 --2- >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f266c0a5700 0x7f266c09b710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3302/0 says I am v2:172.21.15.88:33926/0 (socket says 172.21.15.88:33926) 2023-09-30T01:38:10.287 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.271+0000 7f2661d9b700 1 -- 172.21.15.88:0/2630800655 learned_addr learned my addr 172.21.15.88:0/2630800655 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-09-30T01:38:10.287 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.271+0000 7f2661d9b700 10 monclient(hunting): get_auth_request con 0x7f266c0a5700 auth_method 0 2023-09-30T01:38:10.287 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.271+0000 7f266a59c700 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f266c0a35f0 0x7f266c130bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3301/0 says I am v2:172.21.15.88:53464/0 (socket says 172.21.15.88:53464) 2023-09-30T01:38:10.287 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.271+0000 7f2661d9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:38:10.287 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.271+0000 7f2661d9b700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:38:10.287 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.271+0000 7f2661d9b700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:38:10.287 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.271+0000 7f266a59c700 10 monclient(hunting): get_auth_request con 0x7f266c0a35f0 auth_method 0 2023-09-30T01:38:10.287 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.271+0000 7f266a59c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:38:10.287 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.271+0000 7f266a59c700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:38:10.287 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.271+0000 7f266a59c700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:38:10.287 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.271+0000 7f266ad9d700 10 monclient(hunting): get_auth_request con 0x7f266c1386a0 auth_method 0 2023-09-30T01:38:10.287 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.271+0000 7f266ad9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:38:10.288 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.271+0000 7f266ad9d700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:38:10.288 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.271+0000 7f266ad9d700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:38:10.288 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.271+0000 7f266ad9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:38:10.288 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.271+0000 7f266ad9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:38:10.288 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.271+0000 7f266ad9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:38:10.288 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.271+0000 7f2661d9b700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:38:10.288 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.271+0000 7f2661d9b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:38:10.288 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.271+0000 7f2661d9b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:38:10.288 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.271+0000 7f266a59c700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:38:10.288 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.271+0000 7f266a59c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:38:10.288 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.271+0000 7f266a59c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:38:10.288 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.271+0000 7f266ad9d700 10 monclient(hunting): handle_auth_done global_id 4803 payload 824 2023-09-30T01:38:10.288 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.271+0000 7f266ad9d700 1 -- 172.21.15.88:0/2630800655 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f266c0a5700 msgr2=0x7f266c09b710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:38:10.289 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.271+0000 7f266ad9d700 1 --2- 172.21.15.88:0/2630800655 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f266c0a5700 0x7f266c09b710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:38:10.289 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.271+0000 7f266ad9d700 1 -- 172.21.15.88:0/2630800655 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f266c0a35f0 msgr2=0x7f266c130bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:38:10.289 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.271+0000 7f266ad9d700 1 --2- 172.21.15.88:0/2630800655 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f266c0a35f0 0x7f266c130bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:38:10.289 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.271+0000 7f266ad9d700 10 monclient: _finish_hunting 0 2023-09-30T01:38:10.289 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.271+0000 7f266ad9d700 1 monclient: found mon.noname-d 2023-09-30T01:38:10.289 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.271+0000 7f266ad9d700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:38:10.289 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.271+0000 7f266ad9d700 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.88:3303/0 2023-09-30T01:38:10.289 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.271+0000 7f266ad9d700 1 -- 172.21.15.88:0/2630800655 --> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2664009a20 con 0x7f266c1386a0 2023-09-30T01:38:10.289 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.271+0000 7f266ad9d700 1 --2- 172.21.15.88:0/2630800655 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7f266c1386a0 0x7f266c09bc50 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f2664009e70 tx=0x7f266400b040 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:38:10.289 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.271+0000 7f2662ffd700 1 -- 172.21.15.88:0/2630800655 <== mon.6 v2:172.21.15.88:3303/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f2664014480 con 0x7f266c1386a0 2023-09-30T01:38:10.289 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.271+0000 7f2662ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:38:10.289 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.271+0000 7f2662ffd700 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2023-09-30T01:38:10.290 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.271+0000 7f2662ffd700 10 monclient: dump: 2023-09-30T01:38:10.290 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:38:10.290 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:38:10.290 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:38:10.290 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:38:10.290 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:38:10.290 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:38:10.290 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:38:10.290 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:38:10.290 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:38:10.290 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:38:10.290 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:38:10.291 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:38:10.291 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:38:10.291 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:38:10.291 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:38:10.291 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:38:10.291 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.271+0000 7f2662ffd700 10 monclient: _finish_auth 0 2023-09-30T01:38:10.291 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.271+0000 7f2662ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:38:10.291 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.271+0000 7f2670e74700 5 monclient: authenticate success, global_id 4803 2023-09-30T01:38:10.291 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.271+0000 7f2662ffd700 1 -- 172.21.15.88:0/2630800655 <== mon.6 v2:172.21.15.88:3303/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f266400f040 con 0x7f266c1386a0 2023-09-30T01:38:10.291 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.271+0000 7f2670e74700 10 monclient: _renew_subs 2023-09-30T01:38:10.291 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.271+0000 7f2670e74700 10 monclient: _send_mon_message to mon.d at v2:172.21.15.88:3303/0 2023-09-30T01:38:10.291 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.271+0000 7f2670e74700 1 -- 172.21.15.88:0/2630800655 --> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f26540094f0 con 0x7f266c1386a0 2023-09-30T01:38:10.292 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.271+0000 7f2662ffd700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:38:10.292 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.271+0000 7f2662ffd700 1 -- 172.21.15.88:0/2630800655 <== mon.6 v2:172.21.15.88:3303/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f2664014aa0 con 0x7f266c1386a0 2023-09-30T01:38:10.292 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.271+0000 7f2662ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:38:10.292 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.271+0000 7f266959a700 4 set_mon_vals no callback set 2023-09-30T01:38:10.292 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.271+0000 7f2662ffd700 10 monclient: got monmap 1 from mon.d (according to old e1) 2023-09-30T01:38:10.292 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.271+0000 7f2662ffd700 10 monclient: dump: 2023-09-30T01:38:10.292 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:38:10.292 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:38:10.292 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:38:10.292 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:38:10.292 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:38:10.292 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:38:10.292 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:38:10.293 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:38:10.293 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:38:10.293 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:38:10.293 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:38:10.293 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:38:10.293 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:38:10.293 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:38:10.293 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:38:10.293 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:38:10.293 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.271+0000 7f2670e74700 10 monclient: _renew_subs 2023-09-30T01:38:10.293 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.271+0000 7f2670e74700 10 monclient: _send_mon_message to mon.d at v2:172.21.15.88:3303/0 2023-09-30T01:38:10.293 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.271+0000 7f2670e74700 1 -- 172.21.15.88:0/2630800655 --> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f266c09d2b0 con 0x7f266c1386a0 2023-09-30T01:38:10.294 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.271+0000 7f2662ffd700 1 -- 172.21.15.88:0/2630800655 <== mon.6 v2:172.21.15.88:3303/0 4 ==== mgrmap(e 5) v1 ==== 40144+0+0 (secure 0 0 0) 0x7f2664003020 con 0x7f266c1386a0 2023-09-30T01:38:10.294 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.271+0000 7f2662ffd700 1 --2- 172.21.15.88:0/2630800655 >> v2:172.21.15.107:6812/29926 conn(0x7f264c033360 0x7f264c035810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:38:10.294 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.271+0000 7f2662ffd700 1 -- 172.21.15.88:0/2630800655 <== mon.6 v2:172.21.15.88:3303/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 3502+0+0 (secure 0 0 0) 0x7f2664043ae0 con 0x7f266c1386a0 2023-09-30T01:38:10.294 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.271+0000 7f266a59c700 1 --2- 172.21.15.88:0/2630800655 >> v2:172.21.15.107:6812/29926 conn(0x7f264c033360 0x7f264c035810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:38:10.294 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.271+0000 7f266a59c700 10 monclient: get_auth_request con 0x7f264c033360 auth_method 0 2023-09-30T01:38:10.294 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.271+0000 7f2670e74700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:38:10.294 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.271+0000 7f2670e74700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:38:10.294 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.271+0000 7f2670e74700 10 monclient: _send_mon_message to mon.d at v2:172.21.15.88:3303/0 2023-09-30T01:38:10.294 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.271+0000 7f2670e74700 1 -- 172.21.15.88:0/2630800655 --> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f266c09ef80 con 0x7f266c1386a0 2023-09-30T01:38:10.294 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.275+0000 7f266a59c700 1 --2- 172.21.15.88:0/2630800655 >> v2:172.21.15.107:6812/29926 conn(0x7f264c033360 0x7f264c035810 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f265c007fd0 tx=0x7f265c007d30 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:38:10.294 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.279+0000 7f2662ffd700 1 -- 172.21.15.88:0/2630800655 <== mon.6 v2:172.21.15.88:3303/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f2664018020 con 0x7f266c1386a0 2023-09-30T01:38:10.294 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.279+0000 7f2662ffd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:38:10.295 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.279+0000 7f2662ffd700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:38:10.434 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.431+0000 7f2670e74700 1 -- 172.21.15.88:0/2630800655 --> v2:172.21.15.107:6812/29926 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f266c064c10 con 0x7f264c033360 2023-09-30T01:38:10.440 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.439+0000 7f2662ffd700 1 -- 172.21.15.88:0/2630800655 <== mgr.4108 v2:172.21.15.107:6812/29926 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+53337 (secure 0 0 0) 0x7f266c064c10 con 0x7f264c033360 2023-09-30T01:38:10.441 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-09-30T01:38:10.443 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.439+0000 7f2670e74700 1 -- 172.21.15.88:0/2630800655 >> v2:172.21.15.107:6812/29926 conn(0x7f264c033360 msgr2=0x7f264c035810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:38:10.443 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.439+0000 7f2670e74700 1 --2- 172.21.15.88:0/2630800655 >> v2:172.21.15.107:6812/29926 conn(0x7f264c033360 0x7f264c035810 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f265c007fd0 tx=0x7f265c007d30 comp rx=0 tx=0).stop 2023-09-30T01:38:10.443 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.439+0000 7f2670e74700 10 monclient: shutdown 2023-09-30T01:38:10.443 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.439+0000 7f2670e74700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:38:10.443 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.439+0000 7f2670e74700 1 -- 172.21.15.88:0/2630800655 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7f266c1386a0 msgr2=0x7f266c09bc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:38:10.443 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.439+0000 7f2670e74700 1 --2- 172.21.15.88:0/2630800655 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7f266c1386a0 0x7f266c09bc50 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f2664009e70 tx=0x7f266400b040 comp rx=0 tx=0).stop 2023-09-30T01:38:10.443 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.439+0000 7f2670e74700 1 -- 172.21.15.88:0/2630800655 shutdown_connections 2023-09-30T01:38:10.443 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.439+0000 7f2670e74700 1 --2- 172.21.15.88:0/2630800655 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7f266c1386a0 0x7f266c09bc50 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:38:10.443 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.439+0000 7f2670e74700 1 --2- 172.21.15.88:0/2630800655 >> v2:172.21.15.107:6812/29926 conn(0x7f264c033360 0x7f264c035810 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:38:10.443 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.439+0000 7f2670e74700 1 --2- 172.21.15.88:0/2630800655 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f266c0a5700 0x7f266c09b710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:38:10.443 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.439+0000 7f2670e74700 1 --2- 172.21.15.88:0/2630800655 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f266c0a35f0 0x7f266c130bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:38:10.444 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.439+0000 7f2670e74700 1 -- 172.21.15.88:0/2630800655 >> 172.21.15.88:0/2630800655 conn(0x7f266c06e5e0 msgr2=0x7f266c1325b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:38:10.444 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.439+0000 7f2670e74700 1 -- 172.21.15.88:0/2630800655 shutdown_connections 2023-09-30T01:38:10.444 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.439+0000 7f2670e74700 1 -- 172.21.15.88:0/2630800655 wait complete. 2023-09-30T01:38:10.444 INFO:teuthology.orchestra.run.smithi088.stderr:dumped all 2023-09-30T01:38:10.459 INFO:teuthology.orchestra.run.smithi088.stdout:{"pg_ready":true,"pg_map":{"version":165,"stamp":"2023-09-30T01:38:09.743539+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":32704,"kb_used_data":1808,"kb_used_omap":0,"kb_used_meta":30848,"kb_avail":566198336,"statfs":{"total":579820584960,"available":579787096064,"internally_reserved":0,"allocated":1851392,"data_stored":1284542,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":31588352},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003602"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-09-30T01:33:03.296691+0000","last_change":"2023-09-30T01:33:03.292315+0000","last_active":"2023-09-30T01:33:03.296691+0000","last_peered":"2023-09-30T01:33:03.296691+0000","last_clean":"2023-09-30T01:33:03.296691+0000","last_became_active":"2023-09-30T01:33:03.291889+0000","last_became_peered":"2023-09-30T01:33:03.291889+0000","last_unstale":"2023-09-30T01:33:03.296691+0000","last_undegraded":"2023-09-30T01:33:03.296691+0000","last_fullsized":"2023-09-30T01:33:03.296691+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T01:33:02.252599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T01:33:02.252599+0000","last_clean_scrub_stamp":"2023-09-30T01:33:02.252599+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T11:44:00.574379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-09-30T01:33:03.297822+0000","last_change":"2023-09-30T01:33:03.295081+0000","last_active":"2023-09-30T01:33:03.297822+0000","last_peered":"2023-09-30T01:33:03.297822+0000","last_clean":"2023-09-30T01:33:03.297822+0000","last_became_active":"2023-09-30T01:33:03.294535+0000","last_became_peered":"2023-09-30T01:33:03.294535+0000","last_unstale":"2023-09-30T01:33:03.297822+0000","last_undegraded":"2023-09-30T01:33:03.297822+0000","last_fullsized":"2023-09-30T01:33:03.297822+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T01:33:02.252599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T01:33:02.252599+0000","last_clean_scrub_stamp":"2023-09-30T01:33:02.252599+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T05:49:49.642723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-09-30T01:33:03.300117+0000","last_change":"2023-09-30T01:33:03.297392+0000","last_active":"2023-09-30T01:33:03.300117+0000","last_peered":"2023-09-30T01:33:03.300117+0000","last_clean":"2023-09-30T01:33:03.300117+0000","last_became_active":"2023-09-30T01:33:03.296959+0000","last_became_peered":"2023-09-30T01:33:03.296959+0000","last_unstale":"2023-09-30T01:33:03.300117+0000","last_undegraded":"2023-09-30T01:33:03.300117+0000","last_fullsized":"2023-09-30T01:33:03.300117+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T01:33:02.252599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T01:33:02.252599+0000","last_clean_scrub_stamp":"2023-09-30T01:33:02.252599+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T11:57:13.112607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-09-30T01:33:03.296932+0000","last_change":"2023-09-30T01:33:03.295478+0000","last_active":"2023-09-30T01:33:03.296932+0000","last_peered":"2023-09-30T01:33:03.296932+0000","last_clean":"2023-09-30T01:33:03.296932+0000","last_became_active":"2023-09-30T01:33:03.295106+0000","last_became_peered":"2023-09-30T01:33:03.295106+0000","last_unstale":"2023-09-30T01:33:03.296932+0000","last_undegraded":"2023-09-30T01:33:03.296932+0000","last_fullsized":"2023-09-30T01:33:03.296932+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T01:33:02.252599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T01:33:02.252599+0000","last_clean_scrub_stamp":"2023-09-30T01:33:02.252599+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T07:34:36.123333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"15'117","reported_seq":159,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-09-30T01:33:05.391246+0000","last_change":"2023-09-30T01:33:03.296872+0000","last_active":"2023-09-30T01:33:05.391246+0000","last_peered":"2023-09-30T01:33:05.391246+0000","last_clean":"2023-09-30T01:33:05.391246+0000","last_became_active":"2023-09-30T01:33:03.295200+0000","last_became_peered":"2023-09-30T01:33:03.295200+0000","last_unstale":"2023-09-30T01:33:05.391246+0000","last_undegraded":"2023-09-30T01:33:05.391246+0000","last_fullsized":"2023-09-30T01:33:05.391246+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T01:33:02.252599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T01:33:02.252599+0000","last_clean_scrub_stamp":"2023-09-30T01:33:02.252599+0000","objects_scrubbed":0,"log_size":117,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T01:51:03.655484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-09-30T01:33:03.296258+0000","last_change":"2023-09-30T01:33:03.293385+0000","last_active":"2023-09-30T01:33:03.296258+0000","last_peered":"2023-09-30T01:33:03.296258+0000","last_clean":"2023-09-30T01:33:03.296258+0000","last_became_active":"2023-09-30T01:33:03.293004+0000","last_became_peered":"2023-09-30T01:33:03.293004+0000","last_unstale":"2023-09-30T01:33:03.296258+0000","last_undegraded":"2023-09-30T01:33:03.296258+0000","last_fullsized":"2023-09-30T01:33:03.296258+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T01:33:02.252599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T01:33:02.252599+0000","last_clean_scrub_stamp":"2023-09-30T01:33:02.252599+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T03:32:32.404638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-09-30T01:33:03.299431+0000","last_change":"2023-09-30T01:33:03.296945+0000","last_active":"2023-09-30T01:33:03.299431+0000","last_peered":"2023-09-30T01:33:03.299431+0000","last_clean":"2023-09-30T01:33:03.299431+0000","last_became_active":"2023-09-30T01:33:03.296465+0000","last_became_peered":"2023-09-30T01:33:03.296465+0000","last_unstale":"2023-09-30T01:33:03.299431+0000","last_undegraded":"2023-09-30T01:33:03.299431+0000","last_fullsized":"2023-09-30T01:33:03.299431+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T01:33:02.252599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T01:33:02.252599+0000","last_clean_scrub_stamp":"2023-09-30T01:33:02.252599+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T05:32:13.855128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-09-30T01:33:03.296569+0000","last_change":"2023-09-30T01:33:03.291527+0000","last_active":"2023-09-30T01:33:03.296569+0000","last_peered":"2023-09-30T01:33:03.296569+0000","last_clean":"2023-09-30T01:33:03.296569+0000","last_became_active":"2023-09-30T01:33:03.291084+0000","last_became_peered":"2023-09-30T01:33:03.291084+0000","last_unstale":"2023-09-30T01:33:03.296569+0000","last_undegraded":"2023-09-30T01:33:03.296569+0000","last_fullsized":"2023-09-30T01:33:03.296569+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T01:33:02.252599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T01:33:02.252599+0000","last_clean_scrub_stamp":"2023-09-30T01:33:02.252599+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T12:10:57.124090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-09-30T01:33:03.299303+0000","last_change":"2023-09-30T01:33:03.296313+0000","last_active":"2023-09-30T01:33:03.299303+0000","last_peered":"2023-09-30T01:33:03.299303+0000","last_clean":"2023-09-30T01:33:03.299303+0000","last_became_active":"2023-09-30T01:33:03.294604+0000","last_became_peered":"2023-09-30T01:33:03.294604+0000","last_unstale":"2023-09-30T01:33:03.299303+0000","last_undegraded":"2023-09-30T01:33:03.299303+0000","last_fullsized":"2023-09-30T01:33:03.299303+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T01:33:02.252599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T01:33:02.252599+0000","last_clean_scrub_stamp":"2023-09-30T01:33:02.252599+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T06:00:29.060679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":1057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":6}],"osd_stats":[{"osd":5,"up_from":11,"seq":47244640319,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5128,"kb_used_data":128,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":94366712,"statfs":{"total":96636764160,"available":96631513088,"internally_reserved":0,"allocated":131072,"data_stored":37781,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 01:38:05 2023","interfaces":[{"interface":"back","average":{"1min":0.76700000000000002,"5min":0.85799999999999998,"15min":0.79400000000000004},"min":{"1min":0.46999999999999997,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":1.028,"5min":8.109,"15min":8.109},"last":0.88500000000000001},{"interface":"front","average":{"1min":0.82199999999999995,"5min":0.88600000000000001,"15min":0.85999999999999999},"min":{"1min":0.57699999999999996,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":1.0009999999999999,"5min":8.3780000000000001,"15min":8.3780000000000001},"last":0.91500000000000004}]},{"osd":1,"last update":"Sat Sep 30 01:38:05 2023","interfaces":[{"interface":"back","average":{"1min":0.751,"5min":0.78300000000000003,"15min":0.77700000000000002},"min":{"1min":0.628,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":0.96999999999999997,"5min":1.087,"15min":1.087},"last":0.85499999999999998},{"interface":"front","average":{"1min":0.82899999999999996,"5min":0.81399999999999995,"15min":0.81699999999999995},"min":{"1min":0.67100000000000004,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.97899999999999998,"5min":1.5289999999999999,"15min":1.5289999999999999},"last":0.73999999999999999}]},{"osd":2,"last update":"Sat Sep 30 01:38:05 2023","interfaces":[{"interface":"back","average":{"1min":0.83799999999999997,"5min":0.878,"15min":0.86799999999999999},"min":{"1min":0.67200000000000004,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":1.0820000000000001,"5min":4.0759999999999996,"15min":4.0759999999999996},"last":0.81100000000000005},{"interface":"front","average":{"1min":0.872,"5min":0.91500000000000004,"15min":0.86799999999999999},"min":{"1min":0.65400000000000003,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":1.0629999999999999,"5min":8.3109999999999999,"15min":8.3109999999999999},"last":0.59599999999999997}]},{"osd":3,"last update":"Sat Sep 30 01:38:05 2023","interfaces":[{"interface":"back","average":{"1min":0.751,"5min":0.751,"15min":0.78700000000000003},"min":{"1min":0.51600000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.95699999999999996,"5min":1.446,"15min":1.446},"last":0.67200000000000004},{"interface":"front","average":{"1min":0.98899999999999999,"5min":0.94299999999999995,"15min":0.95899999999999996},"min":{"1min":0.84299999999999997,"5min":0.53900000000000003,"15min":0.53900000000000003},"max":{"1min":1.1299999999999999,"5min":1.4790000000000001,"15min":1.4790000000000001},"last":0.94199999999999995}]},{"osd":4,"last update":"Sat Sep 30 01:38:05 2023","interfaces":[{"interface":"back","average":{"1min":0.85299999999999998,"5min":0.88400000000000001,"15min":0.84199999999999997},"min":{"1min":0.626,"5min":0.441,"15min":0.441},"max":{"1min":1.3109999999999999,"5min":8.2289999999999992,"15min":8.2289999999999992},"last":0.70499999999999996},{"interface":"front","average":{"1min":0.85199999999999998,"5min":0.82199999999999995,"15min":0.83399999999999996},"min":{"1min":0.58999999999999997,"5min":0.52100000000000002,"15min":0.52100000000000002},"max":{"1min":1,"5min":1.2310000000000001,"15min":1.2310000000000001},"last":0.76800000000000002}]}]},{"osd":4,"up_from":11,"seq":47244640319,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5128,"kb_used_data":128,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":94366712,"statfs":{"total":96636764160,"available":96631513088,"internally_reserved":0,"allocated":131072,"data_stored":37781,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 01:37:10 2023","interfaces":[{"interface":"back","average":{"1min":0.82399999999999995,"5min":0.77300000000000002,"15min":0.76900000000000002},"min":{"1min":0.59299999999999997,"5min":0.52500000000000002,"15min":0.52500000000000002},"max":{"1min":1.0169999999999999,"5min":1.0169999999999999,"15min":1.0169999999999999},"last":0.68500000000000005},{"interface":"front","average":{"1min":0.76500000000000001,"5min":0.80000000000000004,"15min":0.80000000000000004},"min":{"1min":0.58899999999999997,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":1.022,"5min":1.6439999999999999,"15min":1.6439999999999999},"last":0.878}]},{"osd":1,"last update":"Sat Sep 30 01:37:10 2023","interfaces":[{"interface":"back","average":{"1min":0.77100000000000002,"5min":0.78700000000000003,"15min":0.78100000000000003},"min":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"max":{"1min":1.028,"5min":1.028,"15min":1.028},"last":0.58699999999999997},{"interface":"front","average":{"1min":0.81200000000000006,"5min":0.79400000000000004,"15min":0.78500000000000003},"min":{"1min":0.59799999999999998,"5min":0.55800000000000005,"15min":0.55800000000000005},"max":{"1min":0.95999999999999996,"5min":1.548,"15min":1.548},"last":0.76300000000000001}]},{"osd":2,"last update":"Sat Sep 30 01:37:10 2023","interfaces":[{"interface":"back","average":{"1min":0.84499999999999997,"5min":0.84899999999999998,"15min":0.86499999999999999},"min":{"1min":0.71299999999999997,"5min":0.625,"15min":0.625},"max":{"1min":1.0449999999999999,"5min":1.1950000000000001,"15min":1.1950000000000001},"last":0.85099999999999998},{"interface":"front","average":{"1min":0.90200000000000002,"5min":0.92000000000000004,"15min":0.93400000000000005},"min":{"1min":0.67500000000000004,"5min":0.64800000000000002,"15min":0.64800000000000002},"max":{"1min":1.0469999999999999,"5min":1.165,"15min":1.165},"last":0.92100000000000004}]},{"osd":3,"last update":"Sat Sep 30 01:37:10 2023","interfaces":[{"interface":"back","average":{"1min":0.72799999999999998,"5min":0.73199999999999998,"15min":0.73799999999999999},"min":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"max":{"1min":0.996,"5min":1.5,"15min":1.5},"last":0.55300000000000005},{"interface":"front","average":{"1min":0.80800000000000005,"5min":0.80100000000000005,"15min":0.81699999999999995},"min":{"1min":0.61099999999999999,"5min":0.54700000000000004,"15min":0.54700000000000004},"max":{"1min":1.1020000000000001,"5min":1.405,"15min":1.405},"last":0.80800000000000005}]},{"osd":5,"last update":"Sat Sep 30 01:37:10 2023","interfaces":[{"interface":"back","average":{"1min":0.80900000000000005,"5min":0.79000000000000004,"15min":0.77900000000000003},"min":{"1min":0.56000000000000005,"5min":0.54800000000000004,"15min":0.54800000000000004},"max":{"1min":1.0860000000000001,"5min":1.7270000000000001,"15min":1.7270000000000001},"last":0.626},{"interface":"front","average":{"1min":0.90600000000000003,"5min":0.90300000000000002,"15min":0.90500000000000003},"min":{"1min":0.80900000000000005,"5min":0.66800000000000004,"15min":0.66800000000000004},"max":{"1min":1.089,"5min":1.0960000000000001,"15min":1.0960000000000001},"last":0.72699999999999998}]}]},{"osd":3,"up_from":11,"seq":47244640320,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6096,"kb_used_data":648,"kb_used_omap":0,"kb_used_meta":5440,"kb_avail":94365744,"statfs":{"total":96636764160,"available":96630521856,"internally_reserved":0,"allocated":663552,"data_stored":566709,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5570560},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 01:37:03 2023","interfaces":[{"interface":"back","average":{"1min":0.83499999999999996,"5min":0.78300000000000003,"15min":0.77300000000000002},"min":{"1min":0.48299999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":1.052,"5min":1.052,"15min":1.052},"last":0.92500000000000004},{"interface":"front","average":{"1min":0.751,"5min":0.75,"15min":0.73499999999999999},"min":{"1min":0.61199999999999999,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":0.96799999999999997,"5min":1.206,"15min":1.206},"last":0.627}]},{"osd":1,"last update":"Sat Sep 30 01:37:03 2023","interfaces":[{"interface":"back","average":{"1min":0.84099999999999997,"5min":0.83099999999999996,"15min":0.82899999999999996},"min":{"1min":0.59799999999999998,"5min":0.56599999999999995,"15min":0.56599999999999995},"max":{"1min":0.96899999999999997,"5min":1.159,"15min":1.159},"last":0.876},{"interface":"front","average":{"1min":0.80000000000000004,"5min":0.77700000000000002,"15min":0.76100000000000001},"min":{"1min":0.56799999999999995,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":1.016,"5min":1.137,"15min":1.137},"last":0.84399999999999997}]},{"osd":2,"last update":"Sat Sep 30 01:37:03 2023","interfaces":[{"interface":"back","average":{"1min":0.86099999999999999,"5min":0.90700000000000003,"15min":0.93999999999999995},"min":{"1min":0.67700000000000005,"5min":0.60599999999999998,"15min":0.60599999999999998},"max":{"1min":1.0269999999999999,"5min":1.296,"15min":1.296},"last":0.98199999999999998},{"interface":"front","average":{"1min":0.88300000000000001,"5min":0.90700000000000003,"15min":0.92500000000000004},"min":{"1min":0.76300000000000001,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":1.0009999999999999,"5min":1.46,"15min":1.46},"last":0.69399999999999995}]},{"osd":4,"last update":"Sat Sep 30 01:37:03 2023","interfaces":[{"interface":"back","average":{"1min":0.749,"5min":0.73699999999999999,"15min":0.73499999999999999},"min":{"1min":0.57399999999999995,"5min":0.51300000000000001,"15min":0.51300000000000001},"max":{"1min":1.0389999999999999,"5min":1.3220000000000001,"15min":1.3220000000000001},"last":0.56399999999999995},{"interface":"front","average":{"1min":0.79200000000000004,"5min":0.84199999999999997,"15min":0.876},"min":{"1min":0.61799999999999999,"5min":0.56299999999999994,"15min":0.56299999999999994},"max":{"1min":1.046,"5min":1.0800000000000001,"15min":1.0800000000000001},"last":0.71599999999999997}]},{"osd":5,"last update":"Sat Sep 30 01:37:03 2023","interfaces":[{"interface":"back","average":{"1min":0.81899999999999995,"5min":0.81499999999999995,"15min":0.82799999999999996},"min":{"1min":0.64400000000000002,"5min":0.61099999999999999,"15min":0.61099999999999999},"max":{"1min":0.95499999999999996,"5min":1.3320000000000001,"15min":1.3320000000000001},"last":0.76500000000000001},{"interface":"front","average":{"1min":0.79700000000000004,"5min":0.84699999999999998,"15min":0.871},"min":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"max":{"1min":1.0740000000000001,"5min":1.387,"15min":1.387},"last":0.80500000000000005}]}]},{"osd":2,"up_from":11,"seq":47244640321,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5128,"kb_used_data":128,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":94366712,"statfs":{"total":96636764160,"available":96631513088,"internally_reserved":0,"allocated":131072,"data_stored":37781,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 01:37:06 2023","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.57599999999999996,"15min":0.56399999999999995},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.95099999999999996,"5min":0.95099999999999996,"15min":0.95099999999999996},"last":0.49299999999999999},{"interface":"front","average":{"1min":0.70899999999999996,"5min":0.67600000000000005,"15min":0.65700000000000003},"min":{"1min":0.46500000000000002,"5min":0.439,"15min":0.439},"max":{"1min":2.2029999999999998,"5min":2.2029999999999998,"15min":2.2029999999999998},"last":0.57599999999999996}]},{"osd":1,"last update":"Sat Sep 30 01:37:06 2023","interfaces":[{"interface":"back","average":{"1min":0.71799999999999997,"5min":0.70399999999999996,"15min":0.70799999999999996},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":2.2490000000000001,"5min":2.2490000000000001,"15min":2.2490000000000001},"last":0.69899999999999995},{"interface":"front","average":{"1min":0.64300000000000002,"5min":0.64600000000000002,"15min":0.65500000000000003},"min":{"1min":0.46899999999999997,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.85799999999999998,"5min":1.0409999999999999,"15min":1.0409999999999999},"last":0.73099999999999998}]},{"osd":3,"last update":"Sat Sep 30 01:37:06 2023","interfaces":[{"interface":"back","average":{"1min":0.77100000000000002,"5min":0.82899999999999996,"15min":0.83599999999999997},"min":{"1min":0.61199999999999999,"5min":0.61199999999999999,"15min":0.61199999999999999},"max":{"1min":1.0660000000000001,"5min":1.343,"15min":1.343},"last":0.66700000000000004},{"interface":"front","average":{"1min":0.96299999999999997,"5min":0.92200000000000004,"15min":0.92500000000000004},"min":{"1min":0.76700000000000002,"5min":0.60499999999999998,"15min":0.60499999999999998},"max":{"1min":2.4390000000000001,"5min":2.4390000000000001,"15min":2.4390000000000001},"last":0.93999999999999995}]},{"osd":4,"last update":"Sat Sep 30 01:37:06 2023","interfaces":[{"interface":"back","average":{"1min":0.94699999999999995,"5min":0.90000000000000002,"15min":0.90800000000000003},"min":{"1min":0.72699999999999998,"5min":0.64300000000000002,"15min":0.64300000000000002},"max":{"1min":2.3679999999999999,"5min":2.3679999999999999,"15min":2.3679999999999999},"last":0.83099999999999996},{"interface":"front","average":{"1min":0.85699999999999998,"5min":0.84799999999999998,"15min":0.84799999999999998},"min":{"1min":0.68500000000000005,"5min":0.505,"15min":0.505},"max":{"1min":1.401,"5min":1.401,"15min":1.401},"last":0.85299999999999998}]},{"osd":5,"last update":"Sat Sep 30 01:37:06 2023","interfaces":[{"interface":"back","average":{"1min":0.85199999999999998,"5min":0.86199999999999999,"15min":0.86599999999999999},"min":{"1min":0.628,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":1.0629999999999999,"5min":1.3959999999999999,"15min":1.3959999999999999},"last":0.77600000000000002},{"interface":"front","average":{"1min":0.88900000000000001,"5min":0.91500000000000004,"15min":0.92700000000000005},"min":{"1min":0.71099999999999997,"5min":0.61799999999999999,"15min":0.61799999999999999},"max":{"1min":1.234,"5min":1.625,"15min":1.625},"last":0.90700000000000003}]}]},{"osd":1,"up_from":11,"seq":47244640320,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6096,"kb_used_data":648,"kb_used_omap":0,"kb_used_meta":5440,"kb_avail":94365744,"statfs":{"total":96636764160,"available":96630521856,"internally_reserved":0,"allocated":663552,"data_stored":566709,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5570560},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 01:37:07 2023","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.57299999999999995,"15min":0.57199999999999995},"min":{"1min":0.48999999999999999,"5min":0.317,"15min":0.317},"max":{"1min":0.86299999999999999,"5min":0.92400000000000004,"15min":0.92400000000000004},"last":0.54300000000000004},{"interface":"front","average":{"1min":0.76800000000000002,"5min":0.65300000000000002,"15min":0.65200000000000002},"min":{"1min":0.47499999999999998,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":2.085,"5min":2.085,"15min":2.085},"last":0.71599999999999997}]},{"osd":2,"last update":"Sat Sep 30 01:37:07 2023","interfaces":[{"interface":"back","average":{"1min":0.64800000000000002,"5min":0.64400000000000002,"15min":0.66900000000000004},"min":{"1min":0.442,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.80900000000000005,"5min":1.159,"15min":1.159},"last":0.58199999999999996},{"interface":"front","average":{"1min":0.749,"5min":0.69899999999999995,"15min":0.70099999999999996},"min":{"1min":0.58799999999999997,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":1.9930000000000001,"5min":1.9930000000000001,"15min":1.9930000000000001},"last":0.66800000000000004}]},{"osd":3,"last update":"Sat Sep 30 01:37:07 2023","interfaces":[{"interface":"back","average":{"1min":0.81599999999999995,"5min":0.82199999999999995,"15min":0.82499999999999996},"min":{"1min":0.54500000000000004,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":0.98899999999999999,"5min":1.1499999999999999,"15min":1.1499999999999999},"last":0.80200000000000005},{"interface":"front","average":{"1min":0.83599999999999997,"5min":0.83299999999999996,"15min":0.84099999999999997},"min":{"1min":0.69999999999999996,"5min":0.61699999999999999,"15min":0.61699999999999999},"max":{"1min":1.024,"5min":1.2929999999999999,"15min":1.2929999999999999},"last":0.88800000000000001}]},{"osd":4,"last update":"Sat Sep 30 01:37:07 2023","interfaces":[{"interface":"back","average":{"1min":0.84899999999999998,"5min":0.86399999999999999,"15min":0.88100000000000001},"min":{"1min":0.60399999999999998,"5min":0.53600000000000003,"15min":0.53600000000000003},"max":{"1min":0.97499999999999998,"5min":1.415,"15min":1.415},"last":0.83199999999999996},{"interface":"front","average":{"1min":0.878,"5min":0.876,"15min":0.88700000000000001},"min":{"1min":0.59299999999999997,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":1.004,"5min":1.381,"15min":1.381},"last":0.94499999999999995}]},{"osd":5,"last update":"Sat Sep 30 01:37:07 2023","interfaces":[{"interface":"back","average":{"1min":0.93300000000000005,"5min":0.90600000000000003,"15min":0.90500000000000003},"min":{"1min":0.77400000000000002,"5min":0.66200000000000003,"15min":0.66200000000000003},"max":{"1min":1.294,"5min":1.395,"15min":1.395},"last":1.0109999999999999},{"interface":"front","average":{"1min":0.98199999999999998,"5min":0.95799999999999996,"15min":0.96299999999999997},"min":{"1min":0.753,"5min":0.56299999999999994,"15min":0.56299999999999994},"max":{"1min":1.349,"5min":1.4179999999999999,"15min":1.4179999999999999},"last":1.077}]}]},{"osd":0,"up_from":11,"seq":47244640321,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5128,"kb_used_data":128,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":94366712,"statfs":{"total":96636764160,"available":96631513088,"internally_reserved":0,"allocated":131072,"data_stored":37781,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Sep 30 01:37:05 2023","interfaces":[{"interface":"back","average":{"1min":0.57399999999999995,"5min":0.58299999999999996,"15min":0.59299999999999997},"min":{"1min":0.48899999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.69099999999999995,"5min":0.92900000000000005,"15min":0.92900000000000005},"last":0.66700000000000004},{"interface":"front","average":{"1min":0.67400000000000004,"5min":0.64300000000000002,"15min":0.64100000000000001},"min":{"1min":0.42599999999999999,"5min":0.314,"15min":0.314},"max":{"1min":0.98599999999999999,"5min":0.98599999999999999,"15min":0.98599999999999999},"last":0.54000000000000004}]},{"osd":2,"last update":"Sat Sep 30 01:37:05 2023","interfaces":[{"interface":"back","average":{"1min":0.67700000000000005,"5min":0.628,"15min":0.61899999999999999},"min":{"1min":0.53000000000000003,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.93799999999999994,"5min":0.93799999999999994,"15min":0.93799999999999994},"last":0.86299999999999999},{"interface":"front","average":{"1min":0.754,"5min":0.72199999999999998,"15min":0.70799999999999996},"min":{"1min":0.56499999999999995,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":1.2669999999999999,"5min":2.069,"15min":2.069},"last":0.63700000000000001}]},{"osd":3,"last update":"Sat Sep 30 01:37:05 2023","interfaces":[{"interface":"back","average":{"1min":0.80200000000000005,"5min":0.84299999999999997,"15min":0.85599999999999998},"min":{"1min":0.67600000000000005,"5min":0.60299999999999998,"15min":0.60299999999999998},"max":{"1min":0.89800000000000002,"5min":1.1319999999999999,"15min":1.1319999999999999},"last":0.84099999999999997},{"interface":"front","average":{"1min":0.88600000000000001,"5min":0.88300000000000001,"15min":0.88},"min":{"1min":0.68400000000000005,"5min":0.62,"15min":0.62},"max":{"1min":1.0649999999999999,"5min":2.4870000000000001,"15min":2.4870000000000001},"last":1.024}]},{"osd":4,"last update":"Sat Sep 30 01:37:05 2023","interfaces":[{"interface":"back","average":{"1min":0.92600000000000005,"5min":0.92400000000000004,"15min":0.92600000000000005},"min":{"1min":0.74299999999999999,"5min":0.66300000000000003,"15min":0.66300000000000003},"max":{"1min":1.3,"5min":2.589,"15min":2.589},"last":0.78700000000000003},{"interface":"front","average":{"1min":0.86799999999999999,"5min":0.878,"15min":0.90800000000000003},"min":{"1min":0.70099999999999996,"5min":0.54900000000000004,"15min":0.54900000000000004},"max":{"1min":1.2170000000000001,"5min":1.413,"15min":1.413},"last":0.93999999999999995}]},{"osd":5,"last update":"Sat Sep 30 01:37:05 2023","interfaces":[{"interface":"back","average":{"1min":0.92800000000000005,"5min":0.90900000000000003,"15min":0.92900000000000005},"min":{"1min":0.68000000000000005,"5min":0.58799999999999997,"15min":0.58799999999999997},"max":{"1min":1.1599999999999999,"5min":1.3200000000000001,"15min":1.3200000000000001},"last":0.81000000000000005},{"interface":"front","average":{"1min":0.97299999999999998,"5min":0.96399999999999997,"15min":0.96899999999999997},"min":{"1min":0.73999999999999999,"5min":0.73999999999999999,"15min":0.73999999999999999},"max":{"1min":1.3879999999999999,"5min":2.4279999999999999,"15min":2.4279999999999999},"last":0.99199999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-09-30T01:38:10.461 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-09-30T01:38:10.563 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.559+0000 7f1e8ae26700 10 monclient: get_monmap_and_config 2023-09-30T01:38:10.563 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.559+0000 7f1e8ae26700 10 monclient: build_initial_monmap 2023-09-30T01:38:10.563 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.559+0000 7f1e8ae26700 10 monclient: monmap: 2023-09-30T01:38:10.563 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2023-09-30T01:38:10.563 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:38:10.563 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:38:10.562905+0000 2023-09-30T01:38:10.563 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:38:10.562905+0000 2023-09-30T01:38:10.563 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2023-09-30T01:38:10.564 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:38:10.564 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a-legacy 2023-09-30T01:38:10.564 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.107:6789/0 mon.noname-f-legacy 2023-09-30T01:38:10.564 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.noname-b 2023-09-30T01:38:10.564 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.noname-g 2023-09-30T01:38:10.564 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.noname-c 2023-09-30T01:38:10.564 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.noname-h 2023-09-30T01:38:10.564 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.noname-d 2023-09-30T01:38:10.564 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.noname-i 2023-09-30T01:38:10.564 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.noname-e 2023-09-30T01:38:10.564 INFO:teuthology.orchestra.run.smithi088.stderr:9: v2:172.21.15.88:3300/0 mon.noname-a 2023-09-30T01:38:10.564 INFO:teuthology.orchestra.run.smithi088.stderr:10: v2:172.21.15.107:3300/0 mon.noname-f 2023-09-30T01:38:10.565 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:38:10.572 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.559+0000 7f1e8ae26700 1 Processor -- start 2023-09-30T01:38:10.572 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.559+0000 7f1e8ae26700 1 -- start start 2023-09-30T01:38:10.572 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.559+0000 7f1e8ae26700 10 monclient: init 2023-09-30T01:38:10.572 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.563+0000 7f1e8ae26700 10 monclient: _reopen_session rank -1 2023-09-30T01:38:10.572 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.563+0000 7f1e8ae26700 10 monclient: _add_conns ranks=[0,2,7,9,8,3,6,5,4,1,10] 2023-09-30T01:38:10.573 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.563+0000 7f1e8ae26700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f1e840689c0 addr v1:172.21.15.88:6789/0 2023-09-30T01:38:10.573 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.563+0000 7f1e8ae26700 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1e84068d90 0x7f1e84138640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:38:10.573 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.563+0000 7f1e8ae26700 10 monclient(hunting): picked mon.noname-b con 0x7f1e84068d90 addr [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] 2023-09-30T01:38:10.573 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.563+0000 7f1e8ae26700 1 --2- >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7f1e84069160 0x7f1e8413ab90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:38:10.573 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.563+0000 7f1e8ae26700 10 monclient(hunting): picked mon.noname-i con 0x7f1e84069160 addr [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] 2023-09-30T01:38:10.573 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.563+0000 7f1e8ae26700 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e8405ecb0 con 0x7f1e840689c0 2023-09-30T01:38:10.573 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.563+0000 7f1e8ae26700 10 monclient(hunting): start opening mon connection 2023-09-30T01:38:10.573 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.563+0000 7f1e8ae26700 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e8406d180 con 0x7f1e84068d90 2023-09-30T01:38:10.573 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.563+0000 7f1e8ae26700 10 monclient(hunting): start opening mon connection 2023-09-30T01:38:10.573 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.563+0000 7f1e8ae26700 1 -- --> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] -- mon_getmap magic: 0 v1 -- 0x7f1e84069d40 con 0x7f1e84069160 2023-09-30T01:38:10.573 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.563+0000 7f1e8ae26700 10 monclient(hunting): _renew_subs 2023-09-30T01:38:10.573 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.563+0000 7f1e8ae26700 10 monclient(hunting): authenticate will time out at 1162.600326s 2023-09-30T01:38:10.574 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.563+0000 7f1e83fff700 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1e84068d90 0x7f1e84138640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:38:10.574 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.563+0000 7f1e88bc2700 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f1e840689c0 0x7f1e8406e1a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:58008/0 (socket says 172.21.15.88:58008) 2023-09-30T01:38:10.574 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.563+0000 7f1e88bc2700 1 -- 172.21.15.88:0/2375332507 learned_addr learned my addr 172.21.15.88:0/2375332507 (peer_addr_for_me v1:172.21.15.88:0/0) 2023-09-30T01:38:10.574 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.563+0000 7f1e83fff700 10 monclient(hunting): get_auth_request con 0x7f1e84068d90 auth_method 0 2023-09-30T01:38:10.574 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.563+0000 7f1e83fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:38:10.574 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.563+0000 7f1e83fff700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:38:10.574 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.563+0000 7f1e83fff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:38:10.574 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.563+0000 7f1e893c3700 1 --2- 172.21.15.88:0/2375332507 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7f1e84069160 0x7f1e8413ab90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:38:10.574 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.563+0000 7f1e83fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:38:10.574 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.563+0000 7f1e83fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:38:10.574 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.563+0000 7f1e83fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:38:10.575 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.563+0000 7f1e893c3700 10 monclient(hunting): get_auth_request con 0x7f1e84069160 auth_method 0 2023-09-30T01:38:10.575 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.563+0000 7f1e893c3700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:38:10.575 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.563+0000 7f1e893c3700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:38:10.575 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.563+0000 7f1e893c3700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:38:10.575 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.563+0000 7f1e893c3700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:38:10.575 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.563+0000 7f1e893c3700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:38:10.575 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.563+0000 7f1e893c3700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:38:10.575 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.563+0000 7f1e83fff700 10 monclient(hunting): handle_auth_done global_id 4826 payload 274 2023-09-30T01:38:10.575 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.563+0000 7f1e83fff700 1 -- 172.21.15.88:0/2375332507 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7f1e84069160 msgr2=0x7f1e8413ab90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:38:10.575 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.563+0000 7f1e83fff700 1 --2- 172.21.15.88:0/2375332507 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7f1e84069160 0x7f1e8413ab90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:38:10.575 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.563+0000 7f1e837fe700 1 -- 172.21.15.88:0/2375332507 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1573398577 0 0) 0x7f1e8405ecb0 con 0x7f1e840689c0 2023-09-30T01:38:10.575 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.563+0000 7f1e83fff700 1 -- 172.21.15.88:0/2375332507 >> v1:172.21.15.88:6789/0 conn(0x7f1e840689c0 legacy=0x7f1e8406e1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:38:10.575 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.563+0000 7f1e83fff700 10 monclient: _finish_hunting 0 2023-09-30T01:38:10.576 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.563+0000 7f1e83fff700 1 monclient: found mon.noname-b 2023-09-30T01:38:10.576 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.563+0000 7f1e83fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:38:10.576 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.563+0000 7f1e83fff700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.88:3301/0 2023-09-30T01:38:10.576 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.563+0000 7f1e83fff700 1 -- 172.21.15.88:0/2375332507 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e8406d460 con 0x7f1e84068d90 2023-09-30T01:38:10.576 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.563+0000 7f1e837fe700 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2023-09-30T01:38:10.576 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.563+0000 7f1e83fff700 1 --2- 172.21.15.88:0/2375332507 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1e84068d90 0x7f1e84138640 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f1e74009910 tx=0x7f1e740091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3cb6e77625514d55 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:38:10.576 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.563+0000 7f1e837fe700 1 -- 172.21.15.88:0/2375332507 <== mon.2 v2:172.21.15.88:3301/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f1e7400bb00 con 0x7f1e84068d90 2023-09-30T01:38:10.576 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.563+0000 7f1e837fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:38:10.576 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.563+0000 7f1e837fe700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T01:38:10.576 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.563+0000 7f1e837fe700 10 monclient: dump: 2023-09-30T01:38:10.576 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:38:10.576 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:38:10.577 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:38:10.577 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:38:10.577 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:38:10.577 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:38:10.577 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:38:10.577 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:38:10.577 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:38:10.577 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:38:10.577 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:38:10.577 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:38:10.577 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:38:10.577 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:38:10.578 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:38:10.578 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:38:10.578 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.563+0000 7f1e837fe700 10 monclient: _finish_auth 0 2023-09-30T01:38:10.578 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.563+0000 7f1e837fe700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:38:10.578 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.563+0000 7f1e8ae26700 5 monclient: authenticate success, global_id 4826 2023-09-30T01:38:10.578 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.563+0000 7f1e837fe700 1 -- 172.21.15.88:0/2375332507 <== mon.2 v2:172.21.15.88:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e7400bc80 con 0x7f1e84068d90 2023-09-30T01:38:10.578 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.563+0000 7f1e8ae26700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:38:10.578 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.563+0000 7f1e837fe700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:38:10.578 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.563+0000 7f1e8ae26700 10 monclient: get_monmap_and_config success 2023-09-30T01:38:10.578 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.563+0000 7f1e837fe700 1 -- 172.21.15.88:0/2375332507 <== mon.2 v2:172.21.15.88:3301/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f1e7400bdf0 con 0x7f1e84068d90 2023-09-30T01:38:10.578 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.563+0000 7f1e8ae26700 4 set_mon_vals no callback set 2023-09-30T01:38:10.578 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.563+0000 7f1e837fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:38:10.579 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.563+0000 7f1e837fe700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T01:38:10.579 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.563+0000 7f1e837fe700 10 monclient: dump: 2023-09-30T01:38:10.579 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:38:10.579 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:38:10.579 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:38:10.579 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:38:10.579 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:38:10.579 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:38:10.579 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:38:10.579 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:38:10.579 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:38:10.579 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:38:10.580 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:38:10.580 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:38:10.580 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:38:10.580 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:38:10.580 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:38:10.580 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:38:10.580 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.563+0000 7f1e8ae26700 10 monclient: shutdown 2023-09-30T01:38:10.580 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.563+0000 7f1e8ae26700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:38:10.580 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.563+0000 7f1e8ae26700 1 -- 172.21.15.88:0/2375332507 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1e84068d90 msgr2=0x7f1e84138640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:38:10.580 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.563+0000 7f1e8ae26700 1 --2- 172.21.15.88:0/2375332507 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1e84068d90 0x7f1e84138640 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f1e74009910 tx=0x7f1e740091a0 comp rx=0 tx=0).stop 2023-09-30T01:38:10.580 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.563+0000 7f1e8ae26700 1 -- 172.21.15.88:0/2375332507 shutdown_connections 2023-09-30T01:38:10.580 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.563+0000 7f1e8ae26700 1 --2- 172.21.15.88:0/2375332507 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7f1e84069160 0x7f1e8413ab90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:38:10.581 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.563+0000 7f1e8ae26700 1 --2- 172.21.15.88:0/2375332507 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1e84068d90 0x7f1e84138640 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:38:10.581 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.563+0000 7f1e8ae26700 1 -- 172.21.15.88:0/2375332507 >> 172.21.15.88:0/2375332507 conn(0x7f1e84063710 msgr2=0x7f1e840a46d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:38:10.581 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.563+0000 7f1e8ae26700 1 -- 172.21.15.88:0/2375332507 shutdown_connections 2023-09-30T01:38:10.581 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.563+0000 7f1e8ae26700 1 -- 172.21.15.88:0/2375332507 wait complete. 2023-09-30T01:38:10.581 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.563+0000 7f1e8ae26700 10 monclient: build_initial_monmap 2023-09-30T01:38:10.582 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.563+0000 7f1e8ae26700 10 monclient: monmap: 2023-09-30T01:38:10.582 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2023-09-30T01:38:10.582 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:38:10.582 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 0.000000 2023-09-30T01:38:10.582 INFO:teuthology.orchestra.run.smithi088.stderr:created 0.000000 2023-09-30T01:38:10.582 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2023-09-30T01:38:10.582 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 1 2023-09-30T01:38:10.583 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.noname-a 2023-09-30T01:38:10.583 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.noname-f 2023-09-30T01:38:10.583 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.noname-b 2023-09-30T01:38:10.583 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.noname-g 2023-09-30T01:38:10.583 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.noname-c 2023-09-30T01:38:10.583 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.noname-h 2023-09-30T01:38:10.583 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.noname-d 2023-09-30T01:38:10.583 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.noname-i 2023-09-30T01:38:10.583 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.noname-e 2023-09-30T01:38:10.583 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:38:10.583 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.563+0000 7f1e8ae26700 1 Processor -- start 2023-09-30T01:38:10.583 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.563+0000 7f1e8ae26700 1 -- start start 2023-09-30T01:38:10.584 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.563+0000 7f1e8ae26700 10 monclient: init 2023-09-30T01:38:10.584 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.567+0000 7f1e8ae26700 10 monclient: _reopen_session rank -1 2023-09-30T01:38:10.584 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.567+0000 7f1e8ae26700 10 monclient: _add_conns ranks=[1,4,8,2,5,0,3,6,7] 2023-09-30T01:38:10.584 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.567+0000 7f1e8ae26700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1e840689c0 0x7f1e84143040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:38:10.584 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.567+0000 7f1e8ae26700 10 monclient(hunting): picked mon.noname-f con 0x7f1e840689c0 addr [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] 2023-09-30T01:38:10.584 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.567+0000 7f1e8ae26700 1 --2- >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f1e84068d90 0x7f1e8412fd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:38:10.584 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.567+0000 7f1e8ae26700 10 monclient(hunting): picked mon.noname-c con 0x7f1e84068d90 addr [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] 2023-09-30T01:38:10.584 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.567+0000 7f1e8ae26700 1 --2- >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f1e84069160 0x7f1e8411ad30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:38:10.584 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.567+0000 7f1e8ae26700 10 monclient(hunting): picked mon.noname-e con 0x7f1e84069160 addr [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] 2023-09-30T01:38:10.584 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.567+0000 7f1e8ae26700 10 monclient(hunting): start opening mon connection 2023-09-30T01:38:10.584 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.567+0000 7f1e8ae26700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e8406e9b0 con 0x7f1e840689c0 2023-09-30T01:38:10.584 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.567+0000 7f1e8ae26700 10 monclient(hunting): start opening mon connection 2023-09-30T01:38:10.585 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.567+0000 7f1e8ae26700 1 -- --> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] -- mon_getmap magic: 0 v1 -- 0x7f1e8411d190 con 0x7f1e84068d90 2023-09-30T01:38:10.585 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.567+0000 7f1e8ae26700 10 monclient(hunting): start opening mon connection 2023-09-30T01:38:10.585 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.567+0000 7f1e8ae26700 1 -- --> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] -- mon_getmap magic: 0 v1 -- 0x7f1e8406c640 con 0x7f1e84069160 2023-09-30T01:38:10.585 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.567+0000 7f1e8ae26700 10 monclient(hunting): _renew_subs 2023-09-30T01:38:10.585 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.567+0000 7f1e8ae26700 10 monclient(hunting): authenticate will time out at 1162.605248s 2023-09-30T01:38:10.585 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.567+0000 7f1e893c3700 1 --2- >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f1e84069160 0x7f1e8411ad30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:38:10.585 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.567+0000 7f1e83fff700 1 --2- >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f1e84068d90 0x7f1e8412fd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:38:10.585 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.567+0000 7f1e893c3700 1 --2- >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f1e84069160 0x7f1e8411ad30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3304/0 says I am v2:172.21.15.88:35718/0 (socket says 172.21.15.88:35718) 2023-09-30T01:38:10.585 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.567+0000 7f1e893c3700 1 -- 172.21.15.88:0/4209953737 learned_addr learned my addr 172.21.15.88:0/4209953737 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-09-30T01:38:10.585 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.567+0000 7f1e893c3700 10 monclient(hunting): get_auth_request con 0x7f1e84069160 auth_method 0 2023-09-30T01:38:10.585 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.567+0000 7f1e893c3700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:38:10.585 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.567+0000 7f1e893c3700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:38:10.586 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.567+0000 7f1e893c3700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:38:10.586 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.567+0000 7f1e83fff700 1 --2- >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f1e84068d90 0x7f1e8412fd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3302/0 says I am v2:172.21.15.88:33928/0 (socket says 172.21.15.88:33928) 2023-09-30T01:38:10.586 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.567+0000 7f1e83fff700 10 monclient(hunting): get_auth_request con 0x7f1e84068d90 auth_method 0 2023-09-30T01:38:10.586 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.567+0000 7f1e83fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:38:10.586 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.567+0000 7f1e83fff700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:38:10.586 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.567+0000 7f1e83fff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:38:10.586 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.567+0000 7f1e88bc2700 1 --2- 172.21.15.88:0/4209953737 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1e840689c0 0x7f1e84143040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:38:10.586 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.567+0000 7f1e88bc2700 10 monclient(hunting): get_auth_request con 0x7f1e840689c0 auth_method 0 2023-09-30T01:38:10.586 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.567+0000 7f1e88bc2700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:38:10.586 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.567+0000 7f1e88bc2700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:38:10.586 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.567+0000 7f1e88bc2700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:38:10.587 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.567+0000 7f1e893c3700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:38:10.587 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.567+0000 7f1e893c3700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:38:10.587 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.567+0000 7f1e893c3700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:38:10.587 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.567+0000 7f1e88bc2700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:38:10.587 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.567+0000 7f1e88bc2700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:38:10.587 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.567+0000 7f1e88bc2700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:38:10.587 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.567+0000 7f1e83fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:38:10.587 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.567+0000 7f1e83fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:38:10.587 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.567+0000 7f1e83fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:38:10.587 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.567+0000 7f1e893c3700 10 monclient(hunting): handle_auth_done global_id 4904 payload 824 2023-09-30T01:38:10.587 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.567+0000 7f1e893c3700 1 -- 172.21.15.88:0/4209953737 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f1e84068d90 msgr2=0x7f1e8412fd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:38:10.587 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.567+0000 7f1e893c3700 1 --2- 172.21.15.88:0/4209953737 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f1e84068d90 0x7f1e8412fd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:38:10.587 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.567+0000 7f1e893c3700 1 -- 172.21.15.88:0/4209953737 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1e840689c0 msgr2=0x7f1e84143040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:38:10.588 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.567+0000 7f1e893c3700 1 --2- 172.21.15.88:0/4209953737 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1e840689c0 0x7f1e84143040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:38:10.588 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.567+0000 7f1e893c3700 10 monclient: _finish_hunting 0 2023-09-30T01:38:10.588 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.567+0000 7f1e893c3700 1 monclient: found mon.noname-e 2023-09-30T01:38:10.588 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.567+0000 7f1e893c3700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:38:10.588 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.567+0000 7f1e893c3700 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.88:3304/0 2023-09-30T01:38:10.588 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.567+0000 7f1e893c3700 1 -- 172.21.15.88:0/4209953737 --> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e74009420 con 0x7f1e84069160 2023-09-30T01:38:10.588 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.567+0000 7f1e893c3700 1 --2- 172.21.15.88:0/4209953737 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f1e84069160 0x7f1e8411ad30 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f1e7c009f20 tx=0x7f1e7c00b040 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:38:10.588 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.567+0000 7f1e80ff9700 1 -- 172.21.15.88:0/4209953737 <== mon.8 v2:172.21.15.88:3304/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f1e7c014480 con 0x7f1e84069160 2023-09-30T01:38:10.588 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.567+0000 7f1e80ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:38:10.588 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.567+0000 7f1e80ff9700 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2023-09-30T01:38:10.588 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.567+0000 7f1e80ff9700 10 monclient: dump: 2023-09-30T01:38:10.588 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:38:10.589 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:38:10.589 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:38:10.589 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:38:10.589 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:38:10.589 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:38:10.589 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:38:10.589 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:38:10.589 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:38:10.589 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:38:10.589 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:38:10.589 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:38:10.589 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:38:10.589 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:38:10.590 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:38:10.590 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:38:10.590 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.567+0000 7f1e80ff9700 10 monclient: _finish_auth 0 2023-09-30T01:38:10.590 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.567+0000 7f1e80ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:38:10.590 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.567+0000 7f1e80ff9700 1 -- 172.21.15.88:0/4209953737 <== mon.8 v2:172.21.15.88:3304/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e7c00f040 con 0x7f1e84069160 2023-09-30T01:38:10.590 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.567+0000 7f1e8ae26700 5 monclient: authenticate success, global_id 4904 2023-09-30T01:38:10.590 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.567+0000 7f1e8ae26700 10 monclient: _renew_subs 2023-09-30T01:38:10.590 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.567+0000 7f1e8ae26700 10 monclient: _send_mon_message to mon.e at v2:172.21.15.88:3304/0 2023-09-30T01:38:10.590 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.567+0000 7f1e8ae26700 1 -- 172.21.15.88:0/4209953737 --> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1e8411d750 con 0x7f1e84069160 2023-09-30T01:38:10.590 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.567+0000 7f1e80ff9700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:38:10.590 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.567+0000 7f1e80ff9700 1 -- 172.21.15.88:0/4209953737 <== mon.8 v2:172.21.15.88:3304/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f1e7c014aa0 con 0x7f1e84069160 2023-09-30T01:38:10.590 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.567+0000 7f1e80ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:38:10.591 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.567+0000 7f1e80ff9700 10 monclient: got monmap 1 from mon.e (according to old e1) 2023-09-30T01:38:10.591 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.567+0000 7f1e80ff9700 10 monclient: dump: 2023-09-30T01:38:10.591 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:38:10.591 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:38:10.591 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:38:10.591 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:38:10.591 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:38:10.591 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:38:10.591 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:38:10.591 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:38:10.591 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:38:10.592 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:38:10.592 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:38:10.592 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:38:10.592 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:38:10.592 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:38:10.592 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:38:10.592 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:38:10.592 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.567+0000 7f1e82ffd700 4 set_mon_vals no callback set 2023-09-30T01:38:10.593 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.567+0000 7f1e8ae26700 10 monclient: _renew_subs 2023-09-30T01:38:10.593 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.567+0000 7f1e8ae26700 10 monclient: _send_mon_message to mon.e at v2:172.21.15.88:3304/0 2023-09-30T01:38:10.593 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.567+0000 7f1e8ae26700 1 -- 172.21.15.88:0/4209953737 --> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1e84064b80 con 0x7f1e84069160 2023-09-30T01:38:10.593 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.571+0000 7f1e8ae26700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:38:10.593 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.571+0000 7f1e8ae26700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:38:10.593 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.571+0000 7f1e8ae26700 10 monclient: _send_mon_message to mon.e at v2:172.21.15.88:3304/0 2023-09-30T01:38:10.593 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.571+0000 7f1e8ae26700 1 -- 172.21.15.88:0/4209953737 --> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1e8411d300 con 0x7f1e84069160 2023-09-30T01:38:10.593 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.571+0000 7f1e80ff9700 1 -- 172.21.15.88:0/4209953737 <== mon.8 v2:172.21.15.88:3304/0 4 ==== mgrmap(e 5) v1 ==== 40144+0+0 (secure 0 0 0) 0x7f1e7c014c20 con 0x7f1e84069160 2023-09-30T01:38:10.593 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.571+0000 7f1e80ff9700 1 --2- 172.21.15.88:0/4209953737 >> v2:172.21.15.107:6812/29926 conn(0x7f1e70033360 0x7f1e70035810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:38:10.593 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.571+0000 7f1e80ff9700 1 -- 172.21.15.88:0/4209953737 <== mon.8 v2:172.21.15.88:3304/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 3502+0+0 (secure 0 0 0) 0x7f1e7c043fc0 con 0x7f1e84069160 2023-09-30T01:38:10.594 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.571+0000 7f1e88bc2700 1 --2- 172.21.15.88:0/4209953737 >> v2:172.21.15.107:6812/29926 conn(0x7f1e70033360 0x7f1e70035810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:38:10.594 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.571+0000 7f1e88bc2700 10 monclient: get_auth_request con 0x7f1e70033360 auth_method 0 2023-09-30T01:38:10.595 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.571+0000 7f1e88bc2700 1 --2- 172.21.15.88:0/4209953737 >> v2:172.21.15.107:6812/29926 conn(0x7f1e70033360 0x7f1e70035810 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f1e78000c00 tx=0x7f1e78007f30 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:38:10.595 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.579+0000 7f1e80ff9700 1 -- 172.21.15.88:0/4209953737 <== mon.8 v2:172.21.15.88:3304/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f1e7c014900 con 0x7f1e84069160 2023-09-30T01:38:10.595 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.579+0000 7f1e80ff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:38:10.595 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.579+0000 7f1e80ff9700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:38:10.749 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.747+0000 7f1e8ae26700 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2023-09-30T01:38:10.749 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.747+0000 7f1e8ae26700 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2023-09-30T01:38:10.749 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.747+0000 7f1e8ae26700 10 monclient: _send_mon_message to mon.e at v2:172.21.15.88:3304/0 2023-09-30T01:38:10.750 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.747+0000 7f1e8ae26700 1 -- 172.21.15.88:0/4209953737 --> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1e840710d0 con 0x7f1e84069160 2023-09-30T01:38:10.750 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.747+0000 7f1e80ff9700 1 -- 172.21.15.88:0/4209953737 <== mon.8 v2:172.21.15.88:3304/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v15) v1 ==== 74+0+9976 (secure 0 0 0) 0x7f1e7c018070 con 0x7f1e84069160 2023-09-30T01:38:10.750 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.747+0000 7f1e80ff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2023-09-30T01:38:10.751 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.747+0000 7f1e80ff9700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:38:10.753 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-09-30T01:38:10.753 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":15,"fsid":"650badb5-899b-4472-88a6-cfdccee24d52","created":"2023-09-30T01:32:51.511154+0000","modified":"2023-09-30T01:33:04.280780+0000","last_up_change":"2023-09-30T01:32:59.732301+0000","last_in_change":"2023-09-30T01:32:53.067458+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":"rbd","create_time":"2023-09-30T01:33:01.387368+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":".mgr","create_time":"2023-09-30T01:33:01.712290+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"1243f825-b99c-4546-a77d-90b8637698b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6808","nonce":30177}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6809","nonce":30177}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6811","nonce":30177}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6810","nonce":30177}]},"public_addr":"172.21.15.88:6808/30177","cluster_addr":"172.21.15.88:6809/30177","heartbeat_back_addr":"172.21.15.88:6811/30177","heartbeat_front_addr":"172.21.15.88:6810/30177","state":["exists","up"]},{"osd":1,"uuid":"e9831b75-399b-4d6d-bf97-9ef104adfdb5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6804","nonce":30175}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6805","nonce":30175}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6807","nonce":30175}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6806","nonce":30175}]},"public_addr":"172.21.15.88:6804/30175","cluster_addr":"172.21.15.88:6805/30175","heartbeat_back_addr":"172.21.15.88:6807/30175","heartbeat_front_addr":"172.21.15.88:6806/30175","state":["exists","up"]},{"osd":2,"uuid":"fbc7be74-cd43-4c72-89c1-469c17055b85","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6800","nonce":30176}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6801","nonce":30176}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6803","nonce":30176}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6802","nonce":30176}]},"public_addr":"172.21.15.88:6800/30176","cluster_addr":"172.21.15.88:6801/30176","heartbeat_back_addr":"172.21.15.88:6803/30176","heartbeat_front_addr":"172.21.15.88:6802/30176","state":["exists","up"]},{"osd":3,"uuid":"ca888e00-8f75-4dc4-b70c-6cd46880fe74","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":30274}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6805","nonce":30274}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6807","nonce":30274}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":30274}]},"public_addr":"172.21.15.107:6804/30274","cluster_addr":"172.21.15.107:6805/30274","heartbeat_back_addr":"172.21.15.107:6807/30274","heartbeat_front_addr":"172.21.15.107:6806/30274","state":["exists","up"]},{"osd":4,"uuid":"c8c73bd1-7674-4529-9104-8690e25af2fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":30276}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6809","nonce":30276}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6811","nonce":30276}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":30276}]},"public_addr":"172.21.15.107:6808/30276","cluster_addr":"172.21.15.107:6809/30276","heartbeat_back_addr":"172.21.15.107:6811/30276","heartbeat_front_addr":"172.21.15.107:6810/30276","state":["exists","up"]},{"osd":5,"uuid":"a3994780-9d6a-4bcd-b718-09f9d8774b3d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":30275}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6801","nonce":30275}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6803","nonce":30275}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":30275}]},"public_addr":"172.21.15.107:6800/30275","cluster_addr":"172.21.15.107:6801/30275","heartbeat_back_addr":"172.21.15.107:6803/30275","heartbeat_front_addr":"172.21.15.107:6802/30275","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T01:32:58.527368+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T01:32:58.431507+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T01:32:58.347215+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T01:32:58.501409+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T01:32:58.565697+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T01:32:58.313349+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-09-30T01:38:10.754 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.751+0000 7f1e8ae26700 1 -- 172.21.15.88:0/4209953737 >> v2:172.21.15.107:6812/29926 conn(0x7f1e70033360 msgr2=0x7f1e70035810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:38:10.754 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.751+0000 7f1e8ae26700 1 --2- 172.21.15.88:0/4209953737 >> v2:172.21.15.107:6812/29926 conn(0x7f1e70033360 0x7f1e70035810 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f1e78000c00 tx=0x7f1e78007f30 comp rx=0 tx=0).stop 2023-09-30T01:38:10.755 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.751+0000 7f1e8ae26700 10 monclient: shutdown 2023-09-30T01:38:10.755 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.751+0000 7f1e8ae26700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:38:10.755 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.751+0000 7f1e8ae26700 1 -- 172.21.15.88:0/4209953737 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f1e84069160 msgr2=0x7f1e8411ad30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:38:10.755 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.751+0000 7f1e8ae26700 1 --2- 172.21.15.88:0/4209953737 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f1e84069160 0x7f1e8411ad30 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f1e7c009f20 tx=0x7f1e7c00b040 comp rx=0 tx=0).stop 2023-09-30T01:38:10.755 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.751+0000 7f1e8ae26700 1 -- 172.21.15.88:0/4209953737 shutdown_connections 2023-09-30T01:38:10.755 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.751+0000 7f1e8ae26700 1 --2- 172.21.15.88:0/4209953737 >> v2:172.21.15.107:6812/29926 conn(0x7f1e70033360 0x7f1e70035810 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:38:10.755 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.751+0000 7f1e8ae26700 1 --2- 172.21.15.88:0/4209953737 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f1e84069160 0x7f1e8411ad30 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:38:10.755 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.751+0000 7f1e8ae26700 1 --2- 172.21.15.88:0/4209953737 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f1e84068d90 0x7f1e8412fd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:38:10.755 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.751+0000 7f1e8ae26700 1 --2- 172.21.15.88:0/4209953737 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1e840689c0 0x7f1e84143040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:38:10.756 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.751+0000 7f1e8ae26700 1 -- 172.21.15.88:0/4209953737 >> 172.21.15.88:0/4209953737 conn(0x7f1e84063710 msgr2=0x7f1e841322c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:38:10.756 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.751+0000 7f1e8ae26700 1 -- 172.21.15.88:0/4209953737 shutdown_connections 2023-09-30T01:38:10.756 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:10.751+0000 7f1e8ae26700 1 -- 172.21.15.88:0/4209953737 wait complete. 2023-09-30T01:38:11.766 INFO:tasks.ceph:Scrubbing osd.0 2023-09-30T01:38:11.766 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2023-09-30T01:38:11.887 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:11.883+0000 7f279ffff700 10 monclient: get_monmap_and_config 2023-09-30T01:38:11.887 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:11.883+0000 7f279ffff700 10 monclient: build_initial_monmap 2023-09-30T01:38:11.887 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:11.883+0000 7f279ffff700 10 monclient: monmap: 2023-09-30T01:38:11.887 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2023-09-30T01:38:11.887 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:38:11.887 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:38:11.886920+0000 2023-09-30T01:38:11.887 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:38:11.886920+0000 2023-09-30T01:38:11.888 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2023-09-30T01:38:11.888 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:38:11.888 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a-legacy 2023-09-30T01:38:11.888 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.107:6789/0 mon.noname-f-legacy 2023-09-30T01:38:11.888 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.noname-b 2023-09-30T01:38:11.888 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.noname-g 2023-09-30T01:38:11.888 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.noname-c 2023-09-30T01:38:11.888 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.noname-h 2023-09-30T01:38:11.888 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.noname-d 2023-09-30T01:38:11.888 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.noname-i 2023-09-30T01:38:11.889 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.noname-e 2023-09-30T01:38:11.889 INFO:teuthology.orchestra.run.smithi088.stderr:9: v2:172.21.15.88:3300/0 mon.noname-a 2023-09-30T01:38:11.889 INFO:teuthology.orchestra.run.smithi088.stderr:10: v2:172.21.15.107:3300/0 mon.noname-f 2023-09-30T01:38:11.889 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:38:11.892 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:11.887+0000 7f279ffff700 1 Processor -- start 2023-09-30T01:38:11.893 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:11.887+0000 7f279ffff700 1 -- start start 2023-09-30T01:38:11.893 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:11.887+0000 7f279ffff700 10 monclient: init 2023-09-30T01:38:11.893 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:11.887+0000 7f279ffff700 10 monclient: _reopen_session rank -1 2023-09-30T01:38:11.893 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:11.887+0000 7f279ffff700 10 monclient: _add_conns ranks=[10,2,8,5,6,4,3,7,0,1,9] 2023-09-30T01:38:11.893 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:11.887+0000 7f279ffff700 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f27a006e070 0x7f27a0066f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:38:11.893 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:11.887+0000 7f279ffff700 10 monclient(hunting): picked mon.noname-f con 0x7f27a006e070 addr v2:172.21.15.107:3300/0 2023-09-30T01:38:11.893 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:11.887+0000 7f279ffff700 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f27a009be50 0x7f27a009c2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:38:11.893 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:11.887+0000 7f279ffff700 10 monclient(hunting): picked mon.noname-b con 0x7f27a009be50 addr [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] 2023-09-30T01:38:11.893 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:11.887+0000 7f279ffff700 1 --2- >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f27a009c7e0 0x7f27a0133470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:38:11.893 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:11.887+0000 7f279ffff700 10 monclient(hunting): picked mon.noname-e con 0x7f27a009c7e0 addr [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] 2023-09-30T01:38:11.893 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:11.887+0000 7f279ffff700 10 monclient(hunting): start opening mon connection 2023-09-30T01:38:11.894 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:11.887+0000 7f279ffff700 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f27a009e930 con 0x7f27a006e070 2023-09-30T01:38:11.894 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:11.887+0000 7f279ffff700 10 monclient(hunting): start opening mon connection 2023-09-30T01:38:11.894 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:11.887+0000 7f279ffff700 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27a009d910 con 0x7f27a009be50 2023-09-30T01:38:11.894 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:11.887+0000 7f279ffff700 10 monclient(hunting): start opening mon connection 2023-09-30T01:38:11.894 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:11.887+0000 7f279ffff700 1 -- --> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] -- mon_getmap magic: 0 v1 -- 0x7f27a006a6b0 con 0x7f27a009c7e0 2023-09-30T01:38:11.894 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:11.887+0000 7f279ffff700 10 monclient(hunting): _renew_subs 2023-09-30T01:38:11.894 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:11.887+0000 7f279ffff700 10 monclient(hunting): authenticate will time out at 1163.925147s 2023-09-30T01:38:11.894 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:11.887+0000 7f2797fff700 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f27a009be50 0x7f27a009c2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:38:11.894 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:11.887+0000 7f279f7fe700 1 --2- >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f27a009c7e0 0x7f27a0133470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:38:11.894 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:11.887+0000 7f2797fff700 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f27a009be50 0x7f27a009c2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3301/0 says I am v2:172.21.15.88:53482/0 (socket says 172.21.15.88:53482) 2023-09-30T01:38:11.895 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:11.887+0000 7f2797fff700 1 -- 172.21.15.88:0/1270854075 learned_addr learned my addr 172.21.15.88:0/1270854075 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-09-30T01:38:11.895 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:11.887+0000 7f2797fff700 10 monclient(hunting): get_auth_request con 0x7f27a009be50 auth_method 0 2023-09-30T01:38:11.895 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:11.887+0000 7f2797fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:38:11.895 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:11.887+0000 7f2797fff700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:38:11.895 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:11.887+0000 7f2797fff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:38:11.895 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:11.887+0000 7f279f7fe700 10 monclient(hunting): get_auth_request con 0x7f27a009c7e0 auth_method 0 2023-09-30T01:38:11.895 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:11.887+0000 7f279f7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:38:11.895 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:11.887+0000 7f279f7fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:38:11.895 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:11.887+0000 7f279f7fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:38:11.895 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:11.887+0000 7f279f7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:38:11.895 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:11.887+0000 7f279f7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:38:11.896 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:11.887+0000 7f279effd700 1 --2- 172.21.15.88:0/1270854075 >> v2:172.21.15.107:3300/0 conn(0x7f27a006e070 0x7f27a0066f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:38:11.896 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:11.887+0000 7f279f7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:38:11.896 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:11.887+0000 7f2797fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:38:11.896 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:11.887+0000 7f2797fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:38:11.896 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:11.887+0000 7f2797fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:38:11.896 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:11.887+0000 7f279effd700 10 monclient(hunting): get_auth_request con 0x7f27a006e070 auth_method 0 2023-09-30T01:38:11.896 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:11.887+0000 7f279effd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:38:11.896 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:11.887+0000 7f279effd700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:38:11.896 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:11.887+0000 7f279effd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:38:11.896 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:11.887+0000 7f279f7fe700 10 monclient(hunting): handle_auth_done global_id 4913 payload 274 2023-09-30T01:38:11.896 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:11.887+0000 7f279f7fe700 1 -- 172.21.15.88:0/1270854075 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f27a009be50 msgr2=0x7f27a009c2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:38:11.896 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:11.887+0000 7f279f7fe700 1 --2- 172.21.15.88:0/1270854075 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f27a009be50 0x7f27a009c2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:38:11.896 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:11.887+0000 7f279f7fe700 1 -- 172.21.15.88:0/1270854075 >> v2:172.21.15.107:3300/0 conn(0x7f27a006e070 msgr2=0x7f27a0066f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:38:11.897 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:11.887+0000 7f279f7fe700 1 --2- 172.21.15.88:0/1270854075 >> v2:172.21.15.107:3300/0 conn(0x7f27a006e070 0x7f27a0066f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:38:11.897 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:11.887+0000 7f279f7fe700 10 monclient: _finish_hunting 0 2023-09-30T01:38:11.897 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:11.887+0000 7f279f7fe700 1 monclient: found mon.noname-e 2023-09-30T01:38:11.897 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:11.887+0000 7f279f7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:38:11.897 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:11.887+0000 7f279f7fe700 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.88:3304/0 2023-09-30T01:38:11.897 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:11.887+0000 7f279f7fe700 1 -- 172.21.15.88:0/1270854075 --> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27a004bb20 con 0x7f27a009c7e0 2023-09-30T01:38:11.897 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:11.887+0000 7f2797fff700 1 --2- 172.21.15.88:0/1270854075 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f27a009be50 0x7f27a009c2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:38:11.897 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:11.887+0000 7f279effd700 1 --2- 172.21.15.88:0/1270854075 >> v2:172.21.15.107:3300/0 conn(0x7f27a006e070 0x7f27a0066f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-30T01:38:11.897 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:11.887+0000 7f279f7fe700 1 --2- 172.21.15.88:0/1270854075 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f27a009c7e0 0x7f27a0133470 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f2798009e60 tx=0x7f2798009780 comp rx=0 tx=0).ready entity=mon.8 client_cookie=cbffb8215327bfed server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:38:11.897 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:11.887+0000 7f279e7fc700 1 -- 172.21.15.88:0/1270854075 <== mon.8 v2:172.21.15.88:3304/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f279800e070 con 0x7f27a009c7e0 2023-09-30T01:38:11.897 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:11.887+0000 7f279e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:38:11.898 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:11.887+0000 7f279e7fc700 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2023-09-30T01:38:11.898 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:11.887+0000 7f279e7fc700 10 monclient: dump: 2023-09-30T01:38:11.898 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:38:11.898 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:38:11.898 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:38:11.898 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:38:11.898 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:38:11.898 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:38:11.898 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:38:11.898 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:38:11.898 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:38:11.898 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:38:11.899 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:38:11.899 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:38:11.899 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:38:11.899 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:38:11.899 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:38:11.899 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:38:11.899 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:11.887+0000 7f279e7fc700 10 monclient: _finish_auth 0 2023-09-30T01:38:11.899 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:11.887+0000 7f279e7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:38:11.899 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:11.887+0000 7f279e7fc700 1 -- 172.21.15.88:0/1270854075 <== mon.8 v2:172.21.15.88:3304/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2798012430 con 0x7f27a009c7e0 2023-09-30T01:38:11.899 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:11.887+0000 7f279ffff700 5 monclient: authenticate success, global_id 4913 2023-09-30T01:38:11.899 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:11.887+0000 7f279ffff700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:38:11.899 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:11.887+0000 7f279e7fc700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:38:11.899 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:11.887+0000 7f279ffff700 10 monclient: get_monmap_and_config success 2023-09-30T01:38:11.900 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:11.887+0000 7f279ffff700 4 set_mon_vals no callback set 2023-09-30T01:38:11.900 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:11.887+0000 7f279e7fc700 1 -- 172.21.15.88:0/1270854075 <== mon.8 v2:172.21.15.88:3304/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f27980125a0 con 0x7f27a009c7e0 2023-09-30T01:38:11.900 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:11.887+0000 7f279e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:38:11.900 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:11.887+0000 7f279e7fc700 10 monclient: got monmap 1 from mon.e (according to old e1) 2023-09-30T01:38:11.900 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:11.887+0000 7f279e7fc700 10 monclient: dump: 2023-09-30T01:38:11.900 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:38:11.900 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:38:11.900 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:38:11.900 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:38:11.900 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:38:11.900 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:38:11.900 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:38:11.901 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:38:11.901 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:38:11.901 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:38:11.901 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:38:11.901 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:38:11.901 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:38:11.901 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:38:11.901 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:38:11.901 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:38:11.901 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:11.887+0000 7f279ffff700 10 monclient: shutdown 2023-09-30T01:38:11.901 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:11.887+0000 7f279ffff700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:38:11.901 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:11.887+0000 7f279ffff700 1 -- 172.21.15.88:0/1270854075 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f27a009c7e0 msgr2=0x7f27a0133470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:38:11.901 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:11.887+0000 7f279ffff700 1 --2- 172.21.15.88:0/1270854075 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f27a009c7e0 0x7f27a0133470 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f2798009e60 tx=0x7f2798009780 comp rx=0 tx=0).stop 2023-09-30T01:38:11.902 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:11.891+0000 7f279ffff700 1 -- 172.21.15.88:0/1270854075 shutdown_connections 2023-09-30T01:38:11.902 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:11.891+0000 7f279ffff700 1 --2- 172.21.15.88:0/1270854075 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f27a009c7e0 0x7f27a0133470 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:38:11.902 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:11.891+0000 7f279ffff700 1 --2- 172.21.15.88:0/1270854075 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f27a009be50 0x7f27a009c2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:38:11.902 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:11.891+0000 7f279ffff700 1 --2- 172.21.15.88:0/1270854075 >> v2:172.21.15.107:3300/0 conn(0x7f27a006e070 0x7f27a0066f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:38:11.902 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:11.891+0000 7f279ffff700 1 -- 172.21.15.88:0/1270854075 >> 172.21.15.88:0/1270854075 conn(0x7f27a00635e0 msgr2=0x7f27a00a2b90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:38:11.902 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:11.891+0000 7f279ffff700 1 -- 172.21.15.88:0/1270854075 shutdown_connections 2023-09-30T01:38:11.902 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:11.891+0000 7f279ffff700 1 -- 172.21.15.88:0/1270854075 wait complete. 2023-09-30T01:38:11.902 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:11.891+0000 7f279ffff700 10 monclient: build_initial_monmap 2023-09-30T01:38:11.902 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:11.891+0000 7f279ffff700 10 monclient: monmap: 2023-09-30T01:38:11.902 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2023-09-30T01:38:11.902 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:38:11.903 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 0.000000 2023-09-30T01:38:11.903 INFO:teuthology.orchestra.run.smithi088.stderr:created 0.000000 2023-09-30T01:38:11.903 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2023-09-30T01:38:11.903 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 1 2023-09-30T01:38:11.903 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.noname-a 2023-09-30T01:38:11.903 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.noname-f 2023-09-30T01:38:11.903 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.noname-b 2023-09-30T01:38:11.903 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.noname-g 2023-09-30T01:38:11.903 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.noname-c 2023-09-30T01:38:11.903 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.noname-h 2023-09-30T01:38:11.903 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.noname-d 2023-09-30T01:38:11.903 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.noname-i 2023-09-30T01:38:11.904 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.noname-e 2023-09-30T01:38:11.904 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:38:11.908 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:11.891+0000 7f279ffff700 1 Processor -- start 2023-09-30T01:38:11.908 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:11.891+0000 7f279ffff700 1 -- start start 2023-09-30T01:38:11.908 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:11.891+0000 7f279ffff700 10 monclient: init 2023-09-30T01:38:11.909 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:11.891+0000 7f279ffff700 10 monclient: _reopen_session rank -1 2023-09-30T01:38:11.909 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:11.891+0000 7f279ffff700 10 monclient: _add_conns ranks=[0,2,4,5,6,8,7,1,3] 2023-09-30T01:38:11.909 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:11.891+0000 7f279ffff700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f27a009be50 0x7f27a0142700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:38:11.909 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:11.891+0000 7f279ffff700 10 monclient(hunting): picked mon.noname-a con 0x7f27a009be50 addr [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] 2023-09-30T01:38:11.909 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:11.891+0000 7f279ffff700 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f27a009c7e0 0x7f27a0142c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:38:11.909 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:11.891+0000 7f279ffff700 10 monclient(hunting): picked mon.noname-b con 0x7f27a009c7e0 addr [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] 2023-09-30T01:38:11.909 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:11.891+0000 7f279ffff700 1 --2- >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f27a0143180 0x7f27a011fba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:38:11.909 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:11.891+0000 7f279ffff700 10 monclient(hunting): picked mon.noname-c con 0x7f27a0143180 addr [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] 2023-09-30T01:38:11.909 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:11.891+0000 7f279ffff700 10 monclient(hunting): start opening mon connection 2023-09-30T01:38:11.909 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:11.891+0000 7f279ffff700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27a01435d0 con 0x7f27a009be50 2023-09-30T01:38:11.909 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:11.891+0000 7f279ffff700 10 monclient(hunting): start opening mon connection 2023-09-30T01:38:11.909 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:11.891+0000 7f279ffff700 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27a0121d20 con 0x7f27a009c7e0 2023-09-30T01:38:11.910 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:11.891+0000 7f279ffff700 10 monclient(hunting): start opening mon connection 2023-09-30T01:38:11.910 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:11.891+0000 7f279ffff700 1 -- --> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] -- mon_getmap magic: 0 v1 -- 0x7f27a009eaa0 con 0x7f27a0143180 2023-09-30T01:38:11.910 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:11.891+0000 7f279ffff700 10 monclient(hunting): _renew_subs 2023-09-30T01:38:11.910 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:11.891+0000 7f279ffff700 10 monclient(hunting): authenticate will time out at 1163.930718s 2023-09-30T01:38:11.910 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:11.891+0000 7f2797fff700 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f27a009c7e0 0x7f27a0142c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:38:11.910 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:11.891+0000 7f279f7fe700 1 --2- >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f27a0143180 0x7f27a011fba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:38:11.910 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:11.891+0000 7f279effd700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f27a009be50 0x7f27a0142700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:38:11.910 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:11.891+0000 7f2797fff700 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f27a009c7e0 0x7f27a0142c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3301/0 says I am v2:172.21.15.88:53494/0 (socket says 172.21.15.88:53494) 2023-09-30T01:38:11.910 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:11.891+0000 7f2797fff700 1 -- 172.21.15.88:0/4214172629 learned_addr learned my addr 172.21.15.88:0/4214172629 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-09-30T01:38:11.910 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:11.891+0000 7f2797fff700 10 monclient(hunting): get_auth_request con 0x7f27a009c7e0 auth_method 0 2023-09-30T01:38:11.910 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:11.891+0000 7f2797fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:38:11.910 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:11.891+0000 7f2797fff700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:38:11.911 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:11.891+0000 7f2797fff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:38:11.911 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:11.891+0000 7f279effd700 10 monclient(hunting): get_auth_request con 0x7f27a009be50 auth_method 0 2023-09-30T01:38:11.911 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:11.891+0000 7f279effd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:38:11.911 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:11.891+0000 7f279effd700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:38:11.911 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:11.891+0000 7f279effd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:38:11.911 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:11.891+0000 7f279f7fe700 10 monclient(hunting): get_auth_request con 0x7f27a0143180 auth_method 0 2023-09-30T01:38:11.911 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:11.891+0000 7f279f7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:38:11.911 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:11.891+0000 7f279f7fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:38:11.911 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:11.891+0000 7f279f7fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:38:11.911 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:11.891+0000 7f2797fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:38:11.911 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:11.891+0000 7f2797fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:38:11.911 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:11.891+0000 7f2797fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:38:11.912 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:11.891+0000 7f279effd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:38:11.912 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:11.891+0000 7f279effd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:38:11.912 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:11.891+0000 7f279effd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:38:11.912 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:11.891+0000 7f279f7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:38:11.912 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:11.891+0000 7f279f7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:38:11.912 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:11.891+0000 7f279f7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:38:11.912 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:11.891+0000 7f279effd700 10 monclient(hunting): handle_auth_done global_id 5202 payload 824 2023-09-30T01:38:11.912 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:11.891+0000 7f279effd700 1 -- 172.21.15.88:0/4214172629 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f27a0143180 msgr2=0x7f27a011fba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:38:11.912 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:11.891+0000 7f279effd700 1 --2- 172.21.15.88:0/4214172629 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f27a0143180 0x7f27a011fba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:38:11.912 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:11.891+0000 7f279effd700 1 -- 172.21.15.88:0/4214172629 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f27a009c7e0 msgr2=0x7f27a0142c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:38:11.912 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:11.891+0000 7f279effd700 1 --2- 172.21.15.88:0/4214172629 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f27a009c7e0 0x7f27a0142c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:38:11.912 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:11.891+0000 7f279effd700 10 monclient: _finish_hunting 0 2023-09-30T01:38:11.912 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:11.891+0000 7f279effd700 1 monclient: found mon.noname-a 2023-09-30T01:38:11.913 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:11.891+0000 7f279effd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:38:11.913 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:11.891+0000 7f279effd700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.88:3300/0 2023-09-30T01:38:11.913 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:11.891+0000 7f279effd700 1 -- 172.21.15.88:0/4214172629 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27880094f0 con 0x7f27a009be50 2023-09-30T01:38:11.913 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:11.891+0000 7f2797fff700 1 --2- 172.21.15.88:0/4214172629 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f27a009c7e0 0x7f27a0142c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:38:11.913 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:11.891+0000 7f279effd700 1 --2- 172.21.15.88:0/4214172629 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f27a009be50 0x7f27a0142700 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f2790006fd0 tx=0x7f2790008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:38:11.913 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:11.895+0000 7f2796ffd700 1 -- 172.21.15.88:0/4214172629 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f2790013890 con 0x7f27a009be50 2023-09-30T01:38:11.913 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:11.895+0000 7f2796ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:38:11.913 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:11.895+0000 7f2796ffd700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T01:38:11.913 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:11.895+0000 7f2796ffd700 10 monclient: dump: 2023-09-30T01:38:11.913 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:38:11.913 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:38:11.913 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:38:11.914 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:38:11.914 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:38:11.914 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:38:11.914 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:38:11.914 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:38:11.914 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:38:11.914 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:38:11.914 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:38:11.914 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:38:11.914 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:38:11.914 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:38:11.914 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:38:11.915 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:38:11.915 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:11.895+0000 7f2796ffd700 10 monclient: _finish_auth 0 2023-09-30T01:38:11.915 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:11.895+0000 7f2796ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:38:11.915 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:11.895+0000 7f279ffff700 5 monclient: authenticate success, global_id 5202 2023-09-30T01:38:11.915 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:11.895+0000 7f2796ffd700 1 -- 172.21.15.88:0/4214172629 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2790013a30 con 0x7f27a009be50 2023-09-30T01:38:11.915 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:11.895+0000 7f279ffff700 10 monclient: _renew_subs 2023-09-30T01:38:11.915 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:11.895+0000 7f279ffff700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.88:3300/0 2023-09-30T01:38:11.915 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:11.895+0000 7f279ffff700 1 -- 172.21.15.88:0/4214172629 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2798009a20 con 0x7f27a009be50 2023-09-30T01:38:11.915 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:11.895+0000 7f2796ffd700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:38:11.915 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:11.895+0000 7f2796ffd700 1 -- 172.21.15.88:0/4214172629 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f2790013ba0 con 0x7f27a009be50 2023-09-30T01:38:11.915 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:11.895+0000 7f2796ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:38:11.915 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:11.895+0000 7f2796ffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:38:11.916 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:11.895+0000 7f279d7fa700 4 set_mon_vals no callback set 2023-09-30T01:38:11.916 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:11.895+0000 7f2796ffd700 10 monclient: dump: 2023-09-30T01:38:11.916 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:38:11.916 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:38:11.916 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:38:11.916 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:38:11.916 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:38:11.916 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:38:11.916 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:38:11.916 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:38:11.916 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:38:11.917 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:38:11.917 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:38:11.917 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:38:11.917 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:38:11.917 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:38:11.917 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:38:11.917 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:38:11.917 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:11.895+0000 7f279ffff700 10 monclient: _renew_subs 2023-09-30T01:38:11.917 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:11.895+0000 7f279ffff700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.88:3300/0 2023-09-30T01:38:11.917 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:11.895+0000 7f279ffff700 1 -- 172.21.15.88:0/4214172629 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f27a0122370 con 0x7f27a009be50 2023-09-30T01:38:11.917 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:11.895+0000 7f2796ffd700 1 -- 172.21.15.88:0/4214172629 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 40144+0+0 (secure 0 0 0) 0x7f2790013d00 con 0x7f27a009be50 2023-09-30T01:38:11.917 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:11.895+0000 7f279ffff700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.88:3300/0 2023-09-30T01:38:11.918 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:11.895+0000 7f279ffff700 1 -- 172.21.15.88:0/4214172629 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f27a0071960 con 0x7f27a009be50 2023-09-30T01:38:11.918 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:11.895+0000 7f2796ffd700 1 --2- 172.21.15.88:0/4214172629 >> v2:172.21.15.107:6812/29926 conn(0x7f2780033390 0x7f2780035840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:38:11.918 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:11.895+0000 7f2796ffd700 1 -- 172.21.15.88:0/4214172629 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 3502+0+0 (secure 0 0 0) 0x7f27900432e0 con 0x7f27a009be50 2023-09-30T01:38:11.918 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:11.895+0000 7f2796ffd700 1 --2- 172.21.15.88:0/4214172629 >> v2:172.21.15.88:6808/30177 conn(0x7f2780038e30 0x7f278003b240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:38:11.918 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:11.895+0000 7f2796ffd700 1 -- 172.21.15.88:0/4214172629 --> v2:172.21.15.88:6808/30177 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f278003b8b0 con 0x7f2780038e30 2023-09-30T01:38:11.918 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:11.895+0000 7f2796ffd700 1 -- 172.21.15.88:0/4214172629 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_get_version_reply(handle=1 version=15) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2790013410 con 0x7f27a009be50 2023-09-30T01:38:11.918 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:11.895+0000 7f2796ffd700 10 monclient: handle_get_version_reply finishing 1 version 15 2023-09-30T01:38:11.918 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:11.895+0000 7f2797fff700 1 --2- 172.21.15.88:0/4214172629 >> v2:172.21.15.107:6812/29926 conn(0x7f2780033390 0x7f2780035840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:38:11.918 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:11.895+0000 7f2797fff700 10 monclient: get_auth_request con 0x7f2780033390 auth_method 0 2023-09-30T01:38:11.918 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:11.895+0000 7f279f7fe700 1 --2- 172.21.15.88:0/4214172629 >> v2:172.21.15.88:6808/30177 conn(0x7f2780038e30 0x7f278003b240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:38:11.918 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:11.895+0000 7f279f7fe700 10 monclient: get_auth_request con 0x7f2780038e30 auth_method 0 2023-09-30T01:38:11.918 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:11.895+0000 7f279f7fe700 1 --2- 172.21.15.88:0/4214172629 >> v2:172.21.15.88:6808/30177 conn(0x7f2780038e30 0x7f278003b240 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:38:11.919 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:11.895+0000 7f2797fff700 1 --2- 172.21.15.88:0/4214172629 >> v2:172.21.15.107:6812/29926 conn(0x7f2780033390 0x7f2780035840 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f27a011faf0 tx=0x7f2788009250 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:38:11.919 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:11.895+0000 7f2796ffd700 1 -- 172.21.15.88:0/4214172629 <== osd.0 v2:172.21.15.88:6808/30177 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23999 (crc 0 0 0) 0x7f278003b8b0 con 0x7f2780038e30 2023-09-30T01:38:11.919 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:11.915+0000 7f279ffff700 1 -- 172.21.15.88:0/4214172629 --> v2:172.21.15.88:6808/30177 -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f27a01245d0 con 0x7f2780038e30 2023-09-30T01:38:11.919 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:11.915+0000 7f2796ffd700 1 -- 172.21.15.88:0/4214172629 <== osd.0 v2:172.21.15.88:6808/30177 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7f27a01245d0 con 0x7f2780038e30 2023-09-30T01:38:11.920 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-09-30T01:38:11.920 INFO:teuthology.orchestra.run.smithi088.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-09-30T01:38:11.920 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-09-30T01:38:11.921 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:11.915+0000 7f279ffff700 1 -- 172.21.15.88:0/4214172629 >> v2:172.21.15.88:6808/30177 conn(0x7f2780038e30 msgr2=0x7f278003b240 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:38:11.921 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:11.915+0000 7f279ffff700 1 --2- 172.21.15.88:0/4214172629 >> v2:172.21.15.88:6808/30177 conn(0x7f2780038e30 0x7f278003b240 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:38:11.921 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:11.915+0000 7f279ffff700 1 -- 172.21.15.88:0/4214172629 >> v2:172.21.15.107:6812/29926 conn(0x7f2780033390 msgr2=0x7f2780035840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:38:11.921 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:11.915+0000 7f279ffff700 1 --2- 172.21.15.88:0/4214172629 >> v2:172.21.15.107:6812/29926 conn(0x7f2780033390 0x7f2780035840 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f27a011faf0 tx=0x7f2788009250 comp rx=0 tx=0).stop 2023-09-30T01:38:11.921 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:11.915+0000 7f279ffff700 10 monclient: shutdown 2023-09-30T01:38:11.921 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:11.915+0000 7f279ffff700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:38:11.921 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:11.915+0000 7f279ffff700 1 -- 172.21.15.88:0/4214172629 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f27a009be50 msgr2=0x7f27a0142700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:38:11.921 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:11.915+0000 7f279ffff700 1 --2- 172.21.15.88:0/4214172629 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f27a009be50 0x7f27a0142700 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f2790006fd0 tx=0x7f2790008040 comp rx=0 tx=0).stop 2023-09-30T01:38:11.921 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:11.915+0000 7f279f7fe700 1 -- 172.21.15.88:0/4214172629 reap_dead start 2023-09-30T01:38:11.922 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:11.915+0000 7f279ffff700 1 -- 172.21.15.88:0/4214172629 shutdown_connections 2023-09-30T01:38:11.922 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:11.915+0000 7f279ffff700 1 -- 172.21.15.88:0/4214172629 >> 172.21.15.88:0/4214172629 conn(0x7f27a00635e0 msgr2=0x7f27a00a21b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:38:11.922 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:11.915+0000 7f279ffff700 1 -- 172.21.15.88:0/4214172629 shutdown_connections 2023-09-30T01:38:11.922 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:11.915+0000 7f279ffff700 1 -- 172.21.15.88:0/4214172629 wait complete. 2023-09-30T01:38:11.931 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2023-09-30T01:38:12.045 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.043+0000 7f2e82f10700 10 monclient: get_monmap_and_config 2023-09-30T01:38:12.045 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.043+0000 7f2e82f10700 10 monclient: build_initial_monmap 2023-09-30T01:38:12.045 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.043+0000 7f2e82f10700 10 monclient: monmap: 2023-09-30T01:38:12.046 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2023-09-30T01:38:12.046 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:38:12.046 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:38:12.044821+0000 2023-09-30T01:38:12.046 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:38:12.044821+0000 2023-09-30T01:38:12.046 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2023-09-30T01:38:12.046 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:38:12.046 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a-legacy 2023-09-30T01:38:12.046 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.107:6789/0 mon.noname-f-legacy 2023-09-30T01:38:12.046 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.noname-b 2023-09-30T01:38:12.046 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.noname-g 2023-09-30T01:38:12.046 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.noname-c 2023-09-30T01:38:12.047 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.noname-h 2023-09-30T01:38:12.047 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.noname-d 2023-09-30T01:38:12.047 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.noname-i 2023-09-30T01:38:12.047 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.noname-e 2023-09-30T01:38:12.047 INFO:teuthology.orchestra.run.smithi088.stderr:9: v2:172.21.15.88:3300/0 mon.noname-a 2023-09-30T01:38:12.047 INFO:teuthology.orchestra.run.smithi088.stderr:10: v2:172.21.15.107:3300/0 mon.noname-f 2023-09-30T01:38:12.047 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:38:12.050 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.043+0000 7f2e82f10700 1 Processor -- start 2023-09-30T01:38:12.050 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.043+0000 7f2e82f10700 1 -- start start 2023-09-30T01:38:12.050 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.043+0000 7f2e82f10700 10 monclient: init 2023-09-30T01:38:12.050 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.043+0000 7f2e82f10700 10 monclient: _reopen_session rank -1 2023-09-30T01:38:12.050 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.043+0000 7f2e82f10700 10 monclient: _add_conns ranks=[0,1,9,2,7,4,3,8,10,6,5] 2023-09-30T01:38:12.050 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.043+0000 7f2e82f10700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f2e7c097a40 addr v1:172.21.15.88:6789/0 2023-09-30T01:38:12.050 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.043+0000 7f2e82f10700 10 monclient(hunting): picked mon.noname-f-legacy con 0x7f2e7c132e10 addr v1:172.21.15.107:6789/0 2023-09-30T01:38:12.051 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.043+0000 7f2e82f10700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f2e7c1311c0 0x7f2e7c13bd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:38:12.051 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.043+0000 7f2e82f10700 10 monclient(hunting): picked mon.noname-a con 0x7f2e7c1311c0 addr v2:172.21.15.88:3300/0 2023-09-30T01:38:12.051 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.043+0000 7f2e82f10700 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e7c04bb20 con 0x7f2e7c097a40 2023-09-30T01:38:12.051 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.043+0000 7f2e82f10700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e7c09a5d0 con 0x7f2e7c132e10 2023-09-30T01:38:12.051 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.043+0000 7f2e82f10700 10 monclient(hunting): start opening mon connection 2023-09-30T01:38:12.051 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.043+0000 7f2e82f10700 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2e7c097e10 con 0x7f2e7c1311c0 2023-09-30T01:38:12.051 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.043+0000 7f2e82f10700 10 monclient(hunting): _renew_subs 2023-09-30T01:38:12.051 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.043+0000 7f2e82f10700 10 monclient(hunting): authenticate will time out at 1164.082291s 2023-09-30T01:38:12.051 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.043+0000 7f2e8270f700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f2e7c1311c0 0x7f2e7c13bd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:38:12.051 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.043+0000 7f2e8270f700 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f2e7c1311c0 0x7f2e7c13bd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:48176/0 (socket says 172.21.15.88:48176) 2023-09-30T01:38:12.052 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.043+0000 7f2e8270f700 1 -- 172.21.15.88:0/1925374127 learned_addr learned my addr 172.21.15.88:0/1925374127 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-09-30T01:38:12.052 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.043+0000 7f2e8270f700 10 monclient(hunting): get_auth_request con 0x7f2e7c1311c0 auth_method 0 2023-09-30T01:38:12.052 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.043+0000 7f2e8270f700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:38:12.052 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.043+0000 7f2e8270f700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:38:12.052 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.043+0000 7f2e8270f700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:38:12.052 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.043+0000 7f2e8270f700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:38:12.052 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.043+0000 7f2e8270f700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:38:12.052 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.043+0000 7f2e8270f700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:38:12.052 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.043+0000 7f2e8270f700 10 monclient(hunting): handle_auth_done global_id 5211 payload 274 2023-09-30T01:38:12.053 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.043+0000 7f2e8270f700 1 -- 172.21.15.88:0/1925374127 >> v1:172.21.15.107:6789/0 conn(0x7f2e7c132e10 legacy=0x7f2e7c138650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:38:12.053 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.043+0000 7f2e8270f700 1 -- 172.21.15.88:0/1925374127 >> v1:172.21.15.88:6789/0 conn(0x7f2e7c097a40 legacy=0x7f2e7c132700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:38:12.053 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.043+0000 7f2e8270f700 10 monclient: _finish_hunting 0 2023-09-30T01:38:12.053 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.043+0000 7f2e8270f700 1 monclient: found mon.noname-a 2023-09-30T01:38:12.053 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.043+0000 7f2e8270f700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:38:12.053 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.043+0000 7f2e8270f700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.88:3300/0 2023-09-30T01:38:12.053 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.043+0000 7f2e8270f700 1 -- 172.21.15.88:0/1925374127 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e7c098d40 con 0x7f2e7c1311c0 2023-09-30T01:38:12.053 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.043+0000 7f2e8270f700 1 --2- 172.21.15.88:0/1925374127 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2e7c1311c0 0x7f2e7c13bd80 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f2e74009f20 tx=0x7f2e74009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=304a9911a1e4e864 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:38:12.053 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.043+0000 7f2e80f0c700 1 -- 172.21.15.88:0/1925374127 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f2e7400e070 con 0x7f2e7c1311c0 2023-09-30T01:38:12.054 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.043+0000 7f2e80f0c700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:38:12.054 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.043+0000 7f2e80f0c700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T01:38:12.054 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.043+0000 7f2e80f0c700 10 monclient: dump: 2023-09-30T01:38:12.054 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:38:12.054 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:38:12.054 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:38:12.054 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:38:12.054 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:38:12.054 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:38:12.054 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:38:12.054 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:38:12.055 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:38:12.055 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:38:12.055 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:38:12.055 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:38:12.055 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:38:12.055 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:38:12.055 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:38:12.055 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:38:12.055 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.043+0000 7f2e80f0c700 10 monclient: _finish_auth 0 2023-09-30T01:38:12.055 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.043+0000 7f2e80f0c700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:38:12.056 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.047+0000 7f2e80f0c700 1 -- 172.21.15.88:0/1925374127 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e74007400 con 0x7f2e7c1311c0 2023-09-30T01:38:12.056 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.047+0000 7f2e82f10700 5 monclient: authenticate success, global_id 5211 2023-09-30T01:38:12.056 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.047+0000 7f2e82f10700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:38:12.056 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.047+0000 7f2e80f0c700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:38:12.056 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.047+0000 7f2e82f10700 10 monclient: get_monmap_and_config success 2023-09-30T01:38:12.056 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.047+0000 7f2e80f0c700 1 -- 172.21.15.88:0/1925374127 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f2e740092e0 con 0x7f2e7c1311c0 2023-09-30T01:38:12.056 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.047+0000 7f2e82f10700 4 set_mon_vals no callback set 2023-09-30T01:38:12.056 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.047+0000 7f2e80f0c700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:38:12.056 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.047+0000 7f2e80f0c700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:38:12.056 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.047+0000 7f2e80f0c700 10 monclient: dump: 2023-09-30T01:38:12.056 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:38:12.057 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:38:12.057 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:38:12.057 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:38:12.057 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:38:12.057 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:38:12.057 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:38:12.057 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:38:12.057 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:38:12.057 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:38:12.057 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:38:12.058 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:38:12.058 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:38:12.058 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:38:12.058 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:38:12.058 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:38:12.058 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.047+0000 7f2e82f10700 10 monclient: shutdown 2023-09-30T01:38:12.058 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.047+0000 7f2e82f10700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:38:12.058 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.047+0000 7f2e82f10700 1 -- 172.21.15.88:0/1925374127 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2e7c1311c0 msgr2=0x7f2e7c13bd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:38:12.058 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.047+0000 7f2e82f10700 1 --2- 172.21.15.88:0/1925374127 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2e7c1311c0 0x7f2e7c13bd80 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f2e74009f20 tx=0x7f2e74009840 comp rx=0 tx=0).stop 2023-09-30T01:38:12.058 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.047+0000 7f2e82f10700 1 -- 172.21.15.88:0/1925374127 shutdown_connections 2023-09-30T01:38:12.059 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.047+0000 7f2e82f10700 1 --2- 172.21.15.88:0/1925374127 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2e7c1311c0 0x7f2e7c13bd80 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:38:12.059 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.047+0000 7f2e82f10700 1 -- 172.21.15.88:0/1925374127 >> 172.21.15.88:0/1925374127 conn(0x7f2e7c06e5e0 msgr2=0x7f2e7c06e9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:38:12.059 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.047+0000 7f2e82f10700 1 -- 172.21.15.88:0/1925374127 shutdown_connections 2023-09-30T01:38:12.059 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.047+0000 7f2e82f10700 1 -- 172.21.15.88:0/1925374127 wait complete. 2023-09-30T01:38:12.059 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.047+0000 7f2e82f10700 10 monclient: build_initial_monmap 2023-09-30T01:38:12.059 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.047+0000 7f2e82f10700 10 monclient: monmap: 2023-09-30T01:38:12.059 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2023-09-30T01:38:12.059 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:38:12.059 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 0.000000 2023-09-30T01:38:12.059 INFO:teuthology.orchestra.run.smithi088.stderr:created 0.000000 2023-09-30T01:38:12.059 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2023-09-30T01:38:12.060 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 1 2023-09-30T01:38:12.060 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.noname-a 2023-09-30T01:38:12.060 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.noname-f 2023-09-30T01:38:12.060 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.noname-b 2023-09-30T01:38:12.060 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.noname-g 2023-09-30T01:38:12.060 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.noname-c 2023-09-30T01:38:12.060 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.noname-h 2023-09-30T01:38:12.060 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.noname-d 2023-09-30T01:38:12.060 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.noname-i 2023-09-30T01:38:12.060 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.noname-e 2023-09-30T01:38:12.060 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:38:12.065 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.047+0000 7f2e82f10700 1 Processor -- start 2023-09-30T01:38:12.065 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.047+0000 7f2e82f10700 1 -- start start 2023-09-30T01:38:12.065 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.047+0000 7f2e82f10700 10 monclient: init 2023-09-30T01:38:12.065 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.051+0000 7f2e82f10700 10 monclient: _reopen_session rank -1 2023-09-30T01:38:12.065 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.051+0000 7f2e82f10700 10 monclient: _add_conns ranks=[2,7,6,3,8,5,1,4,0] 2023-09-30T01:38:12.065 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.051+0000 7f2e82f10700 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2e7c097a40 0x7f2e7c11cd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:38:12.065 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.051+0000 7f2e82f10700 10 monclient(hunting): picked mon.noname-b con 0x7f2e7c097a40 addr [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] 2023-09-30T01:38:12.065 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.051+0000 7f2e82f10700 1 --2- >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7f2e7c1311c0 0x7f2e7c11d290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:38:12.065 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.051+0000 7f2e82f10700 10 monclient(hunting): picked mon.noname-i con 0x7f2e7c1311c0 addr [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] 2023-09-30T01:38:12.065 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.051+0000 7f2e82f10700 1 --2- >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7f2e7c132e10 0x7f2e7c11d7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:38:12.066 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.051+0000 7f2e82f10700 10 monclient(hunting): picked mon.noname-d con 0x7f2e7c132e10 addr [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] 2023-09-30T01:38:12.066 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.051+0000 7f2e82f10700 10 monclient(hunting): start opening mon connection 2023-09-30T01:38:12.066 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.051+0000 7f2e82f10700 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e7c099c30 con 0x7f2e7c097a40 2023-09-30T01:38:12.066 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.051+0000 7f2e82f10700 10 monclient(hunting): start opening mon connection 2023-09-30T01:38:12.066 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.051+0000 7f2e82f10700 1 -- --> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] -- mon_getmap magic: 0 v1 -- 0x7f2e7c0996b0 con 0x7f2e7c132e10 2023-09-30T01:38:12.066 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.051+0000 7f2e82f10700 10 monclient(hunting): start opening mon connection 2023-09-30T01:38:12.066 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.051+0000 7f2e82f10700 1 -- --> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] -- mon_getmap magic: 0 v1 -- 0x7f2e7c0a2f00 con 0x7f2e7c1311c0 2023-09-30T01:38:12.066 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.051+0000 7f2e82f10700 10 monclient(hunting): _renew_subs 2023-09-30T01:38:12.066 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.051+0000 7f2e82f10700 10 monclient(hunting): authenticate will time out at 1164.088953s 2023-09-30T01:38:12.066 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.051+0000 7f2e8270f700 1 --2- >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7f2e7c132e10 0x7f2e7c11d7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:38:12.066 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.051+0000 7f2e81f0e700 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2e7c097a40 0x7f2e7c11cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:38:12.067 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.051+0000 7f2e8270f700 1 --2- >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7f2e7c132e10 0x7f2e7c11d7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3303/0 says I am v2:172.21.15.88:49780/0 (socket says 172.21.15.88:49780) 2023-09-30T01:38:12.067 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.051+0000 7f2e8270f700 1 -- 172.21.15.88:0/667867120 learned_addr learned my addr 172.21.15.88:0/667867120 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-09-30T01:38:12.067 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.051+0000 7f2e8270f700 10 monclient(hunting): get_auth_request con 0x7f2e7c132e10 auth_method 0 2023-09-30T01:38:12.067 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.051+0000 7f2e8270f700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:38:12.067 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.051+0000 7f2e8270f700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:38:12.067 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.051+0000 7f2e8270f700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:38:12.067 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.051+0000 7f2e81f0e700 10 monclient(hunting): get_auth_request con 0x7f2e7c097a40 auth_method 0 2023-09-30T01:38:12.067 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.051+0000 7f2e81f0e700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:38:12.067 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.051+0000 7f2e81f0e700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:38:12.067 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.051+0000 7f2e81f0e700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:38:12.067 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.051+0000 7f2e8170d700 1 --2- 172.21.15.88:0/667867120 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7f2e7c1311c0 0x7f2e7c11d290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:38:12.068 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.051+0000 7f2e8270f700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:38:12.068 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.051+0000 7f2e8270f700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:38:12.068 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.051+0000 7f2e8270f700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:38:12.068 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.051+0000 7f2e8170d700 10 monclient(hunting): get_auth_request con 0x7f2e7c1311c0 auth_method 0 2023-09-30T01:38:12.068 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.051+0000 7f2e8170d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:38:12.068 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.051+0000 7f2e8170d700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:38:12.068 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.051+0000 7f2e8170d700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:38:12.068 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.051+0000 7f2e81f0e700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:38:12.068 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.051+0000 7f2e81f0e700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:38:12.068 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.051+0000 7f2e81f0e700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:38:12.068 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.051+0000 7f2e8170d700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:38:12.068 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.051+0000 7f2e8170d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:38:12.069 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.051+0000 7f2e8170d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:38:12.069 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.051+0000 7f2e8270f700 10 monclient(hunting): handle_auth_done global_id 4812 payload 824 2023-09-30T01:38:12.069 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.051+0000 7f2e8270f700 1 -- 172.21.15.88:0/667867120 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7f2e7c1311c0 msgr2=0x7f2e7c11d290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:38:12.069 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.051+0000 7f2e8270f700 1 --2- 172.21.15.88:0/667867120 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7f2e7c1311c0 0x7f2e7c11d290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:38:12.069 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.051+0000 7f2e8270f700 1 -- 172.21.15.88:0/667867120 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2e7c097a40 msgr2=0x7f2e7c11cd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:38:12.069 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.051+0000 7f2e8270f700 1 --2- 172.21.15.88:0/667867120 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2e7c097a40 0x7f2e7c11cd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:38:12.069 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.051+0000 7f2e8270f700 10 monclient: _finish_hunting 0 2023-09-30T01:38:12.069 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.051+0000 7f2e8270f700 1 monclient: found mon.noname-d 2023-09-30T01:38:12.069 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.051+0000 7f2e8270f700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:38:12.069 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.051+0000 7f2e8270f700 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.88:3303/0 2023-09-30T01:38:12.069 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.051+0000 7f2e8270f700 1 -- 172.21.15.88:0/667867120 --> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e74009ae0 con 0x7f2e7c132e10 2023-09-30T01:38:12.070 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.051+0000 7f2e81f0e700 1 --2- 172.21.15.88:0/667867120 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2e7c097a40 0x7f2e7c11cd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:38:12.070 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.051+0000 7f2e8270f700 1 --2- 172.21.15.88:0/667867120 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7f2e7c132e10 0x7f2e7c11d7d0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f2e7c09a760 tx=0x7f2e74018000 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:38:12.070 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.051+0000 7f2e79ffb700 1 -- 172.21.15.88:0/667867120 <== mon.6 v2:172.21.15.88:3303/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f2e74007d80 con 0x7f2e7c132e10 2023-09-30T01:38:12.070 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.051+0000 7f2e79ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:38:12.070 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.051+0000 7f2e79ffb700 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2023-09-30T01:38:12.070 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.051+0000 7f2e79ffb700 10 monclient: dump: 2023-09-30T01:38:12.070 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:38:12.070 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:38:12.070 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:38:12.070 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:38:12.070 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:38:12.071 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:38:12.071 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:38:12.071 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:38:12.071 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:38:12.071 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:38:12.071 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:38:12.071 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:38:12.071 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:38:12.071 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:38:12.071 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:38:12.071 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:38:12.071 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.051+0000 7f2e79ffb700 10 monclient: _finish_auth 0 2023-09-30T01:38:12.072 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.051+0000 7f2e79ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:38:12.072 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.051+0000 7f2e79ffb700 1 -- 172.21.15.88:0/667867120 <== mon.6 v2:172.21.15.88:3303/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e74026460 con 0x7f2e7c132e10 2023-09-30T01:38:12.072 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.051+0000 7f2e82f10700 5 monclient: authenticate success, global_id 4812 2023-09-30T01:38:12.072 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.051+0000 7f2e79ffb700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:38:12.072 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.051+0000 7f2e79ffb700 1 -- 172.21.15.88:0/667867120 <== mon.6 v2:172.21.15.88:3303/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f2e74026a50 con 0x7f2e7c132e10 2023-09-30T01:38:12.072 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.051+0000 7f2e79ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:38:12.072 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.051+0000 7f2e79ffb700 10 monclient: got monmap 1 from mon.d (according to old e1) 2023-09-30T01:38:12.072 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.051+0000 7f2e7bfff700 4 set_mon_vals no callback set 2023-09-30T01:38:12.072 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.051+0000 7f2e79ffb700 10 monclient: dump: 2023-09-30T01:38:12.072 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:38:12.072 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:38:12.072 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:38:12.072 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:38:12.073 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:38:12.073 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:38:12.073 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:38:12.073 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:38:12.073 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:38:12.073 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:38:12.073 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:38:12.073 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:38:12.073 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:38:12.073 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:38:12.073 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:38:12.073 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:38:12.074 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.051+0000 7f2e82f10700 10 monclient: _renew_subs 2023-09-30T01:38:12.074 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.051+0000 7f2e82f10700 10 monclient: _send_mon_message to mon.d at v2:172.21.15.88:3303/0 2023-09-30T01:38:12.074 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.051+0000 7f2e82f10700 1 -- 172.21.15.88:0/667867120 --> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e7c11e0c0 con 0x7f2e7c132e10 2023-09-30T01:38:12.074 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.051+0000 7f2e82f10700 10 monclient: _renew_subs 2023-09-30T01:38:12.074 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.051+0000 7f2e82f10700 10 monclient: _send_mon_message to mon.d at v2:172.21.15.88:3303/0 2023-09-30T01:38:12.074 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.051+0000 7f2e82f10700 1 -- 172.21.15.88:0/667867120 --> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e7c0a2970 con 0x7f2e7c132e10 2023-09-30T01:38:12.074 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.055+0000 7f2e79ffb700 1 -- 172.21.15.88:0/667867120 <== mon.6 v2:172.21.15.88:3303/0 4 ==== mgrmap(e 5) v1 ==== 40144+0+0 (secure 0 0 0) 0x7f2e740265d0 con 0x7f2e7c132e10 2023-09-30T01:38:12.074 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.055+0000 7f2e82f10700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:38:12.074 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.055+0000 7f2e82f10700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:38:12.074 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.055+0000 7f2e79ffb700 1 --2- 172.21.15.88:0/667867120 >> v2:172.21.15.107:6812/29926 conn(0x7f2e68033360 0x7f2e68035810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:38:12.074 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.055+0000 7f2e82f10700 10 monclient: _send_mon_message to mon.d at v2:172.21.15.88:3303/0 2023-09-30T01:38:12.074 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.055+0000 7f2e82f10700 1 -- 172.21.15.88:0/667867120 --> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2e7c097fb0 con 0x7f2e7c132e10 2023-09-30T01:38:12.074 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.055+0000 7f2e79ffb700 1 -- 172.21.15.88:0/667867120 <== mon.6 v2:172.21.15.88:3303/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 3502+0+0 (secure 0 0 0) 0x7f2e74045130 con 0x7f2e7c132e10 2023-09-30T01:38:12.075 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.055+0000 7f2e81f0e700 1 --2- 172.21.15.88:0/667867120 >> v2:172.21.15.107:6812/29926 conn(0x7f2e68033360 0x7f2e68035810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:38:12.075 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.055+0000 7f2e81f0e700 10 monclient: get_auth_request con 0x7f2e68033360 auth_method 0 2023-09-30T01:38:12.075 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.055+0000 7f2e81f0e700 1 --2- 172.21.15.88:0/667867120 >> v2:172.21.15.107:6812/29926 conn(0x7f2e68033360 0x7f2e68035810 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f2e70001de0 tx=0x7f2e70009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:38:12.075 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.059+0000 7f2e79ffb700 1 -- 172.21.15.88:0/667867120 <== mon.6 v2:172.21.15.88:3303/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f2e74021020 con 0x7f2e7c132e10 2023-09-30T01:38:12.075 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.059+0000 7f2e79ffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:38:12.075 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.059+0000 7f2e79ffb700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:38:12.219 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.215+0000 7f2e82f10700 1 -- 172.21.15.88:0/667867120 --> v2:172.21.15.107:6812/29926 -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "0", "target": ["mon-mgr", ""]}) v1 -- 0x7f2e7c071960 con 0x7f2e68033360 2023-09-30T01:38:12.220 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.219+0000 7f2e79ffb700 1 -- 172.21.15.88:0/667867120 <== mgr.4108 v2:172.21.15.107:6812/29926 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 0 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f2e7c071960 con 0x7f2e68033360 2023-09-30T01:38:12.224 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.219+0000 7f2e82f10700 1 -- 172.21.15.88:0/667867120 >> v2:172.21.15.107:6812/29926 conn(0x7f2e68033360 msgr2=0x7f2e68035810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:38:12.224 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.219+0000 7f2e82f10700 1 --2- 172.21.15.88:0/667867120 >> v2:172.21.15.107:6812/29926 conn(0x7f2e68033360 0x7f2e68035810 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f2e70001de0 tx=0x7f2e70009040 comp rx=0 tx=0).stop 2023-09-30T01:38:12.224 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.223+0000 7f2e82f10700 10 monclient: shutdown 2023-09-30T01:38:12.224 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.223+0000 7f2e82f10700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:38:12.224 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.223+0000 7f2e82f10700 1 -- 172.21.15.88:0/667867120 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7f2e7c132e10 msgr2=0x7f2e7c11d7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:38:12.224 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.223+0000 7f2e82f10700 1 --2- 172.21.15.88:0/667867120 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7f2e7c132e10 0x7f2e7c11d7d0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f2e7c09a760 tx=0x7f2e74018000 comp rx=0 tx=0).stop 2023-09-30T01:38:12.224 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.223+0000 7f2e82f10700 1 -- 172.21.15.88:0/667867120 shutdown_connections 2023-09-30T01:38:12.224 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.223+0000 7f2e82f10700 1 --2- 172.21.15.88:0/667867120 >> v2:172.21.15.107:6812/29926 conn(0x7f2e68033360 0x7f2e68035810 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:38:12.225 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.223+0000 7f2e82f10700 1 --2- 172.21.15.88:0/667867120 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7f2e7c132e10 0x7f2e7c11d7d0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:38:12.225 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.223+0000 7f2e82f10700 1 --2- 172.21.15.88:0/667867120 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7f2e7c1311c0 0x7f2e7c11d290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:38:12.225 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.223+0000 7f2e82f10700 1 --2- 172.21.15.88:0/667867120 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2e7c097a40 0x7f2e7c11cd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:38:12.225 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.223+0000 7f2e82f10700 1 -- 172.21.15.88:0/667867120 >> 172.21.15.88:0/667867120 conn(0x7f2e7c06e5e0 msgr2=0x7f2e7c133b30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:38:12.225 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.223+0000 7f2e82f10700 1 -- 172.21.15.88:0/667867120 shutdown_connections 2023-09-30T01:38:12.225 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.223+0000 7f2e82f10700 1 -- 172.21.15.88:0/667867120 wait complete. 2023-09-30T01:38:12.225 INFO:teuthology.orchestra.run.smithi088.stderr:instructed osd(s) 0 to deep-scrub 2023-09-30T01:38:12.240 INFO:tasks.ceph:Scrubbing osd.1 2023-09-30T01:38:12.240 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2023-09-30T01:38:12.346 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.343+0000 7f8663fff700 10 monclient: get_monmap_and_config 2023-09-30T01:38:12.346 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.343+0000 7f8663fff700 10 monclient: build_initial_monmap 2023-09-30T01:38:12.346 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.343+0000 7f8663fff700 10 monclient: monmap: 2023-09-30T01:38:12.346 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2023-09-30T01:38:12.346 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:38:12.346 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:38:12.345848+0000 2023-09-30T01:38:12.347 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:38:12.345848+0000 2023-09-30T01:38:12.347 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2023-09-30T01:38:12.347 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:38:12.347 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a-legacy 2023-09-30T01:38:12.347 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.107:6789/0 mon.noname-f-legacy 2023-09-30T01:38:12.347 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.noname-b 2023-09-30T01:38:12.347 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.noname-g 2023-09-30T01:38:12.347 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.noname-c 2023-09-30T01:38:12.348 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.noname-h 2023-09-30T01:38:12.348 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.noname-d 2023-09-30T01:38:12.348 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.noname-i 2023-09-30T01:38:12.348 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.noname-e 2023-09-30T01:38:12.348 INFO:teuthology.orchestra.run.smithi088.stderr:9: v2:172.21.15.88:3300/0 mon.noname-a 2023-09-30T01:38:12.348 INFO:teuthology.orchestra.run.smithi088.stderr:10: v2:172.21.15.107:3300/0 mon.noname-f 2023-09-30T01:38:12.348 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:38:12.362 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.343+0000 7f8663fff700 1 Processor -- start 2023-09-30T01:38:12.362 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.343+0000 7f8663fff700 1 -- start start 2023-09-30T01:38:12.362 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.343+0000 7f8663fff700 10 monclient: init 2023-09-30T01:38:12.362 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.343+0000 7f8663fff700 10 monclient: _reopen_session rank -1 2023-09-30T01:38:12.362 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.343+0000 7f8663fff700 10 monclient: _add_conns ranks=[5,1,4,0,6,3,9,10,7,2,8] 2023-09-30T01:38:12.363 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.343+0000 7f8663fff700 1 --2- >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7f8664097a40 0x7f8664131210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:38:12.363 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.343+0000 7f8663fff700 10 monclient(hunting): picked mon.noname-h con 0x7f8664097a40 addr [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] 2023-09-30T01:38:12.363 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.343+0000 7f8663fff700 10 monclient(hunting): picked mon.noname-f-legacy con 0x7f86641327b0 addr v1:172.21.15.107:6789/0 2023-09-30T01:38:12.363 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.343+0000 7f8663fff700 1 --2- >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f8664132b80 0x7f86641339b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:38:12.363 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.343+0000 7f8663fff700 10 monclient(hunting): picked mon.noname-c con 0x7f8664132b80 addr [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] 2023-09-30T01:38:12.363 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.343+0000 7f8663fff700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f866404bb20 con 0x7f86641327b0 2023-09-30T01:38:12.363 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.343+0000 7f8663fff700 10 monclient(hunting): start opening mon connection 2023-09-30T01:38:12.364 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.343+0000 7f8663fff700 1 -- --> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] -- mon_getmap magic: 0 v1 -- 0x7f86640a3620 con 0x7f8664132b80 2023-09-30T01:38:12.364 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.343+0000 7f8663fff700 10 monclient(hunting): start opening mon connection 2023-09-30T01:38:12.364 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.343+0000 7f8663fff700 1 -- --> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] -- mon_getmap magic: 0 v1 -- 0x7f8664097e10 con 0x7f8664097a40 2023-09-30T01:38:12.364 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.343+0000 7f8663fff700 10 monclient(hunting): _renew_subs 2023-09-30T01:38:12.364 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.343+0000 7f8663fff700 10 monclient(hunting): authenticate will time out at 1164.383879s 2023-09-30T01:38:12.364 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.347+0000 7f86637fe700 1 --2- >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f8664132b80 0x7f86641339b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:38:12.364 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.347+0000 7f86637fe700 1 --2- >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f8664132b80 0x7f86641339b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3302/0 says I am v2:172.21.15.88:33944/0 (socket says 172.21.15.88:33944) 2023-09-30T01:38:12.365 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.347+0000 7f86637fe700 1 -- 172.21.15.88:0/2461991501 learned_addr learned my addr 172.21.15.88:0/2461991501 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-09-30T01:38:12.365 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.347+0000 7f86637fe700 10 monclient(hunting): get_auth_request con 0x7f8664132b80 auth_method 0 2023-09-30T01:38:12.365 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.347+0000 7f86637fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:38:12.365 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.347+0000 7f86637fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:38:12.365 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.347+0000 7f86637fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:38:12.365 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.347+0000 7f8662ffd700 1 --2- 172.21.15.88:0/2461991501 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7f8664097a40 0x7f8664131210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:38:12.365 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.347+0000 7f86637fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:38:12.365 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.347+0000 7f86637fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:38:12.366 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.347+0000 7f86637fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:38:12.366 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.347+0000 7f8662ffd700 10 monclient(hunting): get_auth_request con 0x7f8664097a40 auth_method 0 2023-09-30T01:38:12.366 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.347+0000 7f8662ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:38:12.366 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.347+0000 7f8662ffd700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:38:12.366 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.347+0000 7f8662ffd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:38:12.366 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.347+0000 7f8662ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:38:12.366 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.347+0000 7f8662ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:38:12.367 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.347+0000 7f8662ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:38:12.367 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.347+0000 7f86637fe700 10 monclient(hunting): handle_auth_done global_id 4954 payload 274 2023-09-30T01:38:12.367 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.347+0000 7f86637fe700 1 -- 172.21.15.88:0/2461991501 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7f8664097a40 msgr2=0x7f8664131210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:38:12.367 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.347+0000 7f86637fe700 1 --2- 172.21.15.88:0/2461991501 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7f8664097a40 0x7f8664131210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:38:12.367 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.347+0000 7f86637fe700 1 -- 172.21.15.88:0/2461991501 >> v1:172.21.15.107:6789/0 conn(0x7f86641327b0 legacy=0x7f86641332a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:38:12.367 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.347+0000 7f86637fe700 10 monclient: _finish_hunting 0 2023-09-30T01:38:12.367 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.347+0000 7f86637fe700 1 monclient: found mon.noname-c 2023-09-30T01:38:12.367 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.347+0000 7f86637fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:38:12.368 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.347+0000 7f86637fe700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.88:3302/0 2023-09-30T01:38:12.368 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.347+0000 7f86637fe700 1 -- 172.21.15.88:0/2461991501 --> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8664099110 con 0x7f8664132b80 2023-09-30T01:38:12.368 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.347+0000 7f86637fe700 1 --2- 172.21.15.88:0/2461991501 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f8664132b80 0x7f86641339b0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f865c009f50 tx=0x7f865c009840 comp rx=0 tx=0).ready entity=mon.4 client_cookie=a0173720a7ee424c server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:38:12.368 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.347+0000 7f8661ffb700 1 -- 172.21.15.88:0/2461991501 <== mon.4 v2:172.21.15.88:3302/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f865c00e070 con 0x7f8664132b80 2023-09-30T01:38:12.368 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.347+0000 7f8661ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:38:12.368 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.347+0000 7f8661ffb700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T01:38:12.368 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.347+0000 7f8661ffb700 10 monclient: dump: 2023-09-30T01:38:12.369 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:38:12.369 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:38:12.369 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:38:12.369 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:38:12.369 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:38:12.369 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:38:12.369 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:38:12.369 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:38:12.370 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:38:12.370 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:38:12.370 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:38:12.370 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:38:12.370 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:38:12.370 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:38:12.370 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:38:12.371 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:38:12.371 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.347+0000 7f8661ffb700 10 monclient: _finish_auth 0 2023-09-30T01:38:12.371 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.347+0000 7f8661ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:38:12.371 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.347+0000 7f8663fff700 5 monclient: authenticate success, global_id 4954 2023-09-30T01:38:12.371 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.347+0000 7f8663fff700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:38:12.371 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.347+0000 7f8661ffb700 1 -- 172.21.15.88:0/2461991501 <== mon.4 v2:172.21.15.88:3302/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f865c00be80 con 0x7f8664132b80 2023-09-30T01:38:12.371 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.347+0000 7f8661ffb700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:38:12.372 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.347+0000 7f8663fff700 10 monclient: get_monmap_and_config success 2023-09-30T01:38:12.372 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.347+0000 7f8663fff700 4 set_mon_vals no callback set 2023-09-30T01:38:12.372 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.347+0000 7f8661ffb700 1 -- 172.21.15.88:0/2461991501 <== mon.4 v2:172.21.15.88:3302/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f865c0092e0 con 0x7f8664132b80 2023-09-30T01:38:12.372 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.347+0000 7f8661ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:38:12.372 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.347+0000 7f8661ffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T01:38:12.372 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.347+0000 7f8661ffb700 10 monclient: dump: 2023-09-30T01:38:12.372 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:38:12.372 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:38:12.372 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:38:12.372 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:38:12.373 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:38:12.373 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:38:12.373 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:38:12.373 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:38:12.373 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:38:12.373 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:38:12.373 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:38:12.373 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:38:12.373 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:38:12.373 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:38:12.374 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:38:12.374 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:38:12.374 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.347+0000 7f8663fff700 10 monclient: shutdown 2023-09-30T01:38:12.374 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.347+0000 7f8663fff700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:38:12.374 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.347+0000 7f8663fff700 1 -- 172.21.15.88:0/2461991501 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f8664132b80 msgr2=0x7f86641339b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:38:12.374 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.347+0000 7f8663fff700 1 --2- 172.21.15.88:0/2461991501 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f8664132b80 0x7f86641339b0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f865c009f50 tx=0x7f865c009840 comp rx=0 tx=0).stop 2023-09-30T01:38:12.374 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.347+0000 7f8663fff700 1 -- 172.21.15.88:0/2461991501 shutdown_connections 2023-09-30T01:38:12.374 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.347+0000 7f8663fff700 1 --2- 172.21.15.88:0/2461991501 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f8664132b80 0x7f86641339b0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:38:12.374 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.347+0000 7f8663fff700 1 --2- 172.21.15.88:0/2461991501 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7f8664097a40 0x7f8664131210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:38:12.374 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.347+0000 7f8663fff700 1 -- 172.21.15.88:0/2461991501 >> 172.21.15.88:0/2461991501 conn(0x7f866406e5e0 msgr2=0x7f866406e9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:38:12.374 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.347+0000 7f8663fff700 1 -- 172.21.15.88:0/2461991501 shutdown_connections 2023-09-30T01:38:12.375 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.347+0000 7f8663fff700 1 -- 172.21.15.88:0/2461991501 wait complete. 2023-09-30T01:38:12.375 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.347+0000 7f8663fff700 10 monclient: build_initial_monmap 2023-09-30T01:38:12.375 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.347+0000 7f8663fff700 10 monclient: monmap: 2023-09-30T01:38:12.375 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2023-09-30T01:38:12.375 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:38:12.375 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 0.000000 2023-09-30T01:38:12.375 INFO:teuthology.orchestra.run.smithi088.stderr:created 0.000000 2023-09-30T01:38:12.375 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2023-09-30T01:38:12.375 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 1 2023-09-30T01:38:12.375 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.noname-a 2023-09-30T01:38:12.375 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.noname-f 2023-09-30T01:38:12.376 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.noname-b 2023-09-30T01:38:12.376 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.noname-g 2023-09-30T01:38:12.376 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.noname-c 2023-09-30T01:38:12.376 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.noname-h 2023-09-30T01:38:12.376 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.noname-d 2023-09-30T01:38:12.376 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.noname-i 2023-09-30T01:38:12.376 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.noname-e 2023-09-30T01:38:12.376 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:38:12.376 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.347+0000 7f8663fff700 1 Processor -- start 2023-09-30T01:38:12.376 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.347+0000 7f8663fff700 1 -- start start 2023-09-30T01:38:12.376 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.347+0000 7f8663fff700 10 monclient: init 2023-09-30T01:38:12.377 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.351+0000 7f8663fff700 10 monclient: _reopen_session rank -1 2023-09-30T01:38:12.377 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.351+0000 7f8663fff700 10 monclient: _add_conns ranks=[1,7,6,4,8,5,3,0,2] 2023-09-30T01:38:12.377 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.351+0000 7f8663fff700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8664097a40 0x7f86640a1d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:38:12.377 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.351+0000 7f8663fff700 10 monclient(hunting): picked mon.noname-f con 0x7f8664097a40 addr [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] 2023-09-30T01:38:12.377 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.351+0000 7f8663fff700 1 --2- >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7f86641327b0 0x7f866411cdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:38:12.377 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.351+0000 7f8663fff700 10 monclient(hunting): picked mon.noname-i con 0x7f86641327b0 addr [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] 2023-09-30T01:38:12.377 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.351+0000 7f8663fff700 1 --2- >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7f8664132b80 0x7f866411d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:38:12.377 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.351+0000 7f8663fff700 10 monclient(hunting): picked mon.noname-d con 0x7f8664132b80 addr [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] 2023-09-30T01:38:12.377 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.351+0000 7f8663fff700 10 monclient(hunting): start opening mon connection 2023-09-30T01:38:12.377 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.351+0000 7f8663fff700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f866411f350 con 0x7f8664097a40 2023-09-30T01:38:12.377 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.351+0000 7f8663fff700 10 monclient(hunting): start opening mon connection 2023-09-30T01:38:12.377 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.351+0000 7f8663fff700 1 -- --> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] -- mon_getmap magic: 0 v1 -- 0x7f8664099b30 con 0x7f8664132b80 2023-09-30T01:38:12.378 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.351+0000 7f8663fff700 10 monclient(hunting): start opening mon connection 2023-09-30T01:38:12.378 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.351+0000 7f8663fff700 1 -- --> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] -- mon_getmap magic: 0 v1 -- 0x7f8664099640 con 0x7f86641327b0 2023-09-30T01:38:12.378 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.351+0000 7f8663fff700 10 monclient(hunting): _renew_subs 2023-09-30T01:38:12.378 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.351+0000 7f8663fff700 10 monclient(hunting): authenticate will time out at 1164.388789s 2023-09-30T01:38:12.378 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.351+0000 7f86637fe700 1 --2- >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7f8664132b80 0x7f866411d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:38:12.378 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.351+0000 7f86627fc700 1 --2- >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7f86641327b0 0x7f866411cdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:38:12.378 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.351+0000 7f86637fe700 1 --2- >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7f8664132b80 0x7f866411d300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3303/0 says I am v2:172.21.15.88:49786/0 (socket says 172.21.15.88:49786) 2023-09-30T01:38:12.378 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.351+0000 7f8662ffd700 1 --2- 172.21.15.88:0/2335866303 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8664097a40 0x7f86640a1d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:38:12.378 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.351+0000 7f86637fe700 1 -- 172.21.15.88:0/2335866303 learned_addr learned my addr 172.21.15.88:0/2335866303 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-09-30T01:38:12.378 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.351+0000 7f86637fe700 10 monclient(hunting): get_auth_request con 0x7f8664132b80 auth_method 0 2023-09-30T01:38:12.378 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.351+0000 7f86637fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:38:12.379 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.351+0000 7f86637fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:38:12.379 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.351+0000 7f86637fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:38:12.379 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.351+0000 7f86627fc700 10 monclient(hunting): get_auth_request con 0x7f86641327b0 auth_method 0 2023-09-30T01:38:12.379 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.351+0000 7f86627fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:38:12.379 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.351+0000 7f86627fc700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:38:12.379 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.351+0000 7f86627fc700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:38:12.379 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.351+0000 7f8662ffd700 10 monclient(hunting): get_auth_request con 0x7f8664097a40 auth_method 0 2023-09-30T01:38:12.379 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.351+0000 7f8662ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:38:12.379 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.351+0000 7f8662ffd700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:38:12.379 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.351+0000 7f8662ffd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:38:12.379 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.351+0000 7f86637fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:38:12.380 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.351+0000 7f86637fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:38:12.380 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.351+0000 7f86637fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:38:12.380 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.351+0000 7f86627fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:38:12.380 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.351+0000 7f86627fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:38:12.380 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.351+0000 7f86627fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:38:12.380 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.351+0000 7f8662ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:38:12.380 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.351+0000 7f8662ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:38:12.380 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.351+0000 7f8662ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:38:12.380 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.351+0000 7f86637fe700 10 monclient(hunting): handle_auth_done global_id 4821 payload 824 2023-09-30T01:38:12.380 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.351+0000 7f86637fe700 1 -- 172.21.15.88:0/2335866303 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7f86641327b0 msgr2=0x7f866411cdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:38:12.380 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.351+0000 7f86637fe700 1 --2- 172.21.15.88:0/2335866303 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7f86641327b0 0x7f866411cdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:38:12.381 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.351+0000 7f86637fe700 1 -- 172.21.15.88:0/2335866303 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8664097a40 msgr2=0x7f86640a1d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:38:12.381 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.351+0000 7f86637fe700 1 --2- 172.21.15.88:0/2335866303 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8664097a40 0x7f86640a1d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:38:12.381 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.351+0000 7f86637fe700 10 monclient: _finish_hunting 0 2023-09-30T01:38:12.381 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.351+0000 7f86637fe700 1 monclient: found mon.noname-d 2023-09-30T01:38:12.381 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.351+0000 7f86637fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:38:12.381 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.351+0000 7f86637fe700 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.88:3303/0 2023-09-30T01:38:12.381 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.351+0000 7f86637fe700 1 -- 172.21.15.88:0/2335866303 --> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f865c009ae0 con 0x7f8664132b80 2023-09-30T01:38:12.381 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.351+0000 7f8662ffd700 1 --2- 172.21.15.88:0/2335866303 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8664097a40 0x7f86640a1d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:38:12.381 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.351+0000 7f86637fe700 1 --2- 172.21.15.88:0/2335866303 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7f8664132b80 0x7f866411d300 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f86640a3760 tx=0x7f865c018000 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:38:12.381 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.351+0000 7f864affd700 1 -- 172.21.15.88:0/2335866303 <== mon.6 v2:172.21.15.88:3303/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f865c003300 con 0x7f8664132b80 2023-09-30T01:38:12.381 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.351+0000 7f864affd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:38:12.382 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.351+0000 7f864affd700 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2023-09-30T01:38:12.382 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.351+0000 7f864affd700 10 monclient: dump: 2023-09-30T01:38:12.382 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:38:12.382 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:38:12.382 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:38:12.382 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:38:12.382 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:38:12.382 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:38:12.382 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:38:12.382 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:38:12.383 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:38:12.383 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:38:12.383 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:38:12.383 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:38:12.383 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:38:12.383 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:38:12.383 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:38:12.383 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:38:12.383 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.351+0000 7f864affd700 10 monclient: _finish_auth 0 2023-09-30T01:38:12.383 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.351+0000 7f864affd700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:38:12.383 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.351+0000 7f864affd700 1 -- 172.21.15.88:0/2335866303 <== mon.6 v2:172.21.15.88:3303/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f865c0092e0 con 0x7f8664132b80 2023-09-30T01:38:12.384 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.351+0000 7f8663fff700 5 monclient: authenticate success, global_id 4821 2023-09-30T01:38:12.384 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.351+0000 7f8663fff700 10 monclient: _renew_subs 2023-09-30T01:38:12.384 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.351+0000 7f8663fff700 10 monclient: _send_mon_message to mon.d at v2:172.21.15.88:3303/0 2023-09-30T01:38:12.384 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.351+0000 7f8663fff700 1 -- 172.21.15.88:0/2335866303 --> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f866411f070 con 0x7f8664132b80 2023-09-30T01:38:12.384 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.351+0000 7f864affd700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:38:12.384 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.351+0000 7f864affd700 1 -- 172.21.15.88:0/2335866303 <== mon.6 v2:172.21.15.88:3303/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f865c007000 con 0x7f8664132b80 2023-09-30T01:38:12.384 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.351+0000 7f864affd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:38:12.384 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.351+0000 7f864affd700 10 monclient: got monmap 1 from mon.d (according to old e1) 2023-09-30T01:38:12.384 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.351+0000 7f86617fa700 4 set_mon_vals no callback set 2023-09-30T01:38:12.384 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.351+0000 7f864affd700 10 monclient: dump: 2023-09-30T01:38:12.384 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:38:12.384 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:38:12.385 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:38:12.385 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:38:12.385 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:38:12.385 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:38:12.385 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:38:12.385 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:38:12.385 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:38:12.385 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:38:12.385 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:38:12.385 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:38:12.385 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:38:12.385 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:38:12.386 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:38:12.386 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:38:12.386 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.351+0000 7f8663fff700 10 monclient: _renew_subs 2023-09-30T01:38:12.386 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.351+0000 7f8663fff700 10 monclient: _send_mon_message to mon.d at v2:172.21.15.88:3303/0 2023-09-30T01:38:12.386 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.351+0000 7f8663fff700 1 -- 172.21.15.88:0/2335866303 --> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f86640a2810 con 0x7f8664132b80 2023-09-30T01:38:12.386 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.351+0000 7f864affd700 1 -- 172.21.15.88:0/2335866303 <== mon.6 v2:172.21.15.88:3303/0 4 ==== mgrmap(e 5) v1 ==== 40144+0+0 (secure 0 0 0) 0x7f865c023070 con 0x7f8664132b80 2023-09-30T01:38:12.386 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.351+0000 7f8663fff700 10 monclient: _send_mon_message to mon.d at v2:172.21.15.88:3303/0 2023-09-30T01:38:12.386 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.351+0000 7f8663fff700 1 -- 172.21.15.88:0/2335866303 --> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f8664071960 con 0x7f8664132b80 2023-09-30T01:38:12.386 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.351+0000 7f864affd700 1 --2- 172.21.15.88:0/2335866303 >> v2:172.21.15.107:6812/29926 conn(0x7f8650033360 0x7f8650035810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:38:12.386 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.355+0000 7f864affd700 1 -- 172.21.15.88:0/2335866303 <== mon.6 v2:172.21.15.88:3303/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 3502+0+0 (secure 0 0 0) 0x7f865c045090 con 0x7f8664132b80 2023-09-30T01:38:12.386 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.355+0000 7f8662ffd700 1 --2- 172.21.15.88:0/2335866303 >> v2:172.21.15.107:6812/29926 conn(0x7f8650033360 0x7f8650035810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:38:12.386 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.355+0000 7f8662ffd700 10 monclient: get_auth_request con 0x7f8650033360 auth_method 0 2023-09-30T01:38:12.387 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.355+0000 7f864affd700 1 --2- 172.21.15.88:0/2335866303 >> v2:172.21.15.88:6804/30175 conn(0x7f8650038c90 0x7f865003b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:38:12.387 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.355+0000 7f864affd700 1 -- 172.21.15.88:0/2335866303 --> v2:172.21.15.88:6804/30175 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f865003b710 con 0x7f8650038c90 2023-09-30T01:38:12.387 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.355+0000 7f864affd700 1 -- 172.21.15.88:0/2335866303 <== mon.6 v2:172.21.15.88:3303/0 6 ==== mon_get_version_reply(handle=1 version=15) v2 ==== 24+0+0 (secure 0 0 0) 0x7f865c026470 con 0x7f8664132b80 2023-09-30T01:38:12.387 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.355+0000 7f864affd700 10 monclient: handle_get_version_reply finishing 1 version 15 2023-09-30T01:38:12.387 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.355+0000 7f86627fc700 1 --2- 172.21.15.88:0/2335866303 >> v2:172.21.15.88:6804/30175 conn(0x7f8650038c90 0x7f865003b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:38:12.387 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.355+0000 7f86627fc700 10 monclient: get_auth_request con 0x7f8650038c90 auth_method 0 2023-09-30T01:38:12.387 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.355+0000 7f8662ffd700 1 --2- 172.21.15.88:0/2335866303 >> v2:172.21.15.107:6812/29926 conn(0x7f8650033360 0x7f8650035810 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f86540113e0 tx=0x7f86540091a0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:38:12.387 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.355+0000 7f86627fc700 1 --2- 172.21.15.88:0/2335866303 >> v2:172.21.15.88:6804/30175 conn(0x7f8650038c90 0x7f865003b0a0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:38:12.387 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.355+0000 7f864affd700 1 -- 172.21.15.88:0/2335866303 <== osd.1 v2:172.21.15.88:6804/30175 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23999 (crc 0 0 0) 0x7f865003b710 con 0x7f8650038c90 2023-09-30T01:38:12.390 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.371+0000 7f8663fff700 1 -- 172.21.15.88:0/2335866303 --> v2:172.21.15.88:6804/30175 -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f866411d840 con 0x7f8650038c90 2023-09-30T01:38:12.390 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.371+0000 7f864affd700 1 -- 172.21.15.88:0/2335866303 <== osd.1 v2:172.21.15.88:6804/30175 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7f866411d840 con 0x7f8650038c90 2023-09-30T01:38:12.390 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-09-30T01:38:12.390 INFO:teuthology.orchestra.run.smithi088.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-09-30T01:38:12.390 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-09-30T01:38:12.392 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.371+0000 7f8663fff700 1 -- 172.21.15.88:0/2335866303 >> v2:172.21.15.88:6804/30175 conn(0x7f8650038c90 msgr2=0x7f865003b0a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:38:12.392 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.375+0000 7f8663fff700 1 --2- 172.21.15.88:0/2335866303 >> v2:172.21.15.88:6804/30175 conn(0x7f8650038c90 0x7f865003b0a0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:38:12.392 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.375+0000 7f8663fff700 1 -- 172.21.15.88:0/2335866303 >> v2:172.21.15.107:6812/29926 conn(0x7f8650033360 msgr2=0x7f8650035810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:38:12.392 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.375+0000 7f8663fff700 1 --2- 172.21.15.88:0/2335866303 >> v2:172.21.15.107:6812/29926 conn(0x7f8650033360 0x7f8650035810 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f86540113e0 tx=0x7f86540091a0 comp rx=0 tx=0).stop 2023-09-30T01:38:12.392 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.375+0000 7f8663fff700 10 monclient: shutdown 2023-09-30T01:38:12.392 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.375+0000 7f8663fff700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:38:12.392 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.375+0000 7f8663fff700 1 -- 172.21.15.88:0/2335866303 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7f8664132b80 msgr2=0x7f866411d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:38:12.392 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.375+0000 7f8663fff700 1 --2- 172.21.15.88:0/2335866303 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7f8664132b80 0x7f866411d300 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f86640a3760 tx=0x7f865c018000 comp rx=0 tx=0).stop 2023-09-30T01:38:12.393 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.375+0000 7f86637fe700 1 -- 172.21.15.88:0/2335866303 reap_dead start 2023-09-30T01:38:12.393 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.375+0000 7f8663fff700 1 -- 172.21.15.88:0/2335866303 shutdown_connections 2023-09-30T01:38:12.393 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.375+0000 7f8663fff700 1 -- 172.21.15.88:0/2335866303 >> 172.21.15.88:0/2335866303 conn(0x7f866406e5e0 msgr2=0x7f86641316e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:38:12.393 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.375+0000 7f8663fff700 1 -- 172.21.15.88:0/2335866303 shutdown_connections 2023-09-30T01:38:12.393 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.375+0000 7f8663fff700 1 -- 172.21.15.88:0/2335866303 wait complete. 2023-09-30T01:38:12.393 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2023-09-30T01:38:12.499 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.495+0000 7f80d1292700 10 monclient: get_monmap_and_config 2023-09-30T01:38:12.499 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.495+0000 7f80d1292700 10 monclient: build_initial_monmap 2023-09-30T01:38:12.499 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.495+0000 7f80d1292700 10 monclient: monmap: 2023-09-30T01:38:12.499 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2023-09-30T01:38:12.499 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:38:12.499 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:38:12.498570+0000 2023-09-30T01:38:12.499 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:38:12.498570+0000 2023-09-30T01:38:12.500 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2023-09-30T01:38:12.500 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:38:12.500 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a-legacy 2023-09-30T01:38:12.500 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.107:6789/0 mon.noname-f-legacy 2023-09-30T01:38:12.500 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.noname-b 2023-09-30T01:38:12.500 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.noname-g 2023-09-30T01:38:12.500 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.noname-c 2023-09-30T01:38:12.500 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.noname-h 2023-09-30T01:38:12.500 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.noname-d 2023-09-30T01:38:12.500 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.noname-i 2023-09-30T01:38:12.500 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.noname-e 2023-09-30T01:38:12.500 INFO:teuthology.orchestra.run.smithi088.stderr:9: v2:172.21.15.88:3300/0 mon.noname-a 2023-09-30T01:38:12.501 INFO:teuthology.orchestra.run.smithi088.stderr:10: v2:172.21.15.107:3300/0 mon.noname-f 2023-09-30T01:38:12.501 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:38:12.501 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.495+0000 7f80d1292700 1 Processor -- start 2023-09-30T01:38:12.501 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.495+0000 7f80d1292700 1 -- start start 2023-09-30T01:38:12.501 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.495+0000 7f80d1292700 10 monclient: init 2023-09-30T01:38:12.509 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.499+0000 7f80d1292700 10 monclient: _reopen_session rank -1 2023-09-30T01:38:12.509 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.499+0000 7f80d1292700 10 monclient: _add_conns ranks=[2,10,6,7,1,3,9,8,4,5,0] 2023-09-30T01:38:12.510 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.499+0000 7f80d1292700 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f80cc0a35f0 0x7f80cc133330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:38:12.510 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.499+0000 7f80d1292700 10 monclient(hunting): picked mon.noname-b con 0x7f80cc0a35f0 addr [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] 2023-09-30T01:38:12.510 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.499+0000 7f80d1292700 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f80cc133870 0x7f80cc133c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:38:12.510 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.499+0000 7f80d1292700 10 monclient(hunting): picked mon.noname-f con 0x7f80cc133870 addr v2:172.21.15.107:3300/0 2023-09-30T01:38:12.510 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.499+0000 7f80d1292700 1 --2- >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7f80cc0a46f0 0x7f80cc0a4b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:38:12.510 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.499+0000 7f80d1292700 10 monclient(hunting): picked mon.noname-d con 0x7f80cc0a46f0 addr [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] 2023-09-30T01:38:12.510 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.499+0000 7f80d1292700 10 monclient(hunting): start opening mon connection 2023-09-30T01:38:12.510 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.499+0000 7f80d1292700 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f80cc097ad0 con 0x7f80cc133870 2023-09-30T01:38:12.511 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.499+0000 7f80d1292700 10 monclient(hunting): start opening mon connection 2023-09-30T01:38:12.511 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.499+0000 7f80d1292700 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f80cc1341c0 con 0x7f80cc0a35f0 2023-09-30T01:38:12.511 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.499+0000 7f80d1292700 10 monclient(hunting): start opening mon connection 2023-09-30T01:38:12.511 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.499+0000 7f80d1292700 1 -- --> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] -- mon_getmap magic: 0 v1 -- 0x7f80cc0a0940 con 0x7f80cc0a46f0 2023-09-30T01:38:12.511 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.499+0000 7f80d1292700 10 monclient(hunting): _renew_subs 2023-09-30T01:38:12.511 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.499+0000 7f80d1292700 10 monclient(hunting): authenticate will time out at 1164.536270s 2023-09-30T01:38:12.511 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.499+0000 7f80caffd700 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f80cc0a35f0 0x7f80cc133330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:38:12.511 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.499+0000 7f80cb7fe700 1 --2- >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7f80cc0a46f0 0x7f80cc0a4b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:38:12.511 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.499+0000 7f80cb7fe700 1 --2- >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7f80cc0a46f0 0x7f80cc0a4b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3303/0 says I am v2:172.21.15.88:49794/0 (socket says 172.21.15.88:49794) 2023-09-30T01:38:12.511 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.499+0000 7f80caffd700 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f80cc0a35f0 0x7f80cc133330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3301/0 says I am v2:172.21.15.88:53512/0 (socket says 172.21.15.88:53512) 2023-09-30T01:38:12.511 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.499+0000 7f80cb7fe700 1 -- 172.21.15.88:0/314240718 learned_addr learned my addr 172.21.15.88:0/314240718 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-09-30T01:38:12.511 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.499+0000 7f80ca7fc700 1 --2- 172.21.15.88:0/314240718 >> v2:172.21.15.107:3300/0 conn(0x7f80cc133870 0x7f80cc133c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:38:12.512 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.499+0000 7f80cb7fe700 10 monclient(hunting): get_auth_request con 0x7f80cc0a46f0 auth_method 0 2023-09-30T01:38:12.512 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.499+0000 7f80cb7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:38:12.512 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.499+0000 7f80cb7fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:38:12.512 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.499+0000 7f80cb7fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:38:12.512 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.499+0000 7f80caffd700 10 monclient(hunting): get_auth_request con 0x7f80cc0a35f0 auth_method 0 2023-09-30T01:38:12.512 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.499+0000 7f80caffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:38:12.512 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.499+0000 7f80caffd700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:38:12.512 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.499+0000 7f80caffd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:38:12.512 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.499+0000 7f80ca7fc700 10 monclient(hunting): get_auth_request con 0x7f80cc133870 auth_method 0 2023-09-30T01:38:12.512 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.499+0000 7f80ca7fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:38:12.512 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.499+0000 7f80ca7fc700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:38:12.512 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.499+0000 7f80ca7fc700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:38:12.512 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.499+0000 7f80cb7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:38:12.513 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.499+0000 7f80cb7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:38:12.513 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.499+0000 7f80cb7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:38:12.513 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.499+0000 7f80ca7fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:38:12.513 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.499+0000 7f80ca7fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:38:12.513 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.499+0000 7f80ca7fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:38:12.513 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.499+0000 7f80caffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:38:12.513 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.499+0000 7f80caffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:38:12.513 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.499+0000 7f80caffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:38:12.513 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.499+0000 7f80cb7fe700 10 monclient(hunting): handle_auth_done global_id 4830 payload 274 2023-09-30T01:38:12.513 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.499+0000 7f80cb7fe700 1 -- 172.21.15.88:0/314240718 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f80cc0a35f0 msgr2=0x7f80cc133330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:38:12.513 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.499+0000 7f80cb7fe700 1 --2- 172.21.15.88:0/314240718 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f80cc0a35f0 0x7f80cc133330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:38:12.513 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.499+0000 7f80cb7fe700 1 -- 172.21.15.88:0/314240718 >> v2:172.21.15.107:3300/0 conn(0x7f80cc133870 msgr2=0x7f80cc133c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:38:12.513 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.499+0000 7f80cb7fe700 1 --2- 172.21.15.88:0/314240718 >> v2:172.21.15.107:3300/0 conn(0x7f80cc133870 0x7f80cc133c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:38:12.514 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.499+0000 7f80cb7fe700 10 monclient: _finish_hunting 0 2023-09-30T01:38:12.514 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.499+0000 7f80cb7fe700 1 monclient: found mon.noname-d 2023-09-30T01:38:12.514 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.499+0000 7f80cb7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:38:12.514 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.499+0000 7f80cb7fe700 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.88:3303/0 2023-09-30T01:38:12.514 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.499+0000 7f80cb7fe700 1 -- 172.21.15.88:0/314240718 --> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80cc05ecb0 con 0x7f80cc0a46f0 2023-09-30T01:38:12.514 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.499+0000 7f80ca7fc700 1 --2- 172.21.15.88:0/314240718 >> v2:172.21.15.107:3300/0 conn(0x7f80cc133870 0x7f80cc133c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:38:12.514 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.499+0000 7f80cb7fe700 1 --2- 172.21.15.88:0/314240718 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7f80cc0a46f0 0x7f80cc0a4b60 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f80bc0099e0 tx=0x7f80bc009250 comp rx=0 tx=0).ready entity=mon.6 client_cookie=9f6875ab886b76c7 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:38:12.514 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.499+0000 7f80c9ffb700 1 -- 172.21.15.88:0/314240718 <== mon.6 v2:172.21.15.88:3303/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f80bc00bac0 con 0x7f80cc0a46f0 2023-09-30T01:38:12.514 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.499+0000 7f80c9ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:38:12.514 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.499+0000 7f80c9ffb700 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2023-09-30T01:38:12.514 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.499+0000 7f80c9ffb700 10 monclient: dump: 2023-09-30T01:38:12.515 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:38:12.515 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:38:12.515 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:38:12.515 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:38:12.515 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:38:12.515 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:38:12.515 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:38:12.515 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:38:12.515 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:38:12.515 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:38:12.515 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:38:12.515 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:38:12.516 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:38:12.516 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:38:12.516 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:38:12.516 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:38:12.516 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.499+0000 7f80c9ffb700 10 monclient: _finish_auth 0 2023-09-30T01:38:12.516 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.499+0000 7f80c9ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:38:12.516 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.499+0000 7f80c9ffb700 1 -- 172.21.15.88:0/314240718 <== mon.6 v2:172.21.15.88:3303/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f80bc00bc40 con 0x7f80cc0a46f0 2023-09-30T01:38:12.516 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.499+0000 7f80d1292700 5 monclient: authenticate success, global_id 4830 2023-09-30T01:38:12.516 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.499+0000 7f80d1292700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:38:12.516 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.499+0000 7f80c9ffb700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:38:12.516 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.499+0000 7f80d1292700 10 monclient: get_monmap_and_config success 2023-09-30T01:38:12.516 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.499+0000 7f80c9ffb700 1 -- 172.21.15.88:0/314240718 <== mon.6 v2:172.21.15.88:3303/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f80bc00bdb0 con 0x7f80cc0a46f0 2023-09-30T01:38:12.516 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.499+0000 7f80c9ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:38:12.517 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.499+0000 7f80d1292700 4 set_mon_vals no callback set 2023-09-30T01:38:12.517 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.499+0000 7f80c9ffb700 10 monclient: got monmap 1 from mon.d (according to old e1) 2023-09-30T01:38:12.517 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.499+0000 7f80c9ffb700 10 monclient: dump: 2023-09-30T01:38:12.517 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:38:12.517 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:38:12.517 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:38:12.517 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:38:12.517 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:38:12.517 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:38:12.517 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:38:12.517 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:38:12.517 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:38:12.517 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:38:12.518 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:38:12.518 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:38:12.518 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:38:12.518 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:38:12.518 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:38:12.518 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:38:12.518 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.499+0000 7f80d1292700 10 monclient: shutdown 2023-09-30T01:38:12.518 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.499+0000 7f80d1292700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:38:12.518 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.499+0000 7f80d1292700 1 -- 172.21.15.88:0/314240718 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7f80cc0a46f0 msgr2=0x7f80cc0a4b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:38:12.518 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.499+0000 7f80d1292700 1 --2- 172.21.15.88:0/314240718 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7f80cc0a46f0 0x7f80cc0a4b60 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f80bc0099e0 tx=0x7f80bc009250 comp rx=0 tx=0).stop 2023-09-30T01:38:12.518 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.499+0000 7f80d1292700 1 -- 172.21.15.88:0/314240718 shutdown_connections 2023-09-30T01:38:12.518 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.499+0000 7f80d1292700 1 --2- 172.21.15.88:0/314240718 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7f80cc0a46f0 0x7f80cc0a4b60 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:38:12.519 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.499+0000 7f80d1292700 1 --2- 172.21.15.88:0/314240718 >> v2:172.21.15.107:3300/0 conn(0x7f80cc133870 0x7f80cc133c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:38:12.519 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.499+0000 7f80d1292700 1 --2- 172.21.15.88:0/314240718 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f80cc0a35f0 0x7f80cc133330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:38:12.519 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.499+0000 7f80d1292700 1 -- 172.21.15.88:0/314240718 >> 172.21.15.88:0/314240718 conn(0x7f80cc06e5e0 msgr2=0x7f80cc06e9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:38:12.519 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.499+0000 7f80d1292700 1 -- 172.21.15.88:0/314240718 shutdown_connections 2023-09-30T01:38:12.519 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.499+0000 7f80d1292700 1 -- 172.21.15.88:0/314240718 wait complete. 2023-09-30T01:38:12.519 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.503+0000 7f80d1292700 10 monclient: build_initial_monmap 2023-09-30T01:38:12.519 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.503+0000 7f80d1292700 10 monclient: monmap: 2023-09-30T01:38:12.519 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2023-09-30T01:38:12.519 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:38:12.519 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 0.000000 2023-09-30T01:38:12.519 INFO:teuthology.orchestra.run.smithi088.stderr:created 0.000000 2023-09-30T01:38:12.519 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2023-09-30T01:38:12.519 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 1 2023-09-30T01:38:12.520 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.noname-a 2023-09-30T01:38:12.520 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.noname-f 2023-09-30T01:38:12.520 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.noname-b 2023-09-30T01:38:12.520 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.noname-g 2023-09-30T01:38:12.520 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.noname-c 2023-09-30T01:38:12.520 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.noname-h 2023-09-30T01:38:12.520 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.noname-d 2023-09-30T01:38:12.520 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.noname-i 2023-09-30T01:38:12.520 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.noname-e 2023-09-30T01:38:12.520 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:38:12.520 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.503+0000 7f80d1292700 1 Processor -- start 2023-09-30T01:38:12.520 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.503+0000 7f80d1292700 1 -- start start 2023-09-30T01:38:12.520 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.503+0000 7f80d1292700 10 monclient: init 2023-09-30T01:38:12.521 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.503+0000 7f80d1292700 10 monclient: _reopen_session rank -1 2023-09-30T01:38:12.521 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.503+0000 7f80d1292700 10 monclient: _add_conns ranks=[4,3,2,7,6,1,0,5,8] 2023-09-30T01:38:12.521 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.503+0000 7f80d1292700 1 --2- >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f80cc0a35f0 0x7f80cc131010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:38:12.521 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.503+0000 7f80d1292700 10 monclient(hunting): picked mon.noname-c con 0x7f80cc0a35f0 addr [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] 2023-09-30T01:38:12.521 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.503+0000 7f80d1292700 1 --2- >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7f80cc0a46f0 0x7f80cc11da00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:38:12.521 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.503+0000 7f80d1292700 10 monclient(hunting): picked mon.noname-g con 0x7f80cc0a46f0 addr [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] 2023-09-30T01:38:12.521 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.503+0000 7f80d1292700 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f80cc133870 0x7f80cc11df40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:38:12.521 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.503+0000 7f80d1292700 10 monclient(hunting): picked mon.noname-b con 0x7f80cc133870 addr [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] 2023-09-30T01:38:12.521 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.503+0000 7f80d1292700 10 monclient(hunting): start opening mon connection 2023-09-30T01:38:12.521 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.503+0000 7f80d1292700 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f80cc11fed0 con 0x7f80cc133870 2023-09-30T01:38:12.521 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.503+0000 7f80d1292700 10 monclient(hunting): start opening mon connection 2023-09-30T01:38:12.521 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.503+0000 7f80d1292700 1 -- --> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f80cc0a0b70 con 0x7f80cc0a46f0 2023-09-30T01:38:12.522 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.503+0000 7f80d1292700 10 monclient(hunting): start opening mon connection 2023-09-30T01:38:12.522 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.503+0000 7f80d1292700 1 -- --> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] -- mon_getmap magic: 0 v1 -- 0x7f80cc097d90 con 0x7f80cc0a35f0 2023-09-30T01:38:12.522 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.503+0000 7f80d1292700 10 monclient(hunting): _renew_subs 2023-09-30T01:38:12.522 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.503+0000 7f80d1292700 10 monclient(hunting): authenticate will time out at 1164.542211s 2023-09-30T01:38:12.522 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.503+0000 7f80caffd700 1 --2- >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f80cc0a35f0 0x7f80cc131010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:38:12.522 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.503+0000 7f80cb7fe700 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f80cc133870 0x7f80cc11df40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:38:12.522 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.503+0000 7f80caffd700 1 --2- >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f80cc0a35f0 0x7f80cc131010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3302/0 says I am v2:172.21.15.88:33956/0 (socket says 172.21.15.88:33956) 2023-09-30T01:38:12.522 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.503+0000 7f80caffd700 1 -- 172.21.15.88:0/462961525 learned_addr learned my addr 172.21.15.88:0/462961525 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-09-30T01:38:12.522 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.503+0000 7f80caffd700 10 monclient(hunting): get_auth_request con 0x7f80cc0a35f0 auth_method 0 2023-09-30T01:38:12.522 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.503+0000 7f80caffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:38:12.522 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.503+0000 7f80caffd700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:38:12.522 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.503+0000 7f80caffd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:38:12.523 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.503+0000 7f80cb7fe700 10 monclient(hunting): get_auth_request con 0x7f80cc133870 auth_method 0 2023-09-30T01:38:12.523 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.503+0000 7f80cb7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:38:12.523 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.503+0000 7f80cb7fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:38:12.523 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.503+0000 7f80cb7fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:38:12.523 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.503+0000 7f80caffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:38:12.523 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.503+0000 7f80caffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:38:12.523 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.503+0000 7f80caffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:38:12.523 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.503+0000 7f80cb7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:38:12.523 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.503+0000 7f80cb7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:38:12.523 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.503+0000 7f80cb7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:38:12.523 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.503+0000 7f80ca7fc700 1 --2- 172.21.15.88:0/462961525 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7f80cc0a46f0 0x7f80cc11da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:38:12.523 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.503+0000 7f80ca7fc700 10 monclient(hunting): get_auth_request con 0x7f80cc0a46f0 auth_method 0 2023-09-30T01:38:12.524 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.503+0000 7f80ca7fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:38:12.524 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.503+0000 7f80ca7fc700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:38:12.524 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.503+0000 7f80ca7fc700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:38:12.524 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.503+0000 7f80ca7fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:38:12.524 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.503+0000 7f80ca7fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:38:12.524 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.503+0000 7f80ca7fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:38:12.524 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.503+0000 7f80cb7fe700 10 monclient(hunting): handle_auth_done global_id 4871 payload 824 2023-09-30T01:38:12.524 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.503+0000 7f80cb7fe700 1 -- 172.21.15.88:0/462961525 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f80cc0a35f0 msgr2=0x7f80cc131010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:38:12.524 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.503+0000 7f80cb7fe700 1 --2- 172.21.15.88:0/462961525 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f80cc0a35f0 0x7f80cc131010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:38:12.524 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.503+0000 7f80cb7fe700 1 -- 172.21.15.88:0/462961525 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7f80cc0a46f0 msgr2=0x7f80cc11da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:38:12.524 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.503+0000 7f80cb7fe700 1 --2- 172.21.15.88:0/462961525 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7f80cc0a46f0 0x7f80cc11da00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:38:12.524 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.503+0000 7f80cb7fe700 10 monclient: _finish_hunting 0 2023-09-30T01:38:12.524 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.503+0000 7f80cb7fe700 1 monclient: found mon.noname-b 2023-09-30T01:38:12.525 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.503+0000 7f80cb7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:38:12.525 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.503+0000 7f80cb7fe700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.88:3301/0 2023-09-30T01:38:12.525 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.503+0000 7f80cb7fe700 1 -- 172.21.15.88:0/462961525 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80c4009a20 con 0x7f80cc133870 2023-09-30T01:38:12.525 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.503+0000 7f80caffd700 1 --2- 172.21.15.88:0/462961525 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f80cc0a35f0 0x7f80cc131010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:38:12.525 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.503+0000 7f80cb7fe700 1 --2- 172.21.15.88:0/462961525 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f80cc133870 0x7f80cc11df40 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f80cc0a5b50 tx=0x7f80bc017000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:38:12.525 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.503+0000 7f80b2ffd700 1 -- 172.21.15.88:0/462961525 <== mon.2 v2:172.21.15.88:3301/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f80bc006b30 con 0x7f80cc133870 2023-09-30T01:38:12.525 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.503+0000 7f80b2ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:38:12.525 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.503+0000 7f80b2ffd700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T01:38:12.525 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.503+0000 7f80b2ffd700 10 monclient: dump: 2023-09-30T01:38:12.525 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:38:12.525 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:38:12.525 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:38:12.526 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:38:12.526 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:38:12.526 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:38:12.526 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:38:12.526 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:38:12.526 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:38:12.526 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:38:12.526 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:38:12.526 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:38:12.526 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:38:12.526 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:38:12.526 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:38:12.526 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:38:12.527 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.503+0000 7f80b2ffd700 10 monclient: _finish_auth 0 2023-09-30T01:38:12.527 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.503+0000 7f80b2ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:38:12.527 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.503+0000 7f80b2ffd700 1 -- 172.21.15.88:0/462961525 <== mon.2 v2:172.21.15.88:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f80bc006e50 con 0x7f80cc133870 2023-09-30T01:38:12.527 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.503+0000 7f80d1292700 5 monclient: authenticate success, global_id 4871 2023-09-30T01:38:12.527 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.503+0000 7f80d1292700 10 monclient: _renew_subs 2023-09-30T01:38:12.527 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.503+0000 7f80d1292700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.88:3301/0 2023-09-30T01:38:12.527 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.503+0000 7f80d1292700 1 -- 172.21.15.88:0/462961525 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f80bc0094f0 con 0x7f80cc133870 2023-09-30T01:38:12.527 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.503+0000 7f80b2ffd700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:38:12.527 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.503+0000 7f80b2ffd700 1 -- 172.21.15.88:0/462961525 <== mon.2 v2:172.21.15.88:3301/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f80bc0118a0 con 0x7f80cc133870 2023-09-30T01:38:12.527 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.503+0000 7f80b2ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:38:12.527 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.503+0000 7f80b2ffd700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T01:38:12.527 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.503+0000 7f80b2ffd700 10 monclient: dump: 2023-09-30T01:38:12.528 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:38:12.528 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:38:12.528 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:38:12.528 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:38:12.528 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:38:12.528 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:38:12.528 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:38:12.528 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:38:12.528 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:38:12.528 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:38:12.528 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:38:12.528 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:38:12.528 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:38:12.529 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:38:12.529 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:38:12.529 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:38:12.529 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.503+0000 7f80d1292700 10 monclient: _renew_subs 2023-09-30T01:38:12.529 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.503+0000 7f80d1292700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.88:3301/0 2023-09-30T01:38:12.529 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.503+0000 7f80d1292700 1 -- 172.21.15.88:0/462961525 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f80cc121160 con 0x7f80cc133870 2023-09-30T01:38:12.529 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.503+0000 7f80c97fa700 4 set_mon_vals no callback set 2023-09-30T01:38:12.529 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.507+0000 7f80b2ffd700 1 -- 172.21.15.88:0/462961525 <== mon.2 v2:172.21.15.88:3301/0 4 ==== mgrmap(e 5) v1 ==== 40144+0+0 (secure 0 0 0) 0x7f80bc011420 con 0x7f80cc133870 2023-09-30T01:38:12.529 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.507+0000 7f80b2ffd700 1 --2- 172.21.15.88:0/462961525 >> v2:172.21.15.107:6812/29926 conn(0x7f80b8033360 0x7f80b8035810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:38:12.529 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.507+0000 7f80b2ffd700 1 -- 172.21.15.88:0/462961525 <== mon.2 v2:172.21.15.88:3301/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 3502+0+0 (secure 0 0 0) 0x7f80bc044a80 con 0x7f80cc133870 2023-09-30T01:38:12.529 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.507+0000 7f80d1292700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:38:12.529 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.507+0000 7f80d1292700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:38:12.529 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.507+0000 7f80d1292700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.88:3301/0 2023-09-30T01:38:12.530 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.507+0000 7f80d1292700 1 -- 172.21.15.88:0/462961525 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f80cc0710d0 con 0x7f80cc133870 2023-09-30T01:38:12.530 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.507+0000 7f80caffd700 1 --2- 172.21.15.88:0/462961525 >> v2:172.21.15.107:6812/29926 conn(0x7f80b8033360 0x7f80b8035810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:38:12.530 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.507+0000 7f80caffd700 10 monclient: get_auth_request con 0x7f80b8033360 auth_method 0 2023-09-30T01:38:12.530 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.507+0000 7f80caffd700 1 --2- 172.21.15.88:0/462961525 >> v2:172.21.15.107:6812/29926 conn(0x7f80b8033360 0x7f80b8035810 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f80c0009fd0 tx=0x7f80c00091a0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:38:12.531 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.511+0000 7f80b2ffd700 1 -- 172.21.15.88:0/462961525 <== mon.2 v2:172.21.15.88:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f80bc015070 con 0x7f80cc133870 2023-09-30T01:38:12.531 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.515+0000 7f80b2ffd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:38:12.531 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.515+0000 7f80b2ffd700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:38:12.685 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.683+0000 7f80d1292700 1 -- 172.21.15.88:0/462961525 --> v2:172.21.15.107:6812/29926 -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "1", "target": ["mon-mgr", ""]}) v1 -- 0x7f80cc064c40 con 0x7f80b8033360 2023-09-30T01:38:12.685 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.683+0000 7f80b2ffd700 1 -- 172.21.15.88:0/462961525 <== mgr.4108 v2:172.21.15.107:6812/29926 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 1 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f80c0011de0 con 0x7f80b8033360 2023-09-30T01:38:12.687 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.683+0000 7f80d1292700 1 -- 172.21.15.88:0/462961525 >> v2:172.21.15.107:6812/29926 conn(0x7f80b8033360 msgr2=0x7f80b8035810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:38:12.688 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.683+0000 7f80d1292700 1 --2- 172.21.15.88:0/462961525 >> v2:172.21.15.107:6812/29926 conn(0x7f80b8033360 0x7f80b8035810 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f80c0009fd0 tx=0x7f80c00091a0 comp rx=0 tx=0).stop 2023-09-30T01:38:12.688 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.683+0000 7f80d1292700 10 monclient: shutdown 2023-09-30T01:38:12.688 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.683+0000 7f80d1292700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:38:12.688 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.683+0000 7f80d1292700 1 -- 172.21.15.88:0/462961525 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f80cc133870 msgr2=0x7f80cc11df40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:38:12.688 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.683+0000 7f80d1292700 1 --2- 172.21.15.88:0/462961525 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f80cc133870 0x7f80cc11df40 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f80cc0a5b50 tx=0x7f80bc017000 comp rx=0 tx=0).stop 2023-09-30T01:38:12.688 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.687+0000 7f80d1292700 1 -- 172.21.15.88:0/462961525 shutdown_connections 2023-09-30T01:38:12.688 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.687+0000 7f80d1292700 1 --2- 172.21.15.88:0/462961525 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f80cc133870 0x7f80cc11df40 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:38:12.688 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.687+0000 7f80d1292700 1 --2- 172.21.15.88:0/462961525 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7f80cc0a46f0 0x7f80cc11da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:38:12.688 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.687+0000 7f80d1292700 1 --2- 172.21.15.88:0/462961525 >> v2:172.21.15.107:6812/29926 conn(0x7f80b8033360 0x7f80b8035810 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:38:12.688 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.687+0000 7f80d1292700 1 --2- 172.21.15.88:0/462961525 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f80cc0a35f0 0x7f80cc131010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:38:12.689 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.687+0000 7f80d1292700 1 -- 172.21.15.88:0/462961525 >> 172.21.15.88:0/462961525 conn(0x7f80cc06e5e0 msgr2=0x7f80cc0a1550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:38:12.689 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.687+0000 7f80d1292700 1 -- 172.21.15.88:0/462961525 shutdown_connections 2023-09-30T01:38:12.689 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.687+0000 7f80d1292700 1 -- 172.21.15.88:0/462961525 wait complete. 2023-09-30T01:38:12.689 INFO:teuthology.orchestra.run.smithi088.stderr:instructed osd(s) 1 to deep-scrub 2023-09-30T01:38:12.700 INFO:tasks.ceph:Scrubbing osd.2 2023-09-30T01:38:12.700 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2023-09-30T01:38:12.817 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.815+0000 7feed7471700 10 monclient: get_monmap_and_config 2023-09-30T01:38:12.817 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.815+0000 7feed7471700 10 monclient: build_initial_monmap 2023-09-30T01:38:12.817 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.815+0000 7feed7471700 10 monclient: monmap: 2023-09-30T01:38:12.817 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2023-09-30T01:38:12.817 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:38:12.817 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:38:12.816696+0000 2023-09-30T01:38:12.817 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:38:12.816696+0000 2023-09-30T01:38:12.817 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2023-09-30T01:38:12.817 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:38:12.818 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a-legacy 2023-09-30T01:38:12.818 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.107:6789/0 mon.noname-f-legacy 2023-09-30T01:38:12.818 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.noname-b 2023-09-30T01:38:12.818 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.noname-g 2023-09-30T01:38:12.818 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.noname-c 2023-09-30T01:38:12.818 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.noname-h 2023-09-30T01:38:12.818 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.noname-d 2023-09-30T01:38:12.818 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.noname-i 2023-09-30T01:38:12.818 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.noname-e 2023-09-30T01:38:12.818 INFO:teuthology.orchestra.run.smithi088.stderr:9: v2:172.21.15.88:3300/0 mon.noname-a 2023-09-30T01:38:12.818 INFO:teuthology.orchestra.run.smithi088.stderr:10: v2:172.21.15.107:3300/0 mon.noname-f 2023-09-30T01:38:12.818 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:38:12.828 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.815+0000 7feed7471700 1 Processor -- start 2023-09-30T01:38:12.828 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.815+0000 7feed7471700 1 -- start start 2023-09-30T01:38:12.828 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.815+0000 7feed7471700 10 monclient: init 2023-09-30T01:38:12.828 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.815+0000 7feed7471700 10 monclient: _reopen_session rank -1 2023-09-30T01:38:12.829 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.815+0000 7feed7471700 10 monclient: _add_conns ranks=[1,10,3,6,8,9,2,4,5,0,7] 2023-09-30T01:38:12.829 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.815+0000 7feed7471700 10 monclient(hunting): picked mon.noname-f-legacy con 0x7feed0097a40 addr v1:172.21.15.107:6789/0 2023-09-30T01:38:12.829 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.815+0000 7feed7471700 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7feed0131210 0x7feed0138630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:38:12.829 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.815+0000 7feed7471700 10 monclient(hunting): picked mon.noname-f con 0x7feed0131210 addr v2:172.21.15.107:3300/0 2023-09-30T01:38:12.829 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.815+0000 7feed7471700 1 --2- >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7feed0138b70 0x7feed013af50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:38:12.829 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.815+0000 7feed7471700 10 monclient(hunting): picked mon.noname-g con 0x7feed0138b70 addr [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] 2023-09-30T01:38:12.829 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.815+0000 7feed7471700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feed004bb20 con 0x7feed0097a40 2023-09-30T01:38:12.829 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.815+0000 7feed7471700 10 monclient(hunting): start opening mon connection 2023-09-30T01:38:12.829 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.815+0000 7feed7471700 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feed0131620 con 0x7feed0131210 2023-09-30T01:38:12.829 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.815+0000 7feed7471700 10 monclient(hunting): start opening mon connection 2023-09-30T01:38:12.829 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.815+0000 7feed7471700 1 -- --> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feed00a3620 con 0x7feed0138b70 2023-09-30T01:38:12.830 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.815+0000 7feed7471700 10 monclient(hunting): _renew_subs 2023-09-30T01:38:12.830 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.815+0000 7feed7471700 10 monclient(hunting): authenticate will time out at 1164.854380s 2023-09-30T01:38:12.830 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.815+0000 7feed6c70700 1 --2- >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7feed0138b70 0x7feed013af50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:38:12.830 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.815+0000 7feed5c6e700 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7feed0131210 0x7feed0138630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:38:12.830 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.815+0000 7feed646f700 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7feed0097a40 0x7feed01327b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.88:37714/0 (socket says 172.21.15.88:37714) 2023-09-30T01:38:12.830 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.815+0000 7feed646f700 1 -- 172.21.15.88:0/3264955554 learned_addr learned my addr 172.21.15.88:0/3264955554 (peer_addr_for_me v1:172.21.15.88:0/0) 2023-09-30T01:38:12.830 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.815+0000 7feed6c70700 10 monclient(hunting): get_auth_request con 0x7feed0138b70 auth_method 0 2023-09-30T01:38:12.830 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.815+0000 7feed6c70700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:38:12.830 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.815+0000 7feed6c70700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:38:12.830 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.815+0000 7feed6c70700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:38:12.830 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.815+0000 7feed5c6e700 10 monclient(hunting): get_auth_request con 0x7feed0131210 auth_method 0 2023-09-30T01:38:12.830 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.815+0000 7feed5c6e700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:38:12.831 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.815+0000 7feed5c6e700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:38:12.831 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.815+0000 7feed5c6e700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:38:12.831 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.815+0000 7feed6c70700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:38:12.831 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.815+0000 7feed6c70700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:38:12.831 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.815+0000 7feed6c70700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:38:12.831 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.815+0000 7feed5c6e700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:38:12.831 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.815+0000 7feed5c6e700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:38:12.831 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.815+0000 7feed5c6e700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:38:12.831 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.815+0000 7feed6c70700 10 monclient(hunting): handle_auth_done global_id 4764 payload 274 2023-09-30T01:38:12.831 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.815+0000 7feed6c70700 1 -- 172.21.15.88:0/3264955554 >> v2:172.21.15.107:3300/0 conn(0x7feed0131210 msgr2=0x7feed0138630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:38:12.831 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.815+0000 7feed6c70700 1 --2- 172.21.15.88:0/3264955554 >> v2:172.21.15.107:3300/0 conn(0x7feed0131210 0x7feed0138630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:38:12.831 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.815+0000 7feed6c70700 1 -- 172.21.15.88:0/3264955554 >> v1:172.21.15.107:6789/0 conn(0x7feed0097a40 legacy=0x7feed01327b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:38:12.832 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.815+0000 7feed6c70700 10 monclient: _finish_hunting 0 2023-09-30T01:38:12.832 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.815+0000 7feed6c70700 1 monclient: found mon.noname-g 2023-09-30T01:38:12.832 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.815+0000 7feed6c70700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:38:12.832 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.815+0000 7feed6c70700 10 monclient: _send_mon_message to mon.noname-g at v2:172.21.15.107:3301/0 2023-09-30T01:38:12.832 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.815+0000 7feed6c70700 1 -- 172.21.15.88:0/3264955554 --> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feed0097e10 con 0x7feed0138b70 2023-09-30T01:38:12.832 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.815+0000 7feed5c6e700 1 --2- 172.21.15.88:0/3264955554 >> v2:172.21.15.107:3300/0 conn(0x7feed0131210 0x7feed0138630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:38:12.832 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.815+0000 7feed6c70700 1 --2- 172.21.15.88:0/3264955554 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7feed0138b70 0x7feed013af50 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7feec8009e90 tx=0x7feec8009780 comp rx=0 tx=0).ready entity=mon.3 client_cookie=1ab9a84b3ac9a564 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:38:12.832 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.819+0000 7feed546d700 1 -- 172.21.15.88:0/3264955554 <== mon.3 v2:172.21.15.107:3301/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7feec800e070 con 0x7feed0138b70 2023-09-30T01:38:12.832 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.819+0000 7feed546d700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:38:12.832 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.819+0000 7feed546d700 10 monclient: got monmap 1 from mon.noname-g (according to old e1) 2023-09-30T01:38:12.832 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.819+0000 7feed546d700 10 monclient: dump: 2023-09-30T01:38:12.832 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:38:12.833 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:38:12.833 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:38:12.833 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:38:12.833 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:38:12.833 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:38:12.833 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:38:12.833 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:38:12.833 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:38:12.833 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:38:12.833 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:38:12.833 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:38:12.833 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:38:12.834 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:38:12.834 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:38:12.834 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:38:12.834 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.819+0000 7feed546d700 10 monclient: _finish_auth 0 2023-09-30T01:38:12.834 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.819+0000 7feed546d700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:38:12.834 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.819+0000 7feed546d700 1 -- 172.21.15.88:0/3264955554 <== mon.3 v2:172.21.15.107:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feec800bbf0 con 0x7feed0138b70 2023-09-30T01:38:12.834 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.819+0000 7feed7471700 5 monclient: authenticate success, global_id 4764 2023-09-30T01:38:12.834 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.819+0000 7feed7471700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:38:12.834 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.819+0000 7feed546d700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:38:12.834 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.819+0000 7feed7471700 10 monclient: get_monmap_and_config success 2023-09-30T01:38:12.834 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.819+0000 7feed7471700 4 set_mon_vals no callback set 2023-09-30T01:38:12.835 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.819+0000 7feed546d700 1 -- 172.21.15.88:0/3264955554 <== mon.3 v2:172.21.15.107:3301/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7feec800bd60 con 0x7feed0138b70 2023-09-30T01:38:12.835 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.819+0000 7feed546d700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:38:12.835 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.819+0000 7feed546d700 10 monclient: got monmap 1 from mon.g (according to old e1) 2023-09-30T01:38:12.835 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.819+0000 7feed546d700 10 monclient: dump: 2023-09-30T01:38:12.835 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:38:12.835 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:38:12.835 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:38:12.835 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:38:12.835 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:38:12.835 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:38:12.836 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:38:12.836 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:38:12.836 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:38:12.836 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:38:12.836 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:38:12.836 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:38:12.836 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:38:12.836 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:38:12.836 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:38:12.836 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:38:12.836 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.819+0000 7feed7471700 10 monclient: shutdown 2023-09-30T01:38:12.836 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.819+0000 7feed7471700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:38:12.837 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.819+0000 7feed7471700 1 -- 172.21.15.88:0/3264955554 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7feed0138b70 msgr2=0x7feed013af50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:38:12.837 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.819+0000 7feed7471700 1 --2- 172.21.15.88:0/3264955554 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7feed0138b70 0x7feed013af50 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7feec8009e90 tx=0x7feec8009780 comp rx=0 tx=0).stop 2023-09-30T01:38:12.837 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.819+0000 7feed7471700 1 -- 172.21.15.88:0/3264955554 shutdown_connections 2023-09-30T01:38:12.837 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.819+0000 7feed7471700 1 --2- 172.21.15.88:0/3264955554 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7feed0138b70 0x7feed013af50 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:38:12.837 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.819+0000 7feed7471700 1 --2- 172.21.15.88:0/3264955554 >> v2:172.21.15.107:3300/0 conn(0x7feed0131210 0x7feed0138630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:38:12.837 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.819+0000 7feed7471700 1 -- 172.21.15.88:0/3264955554 >> 172.21.15.88:0/3264955554 conn(0x7feed006e5e0 msgr2=0x7feed006e9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:38:12.837 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.819+0000 7feed7471700 1 -- 172.21.15.88:0/3264955554 shutdown_connections 2023-09-30T01:38:12.837 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.819+0000 7feed7471700 1 -- 172.21.15.88:0/3264955554 wait complete. 2023-09-30T01:38:12.837 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.819+0000 7feed7471700 10 monclient: build_initial_monmap 2023-09-30T01:38:12.837 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.819+0000 7feed7471700 10 monclient: monmap: 2023-09-30T01:38:12.837 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2023-09-30T01:38:12.837 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:38:12.837 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 0.000000 2023-09-30T01:38:12.838 INFO:teuthology.orchestra.run.smithi088.stderr:created 0.000000 2023-09-30T01:38:12.838 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2023-09-30T01:38:12.838 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 1 2023-09-30T01:38:12.838 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.noname-a 2023-09-30T01:38:12.838 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.noname-f 2023-09-30T01:38:12.838 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.noname-b 2023-09-30T01:38:12.838 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.noname-g 2023-09-30T01:38:12.838 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.noname-c 2023-09-30T01:38:12.838 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.noname-h 2023-09-30T01:38:12.838 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.noname-d 2023-09-30T01:38:12.838 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.noname-i 2023-09-30T01:38:12.839 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.noname-e 2023-09-30T01:38:12.839 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:38:12.839 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.819+0000 7feed7471700 1 Processor -- start 2023-09-30T01:38:12.839 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.819+0000 7feed7471700 1 -- start start 2023-09-30T01:38:12.839 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.819+0000 7feed7471700 10 monclient: init 2023-09-30T01:38:12.839 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.823+0000 7feed7471700 10 monclient: _reopen_session rank -1 2023-09-30T01:38:12.839 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.823+0000 7feed7471700 10 monclient: _add_conns ranks=[7,2,3,8,0,4,5,6,1] 2023-09-30T01:38:12.839 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.823+0000 7feed7471700 1 --2- >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7feed0097a40 0x7feed0142440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:38:12.839 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.823+0000 7feed7471700 10 monclient(hunting): picked mon.noname-i con 0x7feed0097a40 addr [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] 2023-09-30T01:38:12.839 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.823+0000 7feed7471700 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7feed0131210 0x7feed00a26e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:38:12.839 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.823+0000 7feed7471700 10 monclient(hunting): picked mon.noname-b con 0x7feed0131210 addr [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] 2023-09-30T01:38:12.839 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.823+0000 7feed7471700 1 --2- >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7feed0138b70 0x7feed01367b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:38:12.840 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.823+0000 7feed7471700 10 monclient(hunting): picked mon.noname-g con 0x7feed0138b70 addr [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] 2023-09-30T01:38:12.840 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.823+0000 7feed7471700 10 monclient(hunting): start opening mon connection 2023-09-30T01:38:12.840 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.823+0000 7feed7471700 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feed0138300 con 0x7feed0131210 2023-09-30T01:38:12.840 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.823+0000 7feed7471700 10 monclient(hunting): start opening mon connection 2023-09-30T01:38:12.840 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.823+0000 7feed7471700 1 -- --> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feed0099b30 con 0x7feed0138b70 2023-09-30T01:38:12.840 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.823+0000 7feed7471700 10 monclient(hunting): start opening mon connection 2023-09-30T01:38:12.840 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.823+0000 7feed7471700 1 -- --> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] -- mon_getmap magic: 0 v1 -- 0x7feed0099640 con 0x7feed0097a40 2023-09-30T01:38:12.840 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.823+0000 7feed7471700 10 monclient(hunting): _renew_subs 2023-09-30T01:38:12.840 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.823+0000 7feed7471700 10 monclient(hunting): authenticate will time out at 1164.860805s 2023-09-30T01:38:12.840 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.823+0000 7feed5c6e700 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7feed0131210 0x7feed00a26e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:38:12.840 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.823+0000 7feed6c70700 1 --2- >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7feed0138b70 0x7feed01367b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:38:12.840 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.823+0000 7feed5c6e700 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7feed0131210 0x7feed00a26e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3301/0 says I am v2:172.21.15.88:53526/0 (socket says 172.21.15.88:53526) 2023-09-30T01:38:12.840 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.823+0000 7feed5c6e700 1 -- 172.21.15.88:0/2240675744 learned_addr learned my addr 172.21.15.88:0/2240675744 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-09-30T01:38:12.841 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.823+0000 7feed5c6e700 10 monclient(hunting): get_auth_request con 0x7feed0131210 auth_method 0 2023-09-30T01:38:12.841 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.823+0000 7feed5c6e700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:38:12.841 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.823+0000 7feed5c6e700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:38:12.841 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.823+0000 7feed5c6e700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:38:12.841 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.823+0000 7feed6c70700 10 monclient(hunting): get_auth_request con 0x7feed0138b70 auth_method 0 2023-09-30T01:38:12.841 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.823+0000 7feed6c70700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:38:12.841 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.823+0000 7feed6c70700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:38:12.841 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.823+0000 7feed6c70700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:38:12.841 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.823+0000 7feed646f700 1 --2- 172.21.15.88:0/2240675744 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7feed0097a40 0x7feed0142440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:38:12.841 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.823+0000 7feed646f700 10 monclient(hunting): get_auth_request con 0x7feed0097a40 auth_method 0 2023-09-30T01:38:12.841 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.823+0000 7feed646f700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:38:12.842 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.823+0000 7feed646f700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:38:12.842 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.823+0000 7feed646f700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:38:12.842 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.823+0000 7feed5c6e700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:38:12.842 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.823+0000 7feed5c6e700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:38:12.842 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.823+0000 7feed5c6e700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:38:12.842 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.823+0000 7feed6c70700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:38:12.842 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.823+0000 7feed6c70700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:38:12.842 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.823+0000 7feed6c70700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:38:12.842 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.823+0000 7feed646f700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:38:12.842 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.823+0000 7feed646f700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:38:12.842 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.823+0000 7feed646f700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:38:12.842 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.823+0000 7feed6c70700 10 monclient(hunting): handle_auth_done global_id 4773 payload 824 2023-09-30T01:38:12.843 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.823+0000 7feed6c70700 1 -- 172.21.15.88:0/2240675744 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7feed0097a40 msgr2=0x7feed0142440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:38:12.843 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.823+0000 7feed6c70700 1 --2- 172.21.15.88:0/2240675744 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7feed0097a40 0x7feed0142440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:38:12.843 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.823+0000 7feed6c70700 1 -- 172.21.15.88:0/2240675744 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7feed0131210 msgr2=0x7feed00a26e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:38:12.843 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.823+0000 7feed6c70700 1 --2- 172.21.15.88:0/2240675744 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7feed0131210 0x7feed00a26e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:38:12.843 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.823+0000 7feed6c70700 10 monclient: _finish_hunting 0 2023-09-30T01:38:12.843 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.823+0000 7feed6c70700 1 monclient: found mon.noname-g 2023-09-30T01:38:12.843 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.823+0000 7feed6c70700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:38:12.843 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.823+0000 7feed6c70700 10 monclient: _send_mon_message to mon.noname-g at v2:172.21.15.107:3301/0 2023-09-30T01:38:12.843 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.823+0000 7feed6c70700 1 -- 172.21.15.88:0/2240675744 --> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feecc009420 con 0x7feed0138b70 2023-09-30T01:38:12.843 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.823+0000 7feed6c70700 1 --2- 172.21.15.88:0/2240675744 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7feed0138b70 0x7feed01367b0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7feec8016000 tx=0x7feec8019000 comp rx=0 tx=0).ready entity=mon.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:38:12.843 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.823+0000 7feec67fc700 1 -- 172.21.15.88:0/2240675744 <== mon.3 v2:172.21.15.107:3301/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7feec8007000 con 0x7feed0138b70 2023-09-30T01:38:12.843 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.823+0000 7feec67fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:38:12.844 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.823+0000 7feec67fc700 10 monclient: got monmap 1 from mon.noname-g (according to old e1) 2023-09-30T01:38:12.844 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.823+0000 7feec67fc700 10 monclient: dump: 2023-09-30T01:38:12.844 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:38:12.844 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:38:12.844 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:38:12.844 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:38:12.844 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:38:12.844 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:38:12.844 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:38:12.844 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:38:12.844 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:38:12.844 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:38:12.845 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:38:12.845 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:38:12.845 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:38:12.845 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:38:12.845 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:38:12.845 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:38:12.845 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.823+0000 7feec67fc700 10 monclient: _finish_auth 0 2023-09-30T01:38:12.845 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.823+0000 7feec67fc700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:38:12.845 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.823+0000 7feed7471700 5 monclient: authenticate success, global_id 4773 2023-09-30T01:38:12.845 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.823+0000 7feec67fc700 1 -- 172.21.15.88:0/2240675744 <== mon.3 v2:172.21.15.107:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feec8021070 con 0x7feed0138b70 2023-09-30T01:38:12.845 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.823+0000 7feed7471700 10 monclient: _renew_subs 2023-09-30T01:38:12.845 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.823+0000 7feed7471700 10 monclient: _send_mon_message to mon.g at v2:172.21.15.107:3301/0 2023-09-30T01:38:12.846 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.823+0000 7feed7471700 1 -- 172.21.15.88:0/2240675744 --> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feec8009a20 con 0x7feed0138b70 2023-09-30T01:38:12.846 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.823+0000 7feec67fc700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:38:12.846 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.823+0000 7feec67fc700 1 -- 172.21.15.88:0/2240675744 <== mon.3 v2:172.21.15.107:3301/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7feec801c890 con 0x7feed0138b70 2023-09-30T01:38:12.846 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.823+0000 7feec67fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:38:12.846 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.823+0000 7feec67fc700 10 monclient: got monmap 1 from mon.g (according to old e1) 2023-09-30T01:38:12.846 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.823+0000 7feed4c6c700 4 set_mon_vals no callback set 2023-09-30T01:38:12.846 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.823+0000 7feec67fc700 10 monclient: dump: 2023-09-30T01:38:12.846 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:38:12.846 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:38:12.846 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:38:12.846 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:38:12.846 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:38:12.847 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:38:12.847 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:38:12.847 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:38:12.847 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:38:12.847 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:38:12.847 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:38:12.847 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:38:12.847 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:38:12.847 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:38:12.847 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:38:12.847 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:38:12.847 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.823+0000 7feed7471700 10 monclient: _renew_subs 2023-09-30T01:38:12.847 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.823+0000 7feed7471700 10 monclient: _send_mon_message to mon.g at v2:172.21.15.107:3301/0 2023-09-30T01:38:12.848 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.823+0000 7feed7471700 1 -- 172.21.15.88:0/2240675744 --> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feed0143190 con 0x7feed0138b70 2023-09-30T01:38:12.848 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.823+0000 7feec67fc700 1 -- 172.21.15.88:0/2240675744 <== mon.3 v2:172.21.15.107:3301/0 4 ==== mgrmap(e 5) v1 ==== 40144+0+0 (secure 0 0 0) 0x7feec8023030 con 0x7feed0138b70 2023-09-30T01:38:12.848 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.827+0000 7feec67fc700 1 --2- 172.21.15.88:0/2240675744 >> v2:172.21.15.107:6812/29926 conn(0x7feeb4033360 0x7feeb4035810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:38:12.848 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.827+0000 7feec67fc700 1 -- 172.21.15.88:0/2240675744 <== mon.3 v2:172.21.15.107:3301/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 3502+0+0 (secure 0 0 0) 0x7feec8045300 con 0x7feed0138b70 2023-09-30T01:38:12.848 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.827+0000 7feed7471700 1 --2- 172.21.15.88:0/2240675744 >> v2:172.21.15.88:6800/30176 conn(0x7feed00a2c20 0x7feed0136cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:38:12.848 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.827+0000 7feed7471700 1 -- 172.21.15.88:0/2240675744 --> v2:172.21.15.88:6800/30176 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7feed009ccd0 con 0x7feed00a2c20 2023-09-30T01:38:12.848 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.827+0000 7feed646f700 1 --2- 172.21.15.88:0/2240675744 >> v2:172.21.15.107:6812/29926 conn(0x7feeb4033360 0x7feeb4035810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:38:12.848 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.827+0000 7feed646f700 10 monclient: get_auth_request con 0x7feeb4033360 auth_method 0 2023-09-30T01:38:12.848 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.827+0000 7feed5c6e700 1 --2- 172.21.15.88:0/2240675744 >> v2:172.21.15.88:6800/30176 conn(0x7feed00a2c20 0x7feed0136cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:38:12.849 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.827+0000 7feed5c6e700 10 monclient: get_auth_request con 0x7feed00a2c20 auth_method 0 2023-09-30T01:38:12.850 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.827+0000 7feed646f700 1 --2- 172.21.15.88:0/2240675744 >> v2:172.21.15.107:6812/29926 conn(0x7feeb4033360 0x7feeb4035810 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7feec0007130 tx=0x7feec0000bc0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:38:12.850 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.827+0000 7feed5c6e700 1 --2- 172.21.15.88:0/2240675744 >> v2:172.21.15.88:6800/30176 conn(0x7feed00a2c20 0x7feed0136cf0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:38:12.850 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.827+0000 7feec67fc700 1 -- 172.21.15.88:0/2240675744 <== osd.2 v2:172.21.15.88:6800/30176 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23999 (crc 0 0 0) 0x7feed009ccd0 con 0x7feed00a2c20 2023-09-30T01:38:12.850 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.843+0000 7feed7471700 1 -- 172.21.15.88:0/2240675744 --> v2:172.21.15.88:6800/30176 -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7feed0071960 con 0x7feed00a2c20 2023-09-30T01:38:12.850 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.847+0000 7feec67fc700 1 -- 172.21.15.88:0/2240675744 <== osd.2 v2:172.21.15.88:6800/30176 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7feed0071960 con 0x7feed00a2c20 2023-09-30T01:38:12.850 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-09-30T01:38:12.850 INFO:teuthology.orchestra.run.smithi088.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-09-30T01:38:12.851 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-09-30T01:38:12.852 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.847+0000 7feed7471700 1 -- 172.21.15.88:0/2240675744 >> v2:172.21.15.88:6800/30176 conn(0x7feed00a2c20 msgr2=0x7feed0136cf0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:38:12.852 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.847+0000 7feed7471700 1 --2- 172.21.15.88:0/2240675744 >> v2:172.21.15.88:6800/30176 conn(0x7feed00a2c20 0x7feed0136cf0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:38:12.852 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.847+0000 7feed7471700 1 -- 172.21.15.88:0/2240675744 >> v2:172.21.15.107:6812/29926 conn(0x7feeb4033360 msgr2=0x7feeb4035810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:38:12.852 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.847+0000 7feed7471700 1 --2- 172.21.15.88:0/2240675744 >> v2:172.21.15.107:6812/29926 conn(0x7feeb4033360 0x7feeb4035810 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7feec0007130 tx=0x7feec0000bc0 comp rx=0 tx=0).stop 2023-09-30T01:38:12.852 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.847+0000 7feed7471700 10 monclient: shutdown 2023-09-30T01:38:12.852 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.847+0000 7feed7471700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:38:12.852 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.847+0000 7feed7471700 1 -- 172.21.15.88:0/2240675744 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7feed0138b70 msgr2=0x7feed01367b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:38:12.852 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.847+0000 7feed7471700 1 --2- 172.21.15.88:0/2240675744 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7feed0138b70 0x7feed01367b0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7feec8016000 tx=0x7feec8019000 comp rx=0 tx=0).stop 2023-09-30T01:38:12.852 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.847+0000 7feed6c70700 1 -- 172.21.15.88:0/2240675744 reap_dead start 2023-09-30T01:38:12.852 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.847+0000 7feed7471700 1 -- 172.21.15.88:0/2240675744 shutdown_connections 2023-09-30T01:38:12.852 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.847+0000 7feed7471700 1 -- 172.21.15.88:0/2240675744 >> 172.21.15.88:0/2240675744 conn(0x7feed006e5e0 msgr2=0x7feed00a53f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:38:12.853 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.847+0000 7feed7471700 1 -- 172.21.15.88:0/2240675744 shutdown_connections 2023-09-30T01:38:12.853 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.847+0000 7feed7471700 1 -- 172.21.15.88:0/2240675744 wait complete. 2023-09-30T01:38:12.860 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2023-09-30T01:38:12.969 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.967+0000 7f04e8f87700 10 monclient: get_monmap_and_config 2023-09-30T01:38:12.970 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.967+0000 7f04e8f87700 10 monclient: build_initial_monmap 2023-09-30T01:38:12.970 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.967+0000 7f04e8f87700 10 monclient: monmap: 2023-09-30T01:38:12.970 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2023-09-30T01:38:12.970 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:38:12.970 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:38:12.969554+0000 2023-09-30T01:38:12.970 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:38:12.969554+0000 2023-09-30T01:38:12.970 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2023-09-30T01:38:12.970 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:38:12.970 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a-legacy 2023-09-30T01:38:12.970 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.107:6789/0 mon.noname-f-legacy 2023-09-30T01:38:12.971 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.noname-b 2023-09-30T01:38:12.971 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.noname-g 2023-09-30T01:38:12.971 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.noname-c 2023-09-30T01:38:12.971 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.noname-h 2023-09-30T01:38:12.971 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.noname-d 2023-09-30T01:38:12.971 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.noname-i 2023-09-30T01:38:12.971 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.noname-e 2023-09-30T01:38:12.971 INFO:teuthology.orchestra.run.smithi088.stderr:9: v2:172.21.15.88:3300/0 mon.noname-a 2023-09-30T01:38:12.971 INFO:teuthology.orchestra.run.smithi088.stderr:10: v2:172.21.15.107:3300/0 mon.noname-f 2023-09-30T01:38:12.971 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:38:12.978 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.967+0000 7f04e8f87700 1 Processor -- start 2023-09-30T01:38:12.978 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.967+0000 7f04e8f87700 1 -- start start 2023-09-30T01:38:12.978 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.967+0000 7f04e8f87700 10 monclient: init 2023-09-30T01:38:12.978 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.967+0000 7f04e8f87700 10 monclient: _reopen_session rank -1 2023-09-30T01:38:12.978 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.967+0000 7f04e8f87700 10 monclient: _add_conns ranks=[3,8,7,5,4,10,0,6,9,2,1] 2023-09-30T01:38:12.978 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.967+0000 7f04e8f87700 1 --2- >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7f04e40a17d0 0x7f04e4069830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:38:12.978 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.967+0000 7f04e8f87700 10 monclient(hunting): picked mon.noname-g con 0x7f04e40a17d0 addr [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] 2023-09-30T01:38:12.978 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.967+0000 7f04e8f87700 1 --2- >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f04e4130610 0x7f04e4130a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:38:12.978 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.967+0000 7f04e8f87700 10 monclient(hunting): picked mon.noname-e con 0x7f04e4130610 addr [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] 2023-09-30T01:38:12.978 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.967+0000 7f04e8f87700 1 --2- >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7f04e409edf0 0x7f04e409f240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:38:12.979 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.967+0000 7f04e8f87700 10 monclient(hunting): picked mon.noname-i con 0x7f04e409edf0 addr [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] 2023-09-30T01:38:12.979 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.967+0000 7f04e8f87700 10 monclient(hunting): start opening mon connection 2023-09-30T01:38:12.979 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.967+0000 7f04e8f87700 1 -- --> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04e4068980 con 0x7f04e40a17d0 2023-09-30T01:38:12.979 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.967+0000 7f04e8f87700 10 monclient(hunting): start opening mon connection 2023-09-30T01:38:12.979 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.967+0000 7f04e8f87700 1 -- --> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] -- mon_getmap magic: 0 v1 -- 0x7f04e4067fc0 con 0x7f04e409edf0 2023-09-30T01:38:12.979 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.967+0000 7f04e8f87700 10 monclient(hunting): start opening mon connection 2023-09-30T01:38:12.979 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.967+0000 7f04e8f87700 1 -- --> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] -- mon_getmap magic: 0 v1 -- 0x7f04e412e230 con 0x7f04e4130610 2023-09-30T01:38:12.979 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.967+0000 7f04e8f87700 10 monclient(hunting): _renew_subs 2023-09-30T01:38:12.979 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.967+0000 7f04e8f87700 10 monclient(hunting): authenticate will time out at 1165.007297s 2023-09-30T01:38:12.979 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.967+0000 7f04e2ffd700 1 --2- >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f04e4130610 0x7f04e4130a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:38:12.979 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.967+0000 7f04e2ffd700 1 --2- >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f04e4130610 0x7f04e4130a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3304/0 says I am v2:172.21.15.88:35740/0 (socket says 172.21.15.88:35740) 2023-09-30T01:38:12.979 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.967+0000 7f04e2ffd700 1 -- 172.21.15.88:0/3847777618 learned_addr learned my addr 172.21.15.88:0/3847777618 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-09-30T01:38:12.979 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.967+0000 7f04e2ffd700 10 monclient(hunting): get_auth_request con 0x7f04e4130610 auth_method 0 2023-09-30T01:38:12.980 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.967+0000 7f04e2ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:38:12.980 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.967+0000 7f04e2ffd700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:38:12.980 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.967+0000 7f04e2ffd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:38:12.980 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.967+0000 7f04e37fe700 1 --2- 172.21.15.88:0/3847777618 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7f04e40a17d0 0x7f04e4069830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:38:12.980 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.967+0000 7f04e2ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:38:12.980 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.967+0000 7f04e2ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:38:12.980 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.967+0000 7f04e3fff700 1 --2- 172.21.15.88:0/3847777618 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7f04e409edf0 0x7f04e409f240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:38:12.980 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.967+0000 7f04e2ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:38:12.980 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.967+0000 7f04e37fe700 10 monclient(hunting): get_auth_request con 0x7f04e40a17d0 auth_method 0 2023-09-30T01:38:12.980 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.967+0000 7f04e37fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:38:12.980 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.967+0000 7f04e37fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:38:12.981 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.967+0000 7f04e37fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:38:12.981 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.967+0000 7f04e3fff700 10 monclient(hunting): get_auth_request con 0x7f04e409edf0 auth_method 0 2023-09-30T01:38:12.981 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.967+0000 7f04e3fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:38:12.981 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.967+0000 7f04e3fff700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:38:12.981 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.967+0000 7f04e3fff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:38:12.981 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.967+0000 7f04e2ffd700 10 monclient(hunting): handle_auth_done global_id 4922 payload 274 2023-09-30T01:38:12.981 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.967+0000 7f04e2ffd700 1 -- 172.21.15.88:0/3847777618 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7f04e409edf0 msgr2=0x7f04e409f240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:38:12.981 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.967+0000 7f04e2ffd700 1 --2- 172.21.15.88:0/3847777618 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7f04e409edf0 0x7f04e409f240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:38:12.981 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.967+0000 7f04e2ffd700 1 -- 172.21.15.88:0/3847777618 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7f04e40a17d0 msgr2=0x7f04e4069830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:38:12.981 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.967+0000 7f04e2ffd700 1 --2- 172.21.15.88:0/3847777618 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7f04e40a17d0 0x7f04e4069830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:38:12.981 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.967+0000 7f04e2ffd700 10 monclient: _finish_hunting 0 2023-09-30T01:38:12.981 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.967+0000 7f04e2ffd700 1 monclient: found mon.noname-e 2023-09-30T01:38:12.982 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.967+0000 7f04e2ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:38:12.982 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.967+0000 7f04e2ffd700 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.88:3304/0 2023-09-30T01:38:12.982 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.967+0000 7f04e2ffd700 1 -- 172.21.15.88:0/3847777618 --> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04e404bb20 con 0x7f04e4130610 2023-09-30T01:38:12.982 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.967+0000 7f04e37fe700 1 --2- 172.21.15.88:0/3847777618 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7f04e40a17d0 0x7f04e4069830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-30T01:38:12.982 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.967+0000 7f04e3fff700 1 --2- 172.21.15.88:0/3847777618 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7f04e409edf0 0x7f04e409f240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-30T01:38:12.982 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.971+0000 7f04e2ffd700 1 --2- 172.21.15.88:0/3847777618 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f04e4130610 0x7f04e4130a00 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f04dc009f80 tx=0x7f04dc009840 comp rx=0 tx=0).ready entity=mon.8 client_cookie=cb0104dc33cad91 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:38:12.982 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.971+0000 7f04e27fc700 1 -- 172.21.15.88:0/3847777618 <== mon.8 v2:172.21.15.88:3304/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f04dc00e070 con 0x7f04e4130610 2023-09-30T01:38:12.982 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.971+0000 7f04e27fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:38:12.982 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.971+0000 7f04e27fc700 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2023-09-30T01:38:12.982 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.971+0000 7f04e27fc700 10 monclient: dump: 2023-09-30T01:38:12.982 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:38:12.983 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:38:12.983 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:38:12.983 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:38:12.983 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:38:12.983 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:38:12.983 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:38:12.983 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:38:12.983 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:38:12.983 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:38:12.983 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:38:12.983 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:38:12.983 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:38:12.984 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:38:12.984 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:38:12.984 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:38:12.984 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.971+0000 7f04e27fc700 10 monclient: _finish_auth 0 2023-09-30T01:38:12.984 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.971+0000 7f04e27fc700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:38:12.984 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.971+0000 7f04e8f87700 5 monclient: authenticate success, global_id 4922 2023-09-30T01:38:12.984 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.971+0000 7f04e8f87700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:38:12.984 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.971+0000 7f04e27fc700 1 -- 172.21.15.88:0/3847777618 <== mon.8 v2:172.21.15.88:3304/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f04dc007400 con 0x7f04e4130610 2023-09-30T01:38:12.984 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.971+0000 7f04e27fc700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:38:12.984 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.971+0000 7f04e8f87700 10 monclient: get_monmap_and_config success 2023-09-30T01:38:12.984 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.971+0000 7f04e8f87700 4 set_mon_vals no callback set 2023-09-30T01:38:12.984 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.971+0000 7f04e27fc700 1 -- 172.21.15.88:0/3847777618 <== mon.8 v2:172.21.15.88:3304/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f04dc0092e0 con 0x7f04e4130610 2023-09-30T01:38:12.985 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.971+0000 7f04e27fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:38:12.985 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.971+0000 7f04e27fc700 10 monclient: got monmap 1 from mon.e (according to old e1) 2023-09-30T01:38:12.985 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.971+0000 7f04e27fc700 10 monclient: dump: 2023-09-30T01:38:12.985 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:38:12.985 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:38:12.985 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:38:12.985 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:38:12.985 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:38:12.985 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:38:12.985 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:38:12.985 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:38:12.985 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:38:12.986 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:38:12.986 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:38:12.986 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:38:12.986 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:38:12.986 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:38:12.986 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:38:12.986 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:38:12.986 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.971+0000 7f04e8f87700 10 monclient: shutdown 2023-09-30T01:38:12.986 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.971+0000 7f04e8f87700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:38:12.986 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.971+0000 7f04e8f87700 1 -- 172.21.15.88:0/3847777618 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f04e4130610 msgr2=0x7f04e4130a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:38:12.986 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.971+0000 7f04e8f87700 1 --2- 172.21.15.88:0/3847777618 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f04e4130610 0x7f04e4130a00 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f04dc009f80 tx=0x7f04dc009840 comp rx=0 tx=0).stop 2023-09-30T01:38:12.986 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.971+0000 7f04e8f87700 1 -- 172.21.15.88:0/3847777618 shutdown_connections 2023-09-30T01:38:12.987 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.971+0000 7f04e8f87700 1 --2- 172.21.15.88:0/3847777618 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7f04e409edf0 0x7f04e409f240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:38:12.987 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.971+0000 7f04e8f87700 1 --2- 172.21.15.88:0/3847777618 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7f04e4130610 0x7f04e4130a00 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:38:12.987 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.971+0000 7f04e8f87700 1 --2- 172.21.15.88:0/3847777618 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7f04e40a17d0 0x7f04e4069830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:38:12.987 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.971+0000 7f04e8f87700 1 -- 172.21.15.88:0/3847777618 >> 172.21.15.88:0/3847777618 conn(0x7f04e4097a40 msgr2=0x7f04e40a0ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:38:12.987 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.971+0000 7f04e8f87700 1 -- 172.21.15.88:0/3847777618 shutdown_connections 2023-09-30T01:38:12.987 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.971+0000 7f04e8f87700 1 -- 172.21.15.88:0/3847777618 wait complete. 2023-09-30T01:38:12.987 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.971+0000 7f04e8f87700 10 monclient: build_initial_monmap 2023-09-30T01:38:12.987 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.971+0000 7f04e8f87700 10 monclient: monmap: 2023-09-30T01:38:12.987 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2023-09-30T01:38:12.987 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:38:12.987 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 0.000000 2023-09-30T01:38:12.987 INFO:teuthology.orchestra.run.smithi088.stderr:created 0.000000 2023-09-30T01:38:12.988 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2023-09-30T01:38:12.988 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 1 2023-09-30T01:38:12.988 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.noname-a 2023-09-30T01:38:12.988 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.noname-f 2023-09-30T01:38:12.988 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.noname-b 2023-09-30T01:38:12.988 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.noname-g 2023-09-30T01:38:12.988 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.noname-c 2023-09-30T01:38:12.988 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.noname-h 2023-09-30T01:38:12.988 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.noname-d 2023-09-30T01:38:12.988 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.noname-i 2023-09-30T01:38:12.988 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.noname-e 2023-09-30T01:38:12.988 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:38:12.988 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.971+0000 7f04e8f87700 1 Processor -- start 2023-09-30T01:38:12.989 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.971+0000 7f04e8f87700 1 -- start start 2023-09-30T01:38:12.989 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.971+0000 7f04e8f87700 10 monclient: init 2023-09-30T01:38:12.989 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.971+0000 7f04e8f87700 10 monclient: _reopen_session rank -1 2023-09-30T01:38:12.989 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.971+0000 7f04e8f87700 10 monclient: _add_conns ranks=[2,5,4,8,6,0,3,1,7] 2023-09-30T01:38:12.989 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.971+0000 7f04e8f87700 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f04e409edf0 0x7f04e4142370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:38:12.989 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.971+0000 7f04e8f87700 10 monclient(hunting): picked mon.noname-b con 0x7f04e409edf0 addr [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] 2023-09-30T01:38:12.989 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.971+0000 7f04e8f87700 1 --2- >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7f04e4130610 0x7f04e41428b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:38:12.989 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.971+0000 7f04e8f87700 10 monclient(hunting): picked mon.noname-h con 0x7f04e4130610 addr [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] 2023-09-30T01:38:12.989 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.971+0000 7f04e8f87700 1 --2- >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f04e4142df0 0x7f04e409c7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:38:12.989 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.971+0000 7f04e8f87700 10 monclient(hunting): picked mon.noname-c con 0x7f04e4142df0 addr [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] 2023-09-30T01:38:12.989 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.971+0000 7f04e8f87700 10 monclient(hunting): start opening mon connection 2023-09-30T01:38:12.989 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.971+0000 7f04e8f87700 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04e412e020 con 0x7f04e409edf0 2023-09-30T01:38:12.990 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.971+0000 7f04e8f87700 10 monclient(hunting): start opening mon connection 2023-09-30T01:38:12.990 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.971+0000 7f04e8f87700 1 -- --> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] -- mon_getmap magic: 0 v1 -- 0x7f04e4069e90 con 0x7f04e4142df0 2023-09-30T01:38:12.990 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.971+0000 7f04e8f87700 10 monclient(hunting): start opening mon connection 2023-09-30T01:38:12.990 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.971+0000 7f04e8f87700 1 -- --> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] -- mon_getmap magic: 0 v1 -- 0x7f04e411fe30 con 0x7f04e4130610 2023-09-30T01:38:12.990 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.971+0000 7f04e8f87700 10 monclient(hunting): _renew_subs 2023-09-30T01:38:12.990 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.971+0000 7f04e8f87700 10 monclient(hunting): authenticate will time out at 1165.010655s 2023-09-30T01:38:12.990 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.971+0000 7f04e3fff700 1 --2- >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f04e4142df0 0x7f04e409c7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:38:12.990 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.971+0000 7f04e3fff700 1 --2- >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f04e4142df0 0x7f04e409c7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3302/0 says I am v2:172.21.15.88:33958/0 (socket says 172.21.15.88:33958) 2023-09-30T01:38:12.990 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.971+0000 7f04e3fff700 1 -- 172.21.15.88:0/1586743256 learned_addr learned my addr 172.21.15.88:0/1586743256 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-09-30T01:38:12.990 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.971+0000 7f04e3fff700 10 monclient(hunting): get_auth_request con 0x7f04e4142df0 auth_method 0 2023-09-30T01:38:12.990 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.971+0000 7f04e3fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:38:12.991 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.971+0000 7f04e3fff700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:38:12.991 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.971+0000 7f04e3fff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:38:12.991 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.971+0000 7f04e37fe700 1 --2- 172.21.15.88:0/1586743256 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f04e409edf0 0x7f04e4142370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:38:12.991 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.971+0000 7f04e37fe700 10 monclient(hunting): get_auth_request con 0x7f04e409edf0 auth_method 0 2023-09-30T01:38:12.991 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.971+0000 7f04e37fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:38:12.991 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.971+0000 7f04e37fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:38:12.991 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.971+0000 7f04e37fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:38:12.991 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.971+0000 7f04e2ffd700 1 --2- 172.21.15.88:0/1586743256 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7f04e4130610 0x7f04e41428b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:38:12.991 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.971+0000 7f04e3fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:38:12.991 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.971+0000 7f04e3fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:38:12.991 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.971+0000 7f04e3fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:38:12.991 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.971+0000 7f04e2ffd700 10 monclient(hunting): get_auth_request con 0x7f04e4130610 auth_method 0 2023-09-30T01:38:12.992 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.971+0000 7f04e2ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:38:12.992 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.971+0000 7f04e2ffd700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:38:12.992 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.971+0000 7f04e2ffd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:38:12.992 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.971+0000 7f04e37fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:38:12.992 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.971+0000 7f04e37fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:38:12.992 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.971+0000 7f04e37fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:38:12.992 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.971+0000 7f04e2ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:38:12.992 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.971+0000 7f04e2ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:38:12.992 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.971+0000 7f04e2ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:38:12.992 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.971+0000 7f04e3fff700 10 monclient(hunting): handle_auth_done global_id 4972 payload 824 2023-09-30T01:38:12.992 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.971+0000 7f04e3fff700 1 -- 172.21.15.88:0/1586743256 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7f04e4130610 msgr2=0x7f04e41428b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:38:12.992 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.971+0000 7f04e3fff700 1 --2- 172.21.15.88:0/1586743256 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7f04e4130610 0x7f04e41428b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:38:12.993 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.971+0000 7f04e3fff700 1 -- 172.21.15.88:0/1586743256 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f04e409edf0 msgr2=0x7f04e4142370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:38:12.993 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.971+0000 7f04e3fff700 1 --2- 172.21.15.88:0/1586743256 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f04e409edf0 0x7f04e4142370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:38:12.993 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.971+0000 7f04e3fff700 10 monclient: _finish_hunting 0 2023-09-30T01:38:12.993 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.971+0000 7f04e3fff700 1 monclient: found mon.noname-c 2023-09-30T01:38:12.993 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.971+0000 7f04e3fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:38:12.993 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.971+0000 7f04e3fff700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.88:3302/0 2023-09-30T01:38:12.993 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.971+0000 7f04e3fff700 1 -- 172.21.15.88:0/1586743256 --> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04dc009ae0 con 0x7f04e4142df0 2023-09-30T01:38:12.993 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.971+0000 7f04e3fff700 1 --2- 172.21.15.88:0/1586743256 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f04e4142df0 0x7f04e409c7b0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f04d400e8d0 tx=0x7f04d4008040 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:38:12.993 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.971+0000 7f04db7fe700 1 -- 172.21.15.88:0/1586743256 <== mon.4 v2:172.21.15.88:3302/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f04d4009350 con 0x7f04e4142df0 2023-09-30T01:38:12.993 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.971+0000 7f04db7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:38:12.993 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.971+0000 7f04db7fe700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T01:38:12.993 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.971+0000 7f04db7fe700 10 monclient: dump: 2023-09-30T01:38:12.994 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:38:12.994 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:38:12.994 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:38:12.994 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:38:12.994 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:38:12.994 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:38:12.994 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:38:12.994 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:38:12.994 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:38:12.994 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:38:12.994 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:38:12.994 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:38:12.994 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:38:12.995 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:38:12.995 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:38:12.995 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:38:12.995 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.975+0000 7f04db7fe700 10 monclient: _finish_auth 0 2023-09-30T01:38:12.995 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.975+0000 7f04db7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:38:12.995 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.975+0000 7f04e8f87700 5 monclient: authenticate success, global_id 4972 2023-09-30T01:38:12.995 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.975+0000 7f04e8f87700 10 monclient: _renew_subs 2023-09-30T01:38:12.995 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.975+0000 7f04db7fe700 1 -- 172.21.15.88:0/1586743256 <== mon.4 v2:172.21.15.88:3302/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f04d4013890 con 0x7f04e4142df0 2023-09-30T01:38:12.995 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.975+0000 7f04e8f87700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.88:3302/0 2023-09-30T01:38:12.995 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.975+0000 7f04e8f87700 1 -- 172.21.15.88:0/1586743256 --> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f04e4120370 con 0x7f04e4142df0 2023-09-30T01:38:12.995 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.975+0000 7f04db7fe700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:38:12.995 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.975+0000 7f04db7fe700 1 -- 172.21.15.88:0/1586743256 <== mon.4 v2:172.21.15.88:3302/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f04d4013e80 con 0x7f04e4142df0 2023-09-30T01:38:12.996 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.975+0000 7f04db7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:38:12.996 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.975+0000 7f04e1ffb700 4 set_mon_vals no callback set 2023-09-30T01:38:12.996 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.975+0000 7f04db7fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T01:38:12.996 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.975+0000 7f04db7fe700 10 monclient: dump: 2023-09-30T01:38:12.996 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:38:12.996 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:38:12.996 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:38:12.996 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:38:12.996 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:38:12.996 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:38:12.996 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:38:12.996 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:38:12.997 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:38:12.997 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:38:12.997 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:38:12.997 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:38:12.997 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:38:12.997 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:38:12.997 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:38:12.997 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:38:12.997 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.975+0000 7f04e8f87700 10 monclient: _renew_subs 2023-09-30T01:38:12.997 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.975+0000 7f04e8f87700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.88:3302/0 2023-09-30T01:38:12.997 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.975+0000 7f04e8f87700 1 -- 172.21.15.88:0/1586743256 --> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f04e4120090 con 0x7f04e4142df0 2023-09-30T01:38:12.997 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.975+0000 7f04db7fe700 1 -- 172.21.15.88:0/1586743256 <== mon.4 v2:172.21.15.88:3302/0 4 ==== mgrmap(e 5) v1 ==== 40144+0+0 (secure 0 0 0) 0x7f04d4013a00 con 0x7f04e4142df0 2023-09-30T01:38:12.998 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.975+0000 7f04e8f87700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:38:12.998 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.975+0000 7f04e8f87700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:38:12.998 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.975+0000 7f04e8f87700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.88:3302/0 2023-09-30T01:38:12.998 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.975+0000 7f04e8f87700 1 -- 172.21.15.88:0/1586743256 --> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f04e4071960 con 0x7f04e4142df0 2023-09-30T01:38:12.998 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.975+0000 7f04db7fe700 1 --2- 172.21.15.88:0/1586743256 >> v2:172.21.15.107:6812/29926 conn(0x7f04c4033360 0x7f04c4035810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:38:12.998 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.975+0000 7f04db7fe700 1 -- 172.21.15.88:0/1586743256 <== mon.4 v2:172.21.15.88:3302/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 3502+0+0 (secure 0 0 0) 0x7f04d40430e0 con 0x7f04e4142df0 2023-09-30T01:38:12.998 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.975+0000 7f04e37fe700 1 --2- 172.21.15.88:0/1586743256 >> v2:172.21.15.107:6812/29926 conn(0x7f04c4033360 0x7f04c4035810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:38:12.998 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.975+0000 7f04e37fe700 10 monclient: get_auth_request con 0x7f04c4033360 auth_method 0 2023-09-30T01:38:12.999 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.975+0000 7f04e37fe700 1 --2- 172.21.15.88:0/1586743256 >> v2:172.21.15.107:6812/29926 conn(0x7f04c4033360 0x7f04c4035810 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f04cc009b20 tx=0x7f04cc008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:38:12.999 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.983+0000 7f04db7fe700 1 -- 172.21.15.88:0/1586743256 <== mon.4 v2:172.21.15.88:3302/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f04d4013cb0 con 0x7f04e4142df0 2023-09-30T01:38:12.999 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.983+0000 7f04db7fe700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:38:12.999 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:12.983+0000 7f04db7fe700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:38:13.140 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.139+0000 7f04e8f87700 1 -- 172.21.15.88:0/1586743256 --> v2:172.21.15.107:6812/29926 -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "2", "target": ["mon-mgr", ""]}) v1 -- 0x7f04e411f130 con 0x7f04c4033360 2023-09-30T01:38:13.140 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.139+0000 7f04db7fe700 1 -- 172.21.15.88:0/1586743256 <== mgr.4108 v2:172.21.15.107:6812/29926 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 2 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f04e411f130 con 0x7f04c4033360 2023-09-30T01:38:13.143 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.139+0000 7f04e8f87700 1 -- 172.21.15.88:0/1586743256 >> v2:172.21.15.107:6812/29926 conn(0x7f04c4033360 msgr2=0x7f04c4035810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:38:13.143 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.139+0000 7f04e8f87700 1 --2- 172.21.15.88:0/1586743256 >> v2:172.21.15.107:6812/29926 conn(0x7f04c4033360 0x7f04c4035810 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f04cc009b20 tx=0x7f04cc008040 comp rx=0 tx=0).stop 2023-09-30T01:38:13.144 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.139+0000 7f04e8f87700 10 monclient: shutdown 2023-09-30T01:38:13.144 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.139+0000 7f04e8f87700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:38:13.144 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.139+0000 7f04e8f87700 1 -- 172.21.15.88:0/1586743256 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f04e4142df0 msgr2=0x7f04e409c7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:38:13.144 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.139+0000 7f04e8f87700 1 --2- 172.21.15.88:0/1586743256 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f04e4142df0 0x7f04e409c7b0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f04d400e8d0 tx=0x7f04d4008040 comp rx=0 tx=0).stop 2023-09-30T01:38:13.144 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.139+0000 7f04e8f87700 1 -- 172.21.15.88:0/1586743256 shutdown_connections 2023-09-30T01:38:13.144 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.139+0000 7f04e8f87700 1 --2- 172.21.15.88:0/1586743256 >> v2:172.21.15.107:6812/29926 conn(0x7f04c4033360 0x7f04c4035810 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:38:13.145 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.139+0000 7f04e8f87700 1 --2- 172.21.15.88:0/1586743256 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f04e4142df0 0x7f04e409c7b0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:38:13.145 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.139+0000 7f04e8f87700 1 --2- 172.21.15.88:0/1586743256 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7f04e4130610 0x7f04e41428b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:38:13.145 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.139+0000 7f04e8f87700 1 --2- 172.21.15.88:0/1586743256 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f04e409edf0 0x7f04e4142370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:38:13.145 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.139+0000 7f04e8f87700 1 -- 172.21.15.88:0/1586743256 >> 172.21.15.88:0/1586743256 conn(0x7f04e4097a40 msgr2=0x7f04e406e620 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:38:13.145 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.139+0000 7f04e8f87700 1 -- 172.21.15.88:0/1586743256 shutdown_connections 2023-09-30T01:38:13.146 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.139+0000 7f04e8f87700 1 -- 172.21.15.88:0/1586743256 wait complete. 2023-09-30T01:38:13.146 INFO:teuthology.orchestra.run.smithi088.stderr:instructed osd(s) 2 to deep-scrub 2023-09-30T01:38:13.154 INFO:tasks.ceph:Scrubbing osd.3 2023-09-30T01:38:13.155 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 config set osd_debug_deep_scrub_sleep 0 2023-09-30T01:38:13.264 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.263+0000 7f6569399700 10 monclient: get_monmap_and_config 2023-09-30T01:38:13.265 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.263+0000 7f6569399700 10 monclient: build_initial_monmap 2023-09-30T01:38:13.265 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.263+0000 7f6569399700 10 monclient: monmap: 2023-09-30T01:38:13.265 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2023-09-30T01:38:13.265 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:38:13.265 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:38:13.264744+0000 2023-09-30T01:38:13.265 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:38:13.264744+0000 2023-09-30T01:38:13.265 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2023-09-30T01:38:13.265 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:38:13.265 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a-legacy 2023-09-30T01:38:13.266 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.107:6789/0 mon.noname-f-legacy 2023-09-30T01:38:13.266 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.noname-b 2023-09-30T01:38:13.266 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.noname-g 2023-09-30T01:38:13.266 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.noname-c 2023-09-30T01:38:13.266 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.noname-h 2023-09-30T01:38:13.266 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.noname-d 2023-09-30T01:38:13.266 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.noname-i 2023-09-30T01:38:13.266 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.noname-e 2023-09-30T01:38:13.266 INFO:teuthology.orchestra.run.smithi088.stderr:9: v2:172.21.15.88:3300/0 mon.noname-a 2023-09-30T01:38:13.266 INFO:teuthology.orchestra.run.smithi088.stderr:10: v2:172.21.15.107:3300/0 mon.noname-f 2023-09-30T01:38:13.266 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:38:13.275 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.263+0000 7f6569399700 1 Processor -- start 2023-09-30T01:38:13.275 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.263+0000 7f6569399700 1 -- start start 2023-09-30T01:38:13.275 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.263+0000 7f6569399700 10 monclient: init 2023-09-30T01:38:13.275 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.263+0000 7f6569399700 10 monclient: _reopen_session rank -1 2023-09-30T01:38:13.275 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.263+0000 7f6569399700 10 monclient: _add_conns ranks=[3,2,5,4,9,7,0,10,6,1,8] 2023-09-30T01:38:13.275 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.263+0000 7f6569399700 1 --2- >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7f6564097a40 0x7f6564131210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:38:13.275 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.263+0000 7f6569399700 10 monclient(hunting): picked mon.noname-g con 0x7f6564097a40 addr [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] 2023-09-30T01:38:13.275 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.263+0000 7f6569399700 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f65641327b0 0x7f6564132b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:38:13.276 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.263+0000 7f6569399700 10 monclient(hunting): picked mon.noname-b con 0x7f65641327b0 addr [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] 2023-09-30T01:38:13.276 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.263+0000 7f6569399700 1 --2- >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7f65641332a0 0x7f65641336d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:38:13.276 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.263+0000 7f6569399700 10 monclient(hunting): picked mon.noname-h con 0x7f65641332a0 addr [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] 2023-09-30T01:38:13.276 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.263+0000 7f6569399700 10 monclient(hunting): start opening mon connection 2023-09-30T01:38:13.276 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.263+0000 7f6569399700 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65640a3620 con 0x7f65641327b0 2023-09-30T01:38:13.276 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.263+0000 7f6569399700 10 monclient(hunting): start opening mon connection 2023-09-30T01:38:13.276 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.263+0000 7f6569399700 1 -- --> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6564097e10 con 0x7f6564097a40 2023-09-30T01:38:13.276 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.263+0000 7f6569399700 10 monclient(hunting): start opening mon connection 2023-09-30T01:38:13.276 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.263+0000 7f6569399700 1 -- --> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] -- mon_getmap magic: 0 v1 -- 0x7f6564099110 con 0x7f65641332a0 2023-09-30T01:38:13.276 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.263+0000 7f6569399700 10 monclient(hunting): _renew_subs 2023-09-30T01:38:13.276 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.263+0000 7f6569399700 10 monclient(hunting): authenticate will time out at 1165.303112s 2023-09-30T01:38:13.276 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.263+0000 7f65637fe700 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f65641327b0 0x7f6564132b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:38:13.277 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.263+0000 7f65637fe700 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f65641327b0 0x7f6564132b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3301/0 says I am v2:172.21.15.88:53542/0 (socket says 172.21.15.88:53542) 2023-09-30T01:38:13.277 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.263+0000 7f65637fe700 1 -- 172.21.15.88:0/2357676145 learned_addr learned my addr 172.21.15.88:0/2357676145 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-09-30T01:38:13.277 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.263+0000 7f65637fe700 10 monclient(hunting): get_auth_request con 0x7f65641327b0 auth_method 0 2023-09-30T01:38:13.277 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.263+0000 7f65637fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:38:13.277 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.263+0000 7f65637fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:38:13.277 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.263+0000 7f65637fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:38:13.277 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.263+0000 7f65637fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:38:13.277 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.263+0000 7f65637fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:38:13.277 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.263+0000 7f65637fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:38:13.277 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.263+0000 7f6563fff700 1 --2- 172.21.15.88:0/2357676145 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7f6564097a40 0x7f6564131210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:38:13.278 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.263+0000 7f6568b98700 1 --2- 172.21.15.88:0/2357676145 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7f65641332a0 0x7f65641336d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:38:13.278 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.263+0000 7f6568b98700 10 monclient(hunting): get_auth_request con 0x7f65641332a0 auth_method 0 2023-09-30T01:38:13.278 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.263+0000 7f6568b98700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:38:13.278 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.263+0000 7f6568b98700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:38:13.278 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.263+0000 7f6568b98700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:38:13.278 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.263+0000 7f6563fff700 10 monclient(hunting): get_auth_request con 0x7f6564097a40 auth_method 0 2023-09-30T01:38:13.278 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.263+0000 7f6563fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:38:13.278 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.263+0000 7f6563fff700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:38:13.278 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.263+0000 7f6563fff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:38:13.278 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.263+0000 7f65637fe700 10 monclient(hunting): handle_auth_done global_id 4898 payload 274 2023-09-30T01:38:13.278 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.263+0000 7f65637fe700 1 -- 172.21.15.88:0/2357676145 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7f65641332a0 msgr2=0x7f65641336d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:38:13.279 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.263+0000 7f65637fe700 1 --2- 172.21.15.88:0/2357676145 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7f65641332a0 0x7f65641336d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:38:13.279 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.263+0000 7f65637fe700 1 -- 172.21.15.88:0/2357676145 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7f6564097a40 msgr2=0x7f6564131210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:38:13.279 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.263+0000 7f65637fe700 1 --2- 172.21.15.88:0/2357676145 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7f6564097a40 0x7f6564131210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:38:13.279 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.263+0000 7f65637fe700 10 monclient: _finish_hunting 0 2023-09-30T01:38:13.279 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.263+0000 7f65637fe700 1 monclient: found mon.noname-b 2023-09-30T01:38:13.279 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.263+0000 7f65637fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:38:13.279 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.263+0000 7f65637fe700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.88:3301/0 2023-09-30T01:38:13.279 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.263+0000 7f65637fe700 1 -- 172.21.15.88:0/2357676145 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f656404bb20 con 0x7f65641327b0 2023-09-30T01:38:13.279 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.263+0000 7f65637fe700 1 --2- 172.21.15.88:0/2357676145 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f65641327b0 0x7f6564132b80 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f65580099e0 tx=0x7f6558009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2409725f404254b9 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:38:13.279 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.267+0000 7f6562ffd700 1 -- 172.21.15.88:0/2357676145 <== mon.2 v2:172.21.15.88:3301/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f655800bce0 con 0x7f65641327b0 2023-09-30T01:38:13.279 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.267+0000 7f6562ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:38:13.280 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.267+0000 7f6562ffd700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T01:38:13.280 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.267+0000 7f6562ffd700 10 monclient: dump: 2023-09-30T01:38:13.280 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:38:13.280 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:38:13.280 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:38:13.280 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:38:13.280 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:38:13.280 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:38:13.280 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:38:13.280 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:38:13.280 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:38:13.281 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:38:13.281 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:38:13.281 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:38:13.281 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:38:13.281 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:38:13.281 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:38:13.281 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:38:13.281 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.267+0000 7f6562ffd700 10 monclient: _finish_auth 0 2023-09-30T01:38:13.281 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.267+0000 7f6562ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:38:13.281 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.267+0000 7f6569399700 5 monclient: authenticate success, global_id 4898 2023-09-30T01:38:13.281 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.267+0000 7f6569399700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:38:13.282 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.267+0000 7f6562ffd700 1 -- 172.21.15.88:0/2357676145 <== mon.2 v2:172.21.15.88:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f655800be60 con 0x7f65641327b0 2023-09-30T01:38:13.282 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.267+0000 7f6562ffd700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:38:13.282 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.267+0000 7f6569399700 10 monclient: get_monmap_and_config success 2023-09-30T01:38:13.282 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.267+0000 7f6569399700 4 set_mon_vals no callback set 2023-09-30T01:38:13.282 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.267+0000 7f6562ffd700 1 -- 172.21.15.88:0/2357676145 <== mon.2 v2:172.21.15.88:3301/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f65580118a0 con 0x7f65641327b0 2023-09-30T01:38:13.282 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.267+0000 7f6562ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:38:13.282 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.267+0000 7f6562ffd700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T01:38:13.282 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.267+0000 7f6562ffd700 10 monclient: dump: 2023-09-30T01:38:13.282 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:38:13.282 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:38:13.282 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:38:13.283 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:38:13.283 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:38:13.283 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:38:13.283 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:38:13.283 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:38:13.283 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:38:13.283 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:38:13.283 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:38:13.283 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:38:13.283 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:38:13.283 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:38:13.284 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:38:13.284 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:38:13.284 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.267+0000 7f6569399700 10 monclient: shutdown 2023-09-30T01:38:13.284 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.267+0000 7f6569399700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:38:13.284 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.267+0000 7f6569399700 1 -- 172.21.15.88:0/2357676145 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f65641327b0 msgr2=0x7f6564132b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:38:13.284 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.267+0000 7f6569399700 1 --2- 172.21.15.88:0/2357676145 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f65641327b0 0x7f6564132b80 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f65580099e0 tx=0x7f6558009250 comp rx=0 tx=0).stop 2023-09-30T01:38:13.284 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.267+0000 7f6569399700 1 -- 172.21.15.88:0/2357676145 shutdown_connections 2023-09-30T01:38:13.284 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.267+0000 7f6569399700 1 --2- 172.21.15.88:0/2357676145 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7f65641332a0 0x7f65641336d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:38:13.284 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.267+0000 7f6569399700 1 --2- 172.21.15.88:0/2357676145 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f65641327b0 0x7f6564132b80 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:38:13.284 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.267+0000 7f6569399700 1 --2- 172.21.15.88:0/2357676145 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7f6564097a40 0x7f6564131210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:38:13.284 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.267+0000 7f6569399700 1 -- 172.21.15.88:0/2357676145 >> 172.21.15.88:0/2357676145 conn(0x7f656406e5e0 msgr2=0x7f656406e9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:38:13.285 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.267+0000 7f6569399700 1 -- 172.21.15.88:0/2357676145 shutdown_connections 2023-09-30T01:38:13.285 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.267+0000 7f6569399700 1 -- 172.21.15.88:0/2357676145 wait complete. 2023-09-30T01:38:13.285 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.267+0000 7f6569399700 10 monclient: build_initial_monmap 2023-09-30T01:38:13.285 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.267+0000 7f6569399700 10 monclient: monmap: 2023-09-30T01:38:13.285 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2023-09-30T01:38:13.285 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:38:13.285 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 0.000000 2023-09-30T01:38:13.285 INFO:teuthology.orchestra.run.smithi088.stderr:created 0.000000 2023-09-30T01:38:13.285 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2023-09-30T01:38:13.285 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 1 2023-09-30T01:38:13.285 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.noname-a 2023-09-30T01:38:13.285 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.noname-f 2023-09-30T01:38:13.286 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.noname-b 2023-09-30T01:38:13.286 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.noname-g 2023-09-30T01:38:13.286 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.noname-c 2023-09-30T01:38:13.286 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.noname-h 2023-09-30T01:38:13.286 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.noname-d 2023-09-30T01:38:13.286 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.noname-i 2023-09-30T01:38:13.286 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.noname-e 2023-09-30T01:38:13.286 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:38:13.286 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.267+0000 7f6569399700 1 Processor -- start 2023-09-30T01:38:13.286 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.267+0000 7f6569399700 1 -- start start 2023-09-30T01:38:13.286 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.267+0000 7f6569399700 10 monclient: init 2023-09-30T01:38:13.287 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.267+0000 7f6569399700 10 monclient: _reopen_session rank -1 2023-09-30T01:38:13.287 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.267+0000 7f6569399700 10 monclient: _add_conns ranks=[6,3,2,4,5,1,0,7,8] 2023-09-30T01:38:13.287 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.267+0000 7f6569399700 1 --2- >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7f6564097a40 0x7f656411cdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:38:13.287 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.267+0000 7f6569399700 10 monclient(hunting): picked mon.noname-d con 0x7f6564097a40 addr [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] 2023-09-30T01:38:13.287 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.267+0000 7f6569399700 1 --2- >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7f65641332a0 0x7f656411d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:38:13.287 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.267+0000 7f6569399700 10 monclient(hunting): picked mon.noname-g con 0x7f65641332a0 addr [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] 2023-09-30T01:38:13.287 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.267+0000 7f6569399700 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f656411d840 0x7f6564148720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:38:13.287 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.267+0000 7f6569399700 10 monclient(hunting): picked mon.noname-b con 0x7f656411d840 addr [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] 2023-09-30T01:38:13.287 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.267+0000 7f6569399700 10 monclient(hunting): start opening mon connection 2023-09-30T01:38:13.287 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.267+0000 7f6569399700 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f656411dc90 con 0x7f656411d840 2023-09-30T01:38:13.287 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.267+0000 7f6569399700 10 monclient(hunting): start opening mon connection 2023-09-30T01:38:13.288 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.267+0000 7f6569399700 1 -- --> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6564099b30 con 0x7f65641332a0 2023-09-30T01:38:13.288 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.267+0000 7f6569399700 10 monclient(hunting): start opening mon connection 2023-09-30T01:38:13.288 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.267+0000 7f6569399700 1 -- --> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] -- mon_getmap magic: 0 v1 -- 0x7f6564099640 con 0x7f6564097a40 2023-09-30T01:38:13.288 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.267+0000 7f6569399700 10 monclient(hunting): _renew_subs 2023-09-30T01:38:13.288 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.267+0000 7f6569399700 10 monclient(hunting): authenticate will time out at 1165.306545s 2023-09-30T01:38:13.288 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.267+0000 7f6563fff700 1 --2- >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7f6564097a40 0x7f656411cdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:38:13.288 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.267+0000 7f6568b98700 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f656411d840 0x7f6564148720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:38:13.288 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.267+0000 7f6563fff700 1 --2- >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7f6564097a40 0x7f656411cdc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3303/0 says I am v2:172.21.15.88:49800/0 (socket says 172.21.15.88:49800) 2023-09-30T01:38:13.288 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.267+0000 7f6563fff700 1 -- 172.21.15.88:0/3951775886 learned_addr learned my addr 172.21.15.88:0/3951775886 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-09-30T01:38:13.288 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.267+0000 7f6563fff700 10 monclient(hunting): get_auth_request con 0x7f6564097a40 auth_method 0 2023-09-30T01:38:13.288 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.267+0000 7f6563fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:38:13.289 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.267+0000 7f6563fff700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:38:13.289 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.267+0000 7f6563fff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:38:13.289 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.267+0000 7f6568b98700 10 monclient(hunting): get_auth_request con 0x7f656411d840 auth_method 0 2023-09-30T01:38:13.289 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.267+0000 7f6568b98700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:38:13.289 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.267+0000 7f6568b98700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:38:13.289 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.267+0000 7f6568b98700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:38:13.289 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.267+0000 7f65637fe700 1 --2- 172.21.15.88:0/3951775886 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7f65641332a0 0x7f656411d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:38:13.289 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.267+0000 7f65637fe700 10 monclient(hunting): get_auth_request con 0x7f65641332a0 auth_method 0 2023-09-30T01:38:13.289 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.267+0000 7f65637fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:38:13.289 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.267+0000 7f65637fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:38:13.289 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.267+0000 7f65637fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:38:13.290 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.267+0000 7f6563fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:38:13.290 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.267+0000 7f6563fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:38:13.290 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.267+0000 7f6563fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:38:13.290 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.267+0000 7f6568b98700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:38:13.290 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.267+0000 7f6568b98700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:38:13.290 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.267+0000 7f6568b98700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:38:13.290 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.267+0000 7f65637fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:38:13.290 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.267+0000 7f65637fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:38:13.290 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.267+0000 7f65637fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:38:13.290 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.267+0000 7f6563fff700 10 monclient(hunting): handle_auth_done global_id 4839 payload 824 2023-09-30T01:38:13.291 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.267+0000 7f6563fff700 1 -- 172.21.15.88:0/3951775886 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7f65641332a0 msgr2=0x7f656411d300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:38:13.291 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.267+0000 7f6563fff700 1 --2- 172.21.15.88:0/3951775886 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7f65641332a0 0x7f656411d300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:38:13.291 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.267+0000 7f6563fff700 1 -- 172.21.15.88:0/3951775886 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f656411d840 msgr2=0x7f6564148720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:38:13.291 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.267+0000 7f6563fff700 1 --2- 172.21.15.88:0/3951775886 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f656411d840 0x7f6564148720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:38:13.291 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.267+0000 7f6563fff700 10 monclient: _finish_hunting 0 2023-09-30T01:38:13.291 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.267+0000 7f6563fff700 1 monclient: found mon.noname-d 2023-09-30T01:38:13.291 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.267+0000 7f6563fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:38:13.291 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.267+0000 7f6563fff700 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.88:3303/0 2023-09-30T01:38:13.291 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.267+0000 7f6563fff700 1 -- 172.21.15.88:0/3951775886 --> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65580094f0 con 0x7f6564097a40 2023-09-30T01:38:13.291 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.267+0000 7f6563fff700 1 --2- 172.21.15.88:0/3951775886 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7f6564097a40 0x7f656411cdc0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f655400d890 tx=0x7f6554008040 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:38:13.291 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.267+0000 7f654bfff700 1 -- 172.21.15.88:0/3951775886 <== mon.6 v2:172.21.15.88:3303/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f6554007470 con 0x7f6564097a40 2023-09-30T01:38:13.292 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.267+0000 7f654bfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:38:13.292 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.267+0000 7f654bfff700 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2023-09-30T01:38:13.292 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.267+0000 7f654bfff700 10 monclient: dump: 2023-09-30T01:38:13.292 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:38:13.292 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:38:13.292 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:38:13.292 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:38:13.292 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:38:13.292 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:38:13.292 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:38:13.292 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:38:13.293 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:38:13.293 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:38:13.293 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:38:13.293 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:38:13.293 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:38:13.293 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:38:13.293 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:38:13.293 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:38:13.293 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.271+0000 7f654bfff700 10 monclient: _finish_auth 0 2023-09-30T01:38:13.293 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.271+0000 7f654bfff700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:38:13.293 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.271+0000 7f6569399700 5 monclient: authenticate success, global_id 4839 2023-09-30T01:38:13.293 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.271+0000 7f654bfff700 1 -- 172.21.15.88:0/3951775886 <== mon.6 v2:172.21.15.88:3303/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6554012a70 con 0x7f6564097a40 2023-09-30T01:38:13.294 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.271+0000 7f6569399700 10 monclient: _renew_subs 2023-09-30T01:38:13.294 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.271+0000 7f6569399700 10 monclient: _send_mon_message to mon.d at v2:172.21.15.88:3303/0 2023-09-30T01:38:13.294 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.271+0000 7f6569399700 1 -- 172.21.15.88:0/3951775886 --> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f656411f3a0 con 0x7f6564097a40 2023-09-30T01:38:13.294 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.271+0000 7f654bfff700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:38:13.294 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.271+0000 7f654bfff700 1 -- 172.21.15.88:0/3951775886 <== mon.6 v2:172.21.15.88:3303/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f6554012be0 con 0x7f6564097a40 2023-09-30T01:38:13.294 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.271+0000 7f654bfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:38:13.294 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.271+0000 7f654bfff700 10 monclient: got monmap 1 from mon.d (according to old e1) 2023-09-30T01:38:13.294 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.271+0000 7f654bfff700 10 monclient: dump: 2023-09-30T01:38:13.294 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:38:13.294 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:38:13.295 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:38:13.295 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:38:13.295 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:38:13.295 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:38:13.295 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:38:13.295 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:38:13.295 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:38:13.295 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:38:13.295 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:38:13.295 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:38:13.295 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:38:13.295 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:38:13.296 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:38:13.296 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:38:13.296 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.271+0000 7f65627fc700 4 set_mon_vals no callback set 2023-09-30T01:38:13.296 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.271+0000 7f6569399700 10 monclient: _renew_subs 2023-09-30T01:38:13.296 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.271+0000 7f6569399700 10 monclient: _send_mon_message to mon.d at v2:172.21.15.88:3303/0 2023-09-30T01:38:13.296 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.271+0000 7f6569399700 1 -- 172.21.15.88:0/3951775886 --> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f65640a2c70 con 0x7f6564097a40 2023-09-30T01:38:13.296 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.271+0000 7f654bfff700 1 -- 172.21.15.88:0/3951775886 <== mon.6 v2:172.21.15.88:3303/0 4 ==== mgrmap(e 5) v1 ==== 40144+0+0 (secure 0 0 0) 0x7f6554012d40 con 0x7f6564097a40 2023-09-30T01:38:13.296 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.271+0000 7f6569399700 10 monclient: _send_mon_message to mon.d at v2:172.21.15.88:3303/0 2023-09-30T01:38:13.296 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.271+0000 7f6569399700 1 -- 172.21.15.88:0/3951775886 --> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f6564071960 con 0x7f6564097a40 2023-09-30T01:38:13.296 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.271+0000 7f654bfff700 1 --2- 172.21.15.88:0/3951775886 >> v2:172.21.15.107:6812/29926 conn(0x7f6550033360 0x7f6550035810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:38:13.296 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.271+0000 7f654bfff700 1 -- 172.21.15.88:0/3951775886 <== mon.6 v2:172.21.15.88:3303/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 3502+0+0 (secure 0 0 0) 0x7f6554043910 con 0x7f6564097a40 2023-09-30T01:38:13.297 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.271+0000 7f654bfff700 1 --2- 172.21.15.88:0/3951775886 >> v2:172.21.15.107:6804/30274 conn(0x7f6550038e00 0x7f655003b210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:38:13.297 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.271+0000 7f654bfff700 1 -- 172.21.15.88:0/3951775886 --> v2:172.21.15.107:6804/30274 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f655003b880 con 0x7f6550038e00 2023-09-30T01:38:13.297 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.271+0000 7f654bfff700 1 -- 172.21.15.88:0/3951775886 <== mon.6 v2:172.21.15.88:3303/0 6 ==== mon_get_version_reply(handle=1 version=15) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6554048240 con 0x7f6564097a40 2023-09-30T01:38:13.297 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.271+0000 7f654bfff700 10 monclient: handle_get_version_reply finishing 1 version 15 2023-09-30T01:38:13.297 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.271+0000 7f65637fe700 1 --2- 172.21.15.88:0/3951775886 >> v2:172.21.15.107:6812/29926 conn(0x7f6550033360 0x7f6550035810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:38:13.297 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.271+0000 7f65637fe700 10 monclient: get_auth_request con 0x7f6550033360 auth_method 0 2023-09-30T01:38:13.297 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.271+0000 7f6568b98700 1 --2- 172.21.15.88:0/3951775886 >> v2:172.21.15.107:6804/30274 conn(0x7f6550038e00 0x7f655003b210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:38:13.297 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.271+0000 7f6568b98700 10 monclient: get_auth_request con 0x7f6550038e00 auth_method 0 2023-09-30T01:38:13.298 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.271+0000 7f6568b98700 1 --2- 172.21.15.88:0/3951775886 >> v2:172.21.15.107:6804/30274 conn(0x7f6550038e00 0x7f655003b210 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:38:13.298 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.271+0000 7f65637fe700 1 --2- 172.21.15.88:0/3951775886 >> v2:172.21.15.107:6812/29926 conn(0x7f6550033360 0x7f6550035810 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f655800bfd0 tx=0x7f6558017000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:38:13.298 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.275+0000 7f654bfff700 1 -- 172.21.15.88:0/3951775886 <== osd.3 v2:172.21.15.107:6804/30274 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23999 (crc 0 0 0) 0x7f655003b880 con 0x7f6550038e00 2023-09-30T01:38:13.298 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.291+0000 7f6569399700 1 -- 172.21.15.88:0/3951775886 --> v2:172.21.15.107:6804/30274 -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f65640a2280 con 0x7f6550038e00 2023-09-30T01:38:13.298 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.291+0000 7f654bfff700 1 -- 172.21.15.88:0/3951775886 <== osd.3 v2:172.21.15.107:6804/30274 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7f65640a2280 con 0x7f6550038e00 2023-09-30T01:38:13.299 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-09-30T01:38:13.299 INFO:teuthology.orchestra.run.smithi088.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-09-30T01:38:13.299 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-09-30T01:38:13.300 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.295+0000 7f6569399700 1 -- 172.21.15.88:0/3951775886 >> v2:172.21.15.107:6804/30274 conn(0x7f6550038e00 msgr2=0x7f655003b210 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:38:13.300 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.295+0000 7f6569399700 1 --2- 172.21.15.88:0/3951775886 >> v2:172.21.15.107:6804/30274 conn(0x7f6550038e00 0x7f655003b210 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:38:13.300 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.295+0000 7f6569399700 1 -- 172.21.15.88:0/3951775886 >> v2:172.21.15.107:6812/29926 conn(0x7f6550033360 msgr2=0x7f6550035810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:38:13.300 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.295+0000 7f6569399700 1 --2- 172.21.15.88:0/3951775886 >> v2:172.21.15.107:6812/29926 conn(0x7f6550033360 0x7f6550035810 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f655800bfd0 tx=0x7f6558017000 comp rx=0 tx=0).stop 2023-09-30T01:38:13.300 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.295+0000 7f6569399700 10 monclient: shutdown 2023-09-30T01:38:13.300 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.295+0000 7f6569399700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:38:13.300 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.295+0000 7f6569399700 1 -- 172.21.15.88:0/3951775886 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7f6564097a40 msgr2=0x7f656411cdc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:38:13.300 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.295+0000 7f6569399700 1 --2- 172.21.15.88:0/3951775886 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7f6564097a40 0x7f656411cdc0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f655400d890 tx=0x7f6554008040 comp rx=0 tx=0).stop 2023-09-30T01:38:13.301 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.295+0000 7f6568b98700 1 -- 172.21.15.88:0/3951775886 reap_dead start 2023-09-30T01:38:13.301 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.295+0000 7f6569399700 1 -- 172.21.15.88:0/3951775886 shutdown_connections 2023-09-30T01:38:13.301 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.295+0000 7f6569399700 1 -- 172.21.15.88:0/3951775886 >> 172.21.15.88:0/3951775886 conn(0x7f656406e5e0 msgr2=0x7f65640a4db0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:38:13.301 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.295+0000 7f6569399700 1 -- 172.21.15.88:0/3951775886 shutdown_connections 2023-09-30T01:38:13.301 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.295+0000 7f6569399700 1 -- 172.21.15.88:0/3951775886 wait complete. 2023-09-30T01:38:13.307 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 3 2023-09-30T01:38:13.417 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.415+0000 7f617e55a700 10 monclient: get_monmap_and_config 2023-09-30T01:38:13.417 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.415+0000 7f617e55a700 10 monclient: build_initial_monmap 2023-09-30T01:38:13.417 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.415+0000 7f617e55a700 10 monclient: monmap: 2023-09-30T01:38:13.417 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2023-09-30T01:38:13.418 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:38:13.418 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:38:13.417227+0000 2023-09-30T01:38:13.418 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:38:13.417227+0000 2023-09-30T01:38:13.418 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2023-09-30T01:38:13.418 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:38:13.418 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a-legacy 2023-09-30T01:38:13.418 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.107:6789/0 mon.noname-f-legacy 2023-09-30T01:38:13.418 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.noname-b 2023-09-30T01:38:13.418 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.noname-g 2023-09-30T01:38:13.419 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.noname-c 2023-09-30T01:38:13.419 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.noname-h 2023-09-30T01:38:13.419 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.noname-d 2023-09-30T01:38:13.419 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.noname-i 2023-09-30T01:38:13.419 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.noname-e 2023-09-30T01:38:13.419 INFO:teuthology.orchestra.run.smithi088.stderr:9: v2:172.21.15.88:3300/0 mon.noname-a 2023-09-30T01:38:13.419 INFO:teuthology.orchestra.run.smithi088.stderr:10: v2:172.21.15.107:3300/0 mon.noname-f 2023-09-30T01:38:13.419 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:38:13.420 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.415+0000 7f617e55a700 1 Processor -- start 2023-09-30T01:38:13.421 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.415+0000 7f617e55a700 1 -- start start 2023-09-30T01:38:13.421 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.415+0000 7f617e55a700 10 monclient: init 2023-09-30T01:38:13.430 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.415+0000 7f617e55a700 10 monclient: _reopen_session rank -1 2023-09-30T01:38:13.431 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.415+0000 7f617e55a700 10 monclient: _add_conns ranks=[7,3,2,9,4,10,8,0,5,6,1] 2023-09-30T01:38:13.431 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.415+0000 7f617e55a700 1 --2- >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7f617806dbb0 0x7f617806d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:38:13.431 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.415+0000 7f617e55a700 10 monclient(hunting): picked mon.noname-i con 0x7f617806dbb0 addr [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] 2023-09-30T01:38:13.431 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.415+0000 7f617e55a700 1 --2- >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7f617806df80 0x7f617806e350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:38:13.431 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.415+0000 7f617e55a700 10 monclient(hunting): picked mon.noname-g con 0x7f617806df80 addr [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] 2023-09-30T01:38:13.431 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.415+0000 7f617e55a700 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f617806a670 0x7f617806aaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:38:13.431 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.415+0000 7f617e55a700 10 monclient(hunting): picked mon.noname-b con 0x7f617806a670 addr [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] 2023-09-30T01:38:13.431 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.415+0000 7f617e55a700 10 monclient(hunting): start opening mon connection 2023-09-30T01:38:13.431 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.415+0000 7f617e55a700 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f617806e890 con 0x7f617806a670 2023-09-30T01:38:13.432 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.415+0000 7f617e55a700 10 monclient(hunting): start opening mon connection 2023-09-30T01:38:13.432 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.415+0000 7f617e55a700 1 -- --> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f617806c8f0 con 0x7f617806df80 2023-09-30T01:38:13.432 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.415+0000 7f617e55a700 10 monclient(hunting): start opening mon connection 2023-09-30T01:38:13.432 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.415+0000 7f617e55a700 1 -- --> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] -- mon_getmap magic: 0 v1 -- 0x7f617806c610 con 0x7f617806dbb0 2023-09-30T01:38:13.432 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.415+0000 7f617e55a700 10 monclient(hunting): _renew_subs 2023-09-30T01:38:13.432 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.415+0000 7f617e55a700 10 monclient(hunting): authenticate will time out at 1165.455801s 2023-09-30T01:38:13.432 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.419+0000 7f617caf7700 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f617806a670 0x7f617806aaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:38:13.432 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.419+0000 7f617caf7700 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f617806a670 0x7f617806aaa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3301/0 says I am v2:172.21.15.88:53560/0 (socket says 172.21.15.88:53560) 2023-09-30T01:38:13.432 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.419+0000 7f617caf7700 1 -- 172.21.15.88:0/4064296102 learned_addr learned my addr 172.21.15.88:0/4064296102 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-09-30T01:38:13.433 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.419+0000 7f617caf7700 10 monclient(hunting): get_auth_request con 0x7f617806a670 auth_method 0 2023-09-30T01:38:13.433 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.419+0000 7f617caf7700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:38:13.433 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.419+0000 7f617caf7700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:38:13.433 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.419+0000 7f617caf7700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:38:13.433 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.419+0000 7f6177fff700 1 --2- 172.21.15.88:0/4064296102 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7f617806dbb0 0x7f617806d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:38:13.433 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.419+0000 7f61777fe700 1 --2- 172.21.15.88:0/4064296102 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7f617806df80 0x7f617806e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:38:13.433 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.419+0000 7f6177fff700 10 monclient(hunting): get_auth_request con 0x7f617806dbb0 auth_method 0 2023-09-30T01:38:13.433 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.419+0000 7f6177fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:38:13.433 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.419+0000 7f6177fff700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:38:13.433 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.419+0000 7f6177fff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:38:13.433 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.419+0000 7f61777fe700 10 monclient(hunting): get_auth_request con 0x7f617806df80 auth_method 0 2023-09-30T01:38:13.434 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.419+0000 7f61777fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:38:13.434 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.419+0000 7f61777fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:38:13.434 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.419+0000 7f61777fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:38:13.434 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.419+0000 7f617caf7700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:38:13.434 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.419+0000 7f617caf7700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:38:13.434 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.419+0000 7f617caf7700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:38:13.434 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.419+0000 7f6177fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:38:13.434 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.419+0000 7f6177fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:38:13.434 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.419+0000 7f6177fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:38:13.434 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.419+0000 7f61777fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:38:13.435 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.419+0000 7f61777fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:38:13.435 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.419+0000 7f61777fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:38:13.435 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.419+0000 7f617caf7700 10 monclient(hunting): handle_auth_done global_id 4916 payload 274 2023-09-30T01:38:13.435 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.419+0000 7f617caf7700 1 -- 172.21.15.88:0/4064296102 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7f617806dbb0 msgr2=0x7f617806d5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:38:13.435 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.419+0000 7f617caf7700 1 --2- 172.21.15.88:0/4064296102 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7f617806dbb0 0x7f617806d5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:38:13.435 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.419+0000 7f617caf7700 1 -- 172.21.15.88:0/4064296102 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7f617806df80 msgr2=0x7f617806e350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:38:13.435 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.419+0000 7f617caf7700 1 --2- 172.21.15.88:0/4064296102 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7f617806df80 0x7f617806e350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:38:13.435 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.419+0000 7f617caf7700 10 monclient: _finish_hunting 0 2023-09-30T01:38:13.435 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.419+0000 7f617caf7700 1 monclient: found mon.noname-b 2023-09-30T01:38:13.436 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.419+0000 7f617caf7700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:38:13.436 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.419+0000 7f617caf7700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.88:3301/0 2023-09-30T01:38:13.436 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.419+0000 7f617caf7700 1 -- 172.21.15.88:0/4064296102 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f617805ecb0 con 0x7f617806a670 2023-09-30T01:38:13.436 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.419+0000 7f6177fff700 1 --2- 172.21.15.88:0/4064296102 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7f617806dbb0 0x7f617806d5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:38:13.436 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.419+0000 7f61777fe700 1 --2- 172.21.15.88:0/4064296102 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7f617806df80 0x7f617806e350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:38:13.436 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.419+0000 7f617caf7700 1 --2- 172.21.15.88:0/4064296102 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f617806a670 0x7f617806aaa0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f617000d1d0 tx=0x7f617000ba50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d4d918411e2db09b server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:38:13.436 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.419+0000 7f6176ffd700 1 -- 172.21.15.88:0/4064296102 <== mon.2 v2:172.21.15.88:3301/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f6170010070 con 0x7f617806a670 2023-09-30T01:38:13.436 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.419+0000 7f6176ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:38:13.436 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.419+0000 7f6176ffd700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T01:38:13.436 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.419+0000 7f6176ffd700 10 monclient: dump: 2023-09-30T01:38:13.437 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:38:13.437 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:38:13.437 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:38:13.437 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:38:13.437 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:38:13.437 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:38:13.437 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:38:13.437 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:38:13.437 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:38:13.437 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:38:13.437 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:38:13.438 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:38:13.438 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:38:13.438 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:38:13.438 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:38:13.438 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:38:13.438 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.419+0000 7f6176ffd700 10 monclient: _finish_auth 0 2023-09-30T01:38:13.438 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.419+0000 7f6176ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:38:13.438 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.419+0000 7f6176ffd700 1 -- 172.21.15.88:0/4064296102 <== mon.2 v2:172.21.15.88:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f617000dd90 con 0x7f617806a670 2023-09-30T01:38:13.438 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.419+0000 7f617e55a700 5 monclient: authenticate success, global_id 4916 2023-09-30T01:38:13.439 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.419+0000 7f617e55a700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:38:13.439 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.419+0000 7f6176ffd700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:38:13.439 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.419+0000 7f617e55a700 10 monclient: get_monmap_and_config success 2023-09-30T01:38:13.439 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.419+0000 7f6176ffd700 1 -- 172.21.15.88:0/4064296102 <== mon.2 v2:172.21.15.88:3301/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f6170009a90 con 0x7f617806a670 2023-09-30T01:38:13.439 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.419+0000 7f6176ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:38:13.439 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.419+0000 7f617e55a700 4 set_mon_vals no callback set 2023-09-30T01:38:13.439 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.419+0000 7f6176ffd700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T01:38:13.439 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.419+0000 7f6176ffd700 10 monclient: dump: 2023-09-30T01:38:13.439 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:38:13.440 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:38:13.440 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:38:13.440 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:38:13.440 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:38:13.440 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:38:13.440 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:38:13.440 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:38:13.440 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:38:13.440 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:38:13.441 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:38:13.441 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:38:13.441 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:38:13.441 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:38:13.441 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:38:13.441 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:38:13.441 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.419+0000 7f617e55a700 10 monclient: shutdown 2023-09-30T01:38:13.441 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.419+0000 7f617e55a700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:38:13.441 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.419+0000 7f617e55a700 1 -- 172.21.15.88:0/4064296102 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f617806a670 msgr2=0x7f617806aaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:38:13.441 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.419+0000 7f617e55a700 1 --2- 172.21.15.88:0/4064296102 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f617806a670 0x7f617806aaa0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f617000d1d0 tx=0x7f617000ba50 comp rx=0 tx=0).stop 2023-09-30T01:38:13.442 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.419+0000 7f617e55a700 1 -- 172.21.15.88:0/4064296102 shutdown_connections 2023-09-30T01:38:13.442 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.419+0000 7f617e55a700 1 --2- 172.21.15.88:0/4064296102 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f617806a670 0x7f617806aaa0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:38:13.442 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.419+0000 7f617e55a700 1 --2- 172.21.15.88:0/4064296102 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7f617806df80 0x7f617806e350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:38:13.442 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.419+0000 7f617e55a700 1 --2- 172.21.15.88:0/4064296102 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7f617806dbb0 0x7f617806d5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:38:13.442 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.419+0000 7f617e55a700 1 -- 172.21.15.88:0/4064296102 >> 172.21.15.88:0/4064296102 conn(0x7f61780a01d0 msgr2=0x7f6178062c70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:38:13.442 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.419+0000 7f617e55a700 1 -- 172.21.15.88:0/4064296102 shutdown_connections 2023-09-30T01:38:13.442 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.419+0000 7f617e55a700 1 -- 172.21.15.88:0/4064296102 wait complete. 2023-09-30T01:38:13.442 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.423+0000 7f617e55a700 10 monclient: build_initial_monmap 2023-09-30T01:38:13.442 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.423+0000 7f617e55a700 10 monclient: monmap: 2023-09-30T01:38:13.443 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2023-09-30T01:38:13.443 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:38:13.443 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 0.000000 2023-09-30T01:38:13.443 INFO:teuthology.orchestra.run.smithi088.stderr:created 0.000000 2023-09-30T01:38:13.443 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2023-09-30T01:38:13.443 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 1 2023-09-30T01:38:13.443 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.noname-a 2023-09-30T01:38:13.443 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.noname-f 2023-09-30T01:38:13.443 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.noname-b 2023-09-30T01:38:13.444 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.noname-g 2023-09-30T01:38:13.444 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.noname-c 2023-09-30T01:38:13.444 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.noname-h 2023-09-30T01:38:13.444 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.noname-d 2023-09-30T01:38:13.444 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.noname-i 2023-09-30T01:38:13.444 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.noname-e 2023-09-30T01:38:13.444 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:38:13.444 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.423+0000 7f617e55a700 1 Processor -- start 2023-09-30T01:38:13.444 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.423+0000 7f617e55a700 1 -- start start 2023-09-30T01:38:13.445 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.423+0000 7f617e55a700 10 monclient: init 2023-09-30T01:38:13.445 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.423+0000 7f617e55a700 10 monclient: _reopen_session rank -1 2023-09-30T01:38:13.445 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.423+0000 7f617e55a700 10 monclient: _add_conns ranks=[4,2,3,6,0,7,8,1,5] 2023-09-30T01:38:13.445 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.423+0000 7f617e55a700 1 --2- >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f617806a670 0x7f617813a920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:38:13.445 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.423+0000 7f617e55a700 10 monclient(hunting): picked mon.noname-c con 0x7f617806a670 addr [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] 2023-09-30T01:38:13.445 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.423+0000 7f617e55a700 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f617806dbb0 0x7f6178139fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:38:13.445 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.423+0000 7f617e55a700 10 monclient(hunting): picked mon.noname-b con 0x7f617806dbb0 addr [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] 2023-09-30T01:38:13.445 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.423+0000 7f617e55a700 1 --2- >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7f617806df80 0x7f617813de90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:38:13.445 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.423+0000 7f617e55a700 10 monclient(hunting): picked mon.noname-g con 0x7f617806df80 addr [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] 2023-09-30T01:38:13.445 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.423+0000 7f617e55a700 10 monclient(hunting): start opening mon connection 2023-09-30T01:38:13.446 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.423+0000 7f617e55a700 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f617811f350 con 0x7f617806dbb0 2023-09-30T01:38:13.446 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.423+0000 7f617e55a700 10 monclient(hunting): start opening mon connection 2023-09-30T01:38:13.446 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.423+0000 7f617e55a700 1 -- --> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f617811d640 con 0x7f617806df80 2023-09-30T01:38:13.446 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.423+0000 7f617e55a700 10 monclient(hunting): start opening mon connection 2023-09-30T01:38:13.446 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.423+0000 7f617e55a700 1 -- --> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] -- mon_getmap magic: 0 v1 -- 0x7f617813b480 con 0x7f617806a670 2023-09-30T01:38:13.446 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.423+0000 7f617e55a700 10 monclient(hunting): _renew_subs 2023-09-30T01:38:13.446 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.423+0000 7f617e55a700 10 monclient(hunting): authenticate will time out at 1165.461271s 2023-09-30T01:38:13.446 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.423+0000 7f6177fff700 1 --2- >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f617806a670 0x7f617813a920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:38:13.446 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.423+0000 7f61777fe700 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f617806dbb0 0x7f6178139fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:38:13.447 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.423+0000 7f61777fe700 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f617806dbb0 0x7f6178139fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3301/0 says I am v2:172.21.15.88:53566/0 (socket says 172.21.15.88:53566) 2023-09-30T01:38:13.447 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.423+0000 7f617caf7700 1 --2- >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7f617806df80 0x7f617813de90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:38:13.447 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.423+0000 7f61777fe700 1 -- 172.21.15.88:0/1198387097 learned_addr learned my addr 172.21.15.88:0/1198387097 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-09-30T01:38:13.447 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.423+0000 7f6177fff700 10 monclient(hunting): get_auth_request con 0x7f617806a670 auth_method 0 2023-09-30T01:38:13.447 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.423+0000 7f6177fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:38:13.447 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.423+0000 7f6177fff700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:38:13.447 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.423+0000 7f6177fff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:38:13.447 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.423+0000 7f61777fe700 10 monclient(hunting): get_auth_request con 0x7f617806dbb0 auth_method 0 2023-09-30T01:38:13.447 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.423+0000 7f61777fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:38:13.448 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.423+0000 7f61777fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:38:13.448 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.423+0000 7f61777fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:38:13.448 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.423+0000 7f617caf7700 10 monclient(hunting): get_auth_request con 0x7f617806df80 auth_method 0 2023-09-30T01:38:13.448 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.423+0000 7f617caf7700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:38:13.448 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.423+0000 7f617caf7700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:38:13.448 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.423+0000 7f617caf7700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:38:13.448 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.423+0000 7f61777fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:38:13.448 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.423+0000 7f61777fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:38:13.448 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.423+0000 7f61777fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:38:13.448 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.423+0000 7f6177fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:38:13.449 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.423+0000 7f6177fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:38:13.449 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.423+0000 7f6177fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:38:13.449 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.423+0000 7f617caf7700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:38:13.449 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.423+0000 7f617caf7700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:38:13.449 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.423+0000 7f617caf7700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:38:13.449 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.423+0000 7f61777fe700 10 monclient(hunting): handle_auth_done global_id 4925 payload 824 2023-09-30T01:38:13.449 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.423+0000 7f61777fe700 1 -- 172.21.15.88:0/1198387097 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f617806a670 msgr2=0x7f617813a920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:38:13.449 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.423+0000 7f61777fe700 1 --2- 172.21.15.88:0/1198387097 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f617806a670 0x7f617813a920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:38:13.449 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.423+0000 7f61777fe700 1 -- 172.21.15.88:0/1198387097 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7f617806df80 msgr2=0x7f617813de90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:38:13.449 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.423+0000 7f61777fe700 1 --2- 172.21.15.88:0/1198387097 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7f617806df80 0x7f617813de90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:38:13.450 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.423+0000 7f61777fe700 10 monclient: _finish_hunting 0 2023-09-30T01:38:13.450 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.423+0000 7f61777fe700 1 monclient: found mon.noname-b 2023-09-30T01:38:13.450 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.423+0000 7f61777fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:38:13.450 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.423+0000 7f61777fe700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.88:3301/0 2023-09-30T01:38:13.450 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.423+0000 7f61777fe700 1 -- 172.21.15.88:0/1198387097 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6168009420 con 0x7f617806dbb0 2023-09-30T01:38:13.450 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.423+0000 7f61777fe700 1 --2- 172.21.15.88:0/1198387097 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f617806dbb0 0x7f6178139fd0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f616c00ec80 tx=0x7f616c0091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:38:13.450 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.423+0000 7f615ffff700 1 -- 172.21.15.88:0/1198387097 <== mon.2 v2:172.21.15.88:3301/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f616c00ccd0 con 0x7f617806dbb0 2023-09-30T01:38:13.450 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.423+0000 7f615ffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:38:13.450 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.423+0000 7f615ffff700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T01:38:13.451 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.423+0000 7f615ffff700 10 monclient: dump: 2023-09-30T01:38:13.451 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:38:13.451 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:38:13.451 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:38:13.451 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:38:13.451 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:38:13.451 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:38:13.451 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:38:13.451 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:38:13.451 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:38:13.452 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:38:13.452 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:38:13.452 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:38:13.452 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:38:13.452 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:38:13.452 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:38:13.452 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:38:13.452 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.423+0000 7f615ffff700 10 monclient: _finish_auth 0 2023-09-30T01:38:13.452 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.423+0000 7f615ffff700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:38:13.453 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.423+0000 7f617e55a700 5 monclient: authenticate success, global_id 4925 2023-09-30T01:38:13.453 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.423+0000 7f615ffff700 1 -- 172.21.15.88:0/1198387097 <== mon.2 v2:172.21.15.88:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f616c00ce30 con 0x7f617806dbb0 2023-09-30T01:38:13.453 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.423+0000 7f615ffff700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:38:13.453 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.423+0000 7f617e55a700 10 monclient: _renew_subs 2023-09-30T01:38:13.453 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.423+0000 7f615ffff700 1 -- 172.21.15.88:0/1198387097 <== mon.2 v2:172.21.15.88:3301/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f616c0138f0 con 0x7f617806dbb0 2023-09-30T01:38:13.453 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.423+0000 7f617e55a700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.88:3301/0 2023-09-30T01:38:13.453 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.423+0000 7f617e55a700 1 -- 172.21.15.88:0/1198387097 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f616c009420 con 0x7f617806dbb0 2023-09-30T01:38:13.453 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.423+0000 7f615ffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:38:13.453 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.423+0000 7f61767fc700 4 set_mon_vals no callback set 2023-09-30T01:38:13.453 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.423+0000 7f615ffff700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T01:38:13.454 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.423+0000 7f615ffff700 10 monclient: dump: 2023-09-30T01:38:13.454 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:38:13.454 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:38:13.454 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:38:13.454 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:38:13.454 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:38:13.454 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:38:13.454 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:38:13.454 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:38:13.455 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:38:13.455 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:38:13.455 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:38:13.455 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:38:13.455 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:38:13.455 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:38:13.455 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:38:13.455 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:38:13.455 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.423+0000 7f617e55a700 10 monclient: _renew_subs 2023-09-30T01:38:13.455 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.423+0000 7f617e55a700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.88:3301/0 2023-09-30T01:38:13.456 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.423+0000 7f617e55a700 1 -- 172.21.15.88:0/1198387097 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f617000bcf0 con 0x7f617806dbb0 2023-09-30T01:38:13.456 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.427+0000 7f615ffff700 1 -- 172.21.15.88:0/1198387097 <== mon.2 v2:172.21.15.88:3301/0 4 ==== mgrmap(e 5) v1 ==== 40144+0+0 (secure 0 0 0) 0x7f616c013470 con 0x7f617806dbb0 2023-09-30T01:38:13.456 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.427+0000 7f615ffff700 1 --2- 172.21.15.88:0/1198387097 >> v2:172.21.15.107:6812/29926 conn(0x7f6164033360 0x7f6164035810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:38:13.456 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.427+0000 7f615ffff700 1 -- 172.21.15.88:0/1198387097 <== mon.2 v2:172.21.15.88:3301/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 3502+0+0 (secure 0 0 0) 0x7f616c043870 con 0x7f617806dbb0 2023-09-30T01:38:13.456 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.427+0000 7f617e55a700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:38:13.456 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.427+0000 7f617e55a700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:38:13.456 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.427+0000 7f617e55a700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.88:3301/0 2023-09-30T01:38:13.456 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.427+0000 7f617e55a700 1 -- 172.21.15.88:0/1198387097 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6178071960 con 0x7f617806dbb0 2023-09-30T01:38:13.456 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.427+0000 7f6177fff700 1 --2- 172.21.15.88:0/1198387097 >> v2:172.21.15.107:6812/29926 conn(0x7f6164033360 0x7f6164035810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:38:13.456 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.427+0000 7f6177fff700 10 monclient: get_auth_request con 0x7f6164033360 auth_method 0 2023-09-30T01:38:13.457 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.427+0000 7f6177fff700 1 --2- 172.21.15.88:0/1198387097 >> v2:172.21.15.107:6812/29926 conn(0x7f6164033360 0x7f6164035810 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f6168009f40 tx=0x7f61680091a0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:38:13.457 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.431+0000 7f615ffff700 1 -- 172.21.15.88:0/1198387097 <== mon.2 v2:172.21.15.88:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f616c013b10 con 0x7f617806dbb0 2023-09-30T01:38:13.457 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.431+0000 7f615ffff700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:38:13.457 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.431+0000 7f615ffff700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:38:13.588 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.587+0000 7f617e55a700 1 -- 172.21.15.88:0/1198387097 --> v2:172.21.15.107:6812/29926 -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "3", "target": ["mon-mgr", ""]}) v1 -- 0x7f6178068610 con 0x7f6164033360 2023-09-30T01:38:13.590 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.587+0000 7f615ffff700 1 -- 172.21.15.88:0/1198387097 <== mgr.4108 v2:172.21.15.107:6812/29926 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 3 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f6178068610 con 0x7f6164033360 2023-09-30T01:38:13.593 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.591+0000 7f617e55a700 1 -- 172.21.15.88:0/1198387097 >> v2:172.21.15.107:6812/29926 conn(0x7f6164033360 msgr2=0x7f6164035810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:38:13.594 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.591+0000 7f617e55a700 1 --2- 172.21.15.88:0/1198387097 >> v2:172.21.15.107:6812/29926 conn(0x7f6164033360 0x7f6164035810 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f6168009f40 tx=0x7f61680091a0 comp rx=0 tx=0).stop 2023-09-30T01:38:13.594 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.591+0000 7f617e55a700 10 monclient: shutdown 2023-09-30T01:38:13.594 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.591+0000 7f617e55a700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:38:13.594 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.591+0000 7f617e55a700 1 -- 172.21.15.88:0/1198387097 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f617806dbb0 msgr2=0x7f6178139fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:38:13.594 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.591+0000 7f617e55a700 1 --2- 172.21.15.88:0/1198387097 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f617806dbb0 0x7f6178139fd0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f616c00ec80 tx=0x7f616c0091a0 comp rx=0 tx=0).stop 2023-09-30T01:38:13.594 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.591+0000 7f617e55a700 1 -- 172.21.15.88:0/1198387097 shutdown_connections 2023-09-30T01:38:13.594 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.591+0000 7f617e55a700 1 --2- 172.21.15.88:0/1198387097 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7f617806df80 0x7f617813de90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:38:13.594 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.591+0000 7f617e55a700 1 --2- 172.21.15.88:0/1198387097 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f617806dbb0 0x7f6178139fd0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:38:13.594 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.591+0000 7f617e55a700 1 --2- 172.21.15.88:0/1198387097 >> v2:172.21.15.107:6812/29926 conn(0x7f6164033360 0x7f6164035810 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:38:13.594 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.591+0000 7f617e55a700 1 --2- 172.21.15.88:0/1198387097 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f617806a670 0x7f617813a920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:38:13.594 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.591+0000 7f617e55a700 1 -- 172.21.15.88:0/1198387097 >> 172.21.15.88:0/1198387097 conn(0x7f61780a01d0 msgr2=0x7f61780a2db0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:38:13.595 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.591+0000 7f617e55a700 1 -- 172.21.15.88:0/1198387097 shutdown_connections 2023-09-30T01:38:13.595 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.591+0000 7f617e55a700 1 -- 172.21.15.88:0/1198387097 wait complete. 2023-09-30T01:38:13.596 INFO:teuthology.orchestra.run.smithi088.stderr:instructed osd(s) 3 to deep-scrub 2023-09-30T01:38:13.610 INFO:tasks.ceph:Scrubbing osd.4 2023-09-30T01:38:13.610 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 config set osd_debug_deep_scrub_sleep 0 2023-09-30T01:38:13.711 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.707+0000 7f6e83fb8700 10 monclient: get_monmap_and_config 2023-09-30T01:38:13.711 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.707+0000 7f6e83fb8700 10 monclient: build_initial_monmap 2023-09-30T01:38:13.711 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.707+0000 7f6e83fb8700 10 monclient: monmap: 2023-09-30T01:38:13.711 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2023-09-30T01:38:13.711 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:38:13.712 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:38:13.711152+0000 2023-09-30T01:38:13.712 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:38:13.711152+0000 2023-09-30T01:38:13.712 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2023-09-30T01:38:13.712 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:38:13.712 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a-legacy 2023-09-30T01:38:13.712 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.107:6789/0 mon.noname-f-legacy 2023-09-30T01:38:13.712 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.noname-b 2023-09-30T01:38:13.712 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.noname-g 2023-09-30T01:38:13.712 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.noname-c 2023-09-30T01:38:13.712 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.noname-h 2023-09-30T01:38:13.712 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.noname-d 2023-09-30T01:38:13.713 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.noname-i 2023-09-30T01:38:13.713 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.noname-e 2023-09-30T01:38:13.713 INFO:teuthology.orchestra.run.smithi088.stderr:9: v2:172.21.15.88:3300/0 mon.noname-a 2023-09-30T01:38:13.713 INFO:teuthology.orchestra.run.smithi088.stderr:10: v2:172.21.15.107:3300/0 mon.noname-f 2023-09-30T01:38:13.713 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:38:13.721 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.711+0000 7f6e83fb8700 1 Processor -- start 2023-09-30T01:38:13.721 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.711+0000 7f6e83fb8700 1 -- start start 2023-09-30T01:38:13.721 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.711+0000 7f6e83fb8700 10 monclient: init 2023-09-30T01:38:13.721 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.711+0000 7f6e83fb8700 10 monclient: _reopen_session rank -1 2023-09-30T01:38:13.721 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.711+0000 7f6e83fb8700 10 monclient: _add_conns ranks=[10,3,1,0,7,8,5,2,6,9,4] 2023-09-30T01:38:13.721 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.711+0000 7f6e83fb8700 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f6e7c133ee0 0x7f6e7c133670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:38:13.722 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.711+0000 7f6e83fb8700 10 monclient(hunting): picked mon.noname-f con 0x7f6e7c133ee0 addr v2:172.21.15.107:3300/0 2023-09-30T01:38:13.722 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.711+0000 7f6e83fb8700 1 --2- >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7f6e7c0a0040 0x7f6e7c0a1360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:38:13.722 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.711+0000 7f6e83fb8700 10 monclient(hunting): picked mon.noname-g con 0x7f6e7c0a0040 addr [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] 2023-09-30T01:38:13.722 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.711+0000 7f6e83fb8700 10 monclient(hunting): picked mon.noname-f-legacy con 0x7f6e7c0a0410 addr v1:172.21.15.107:6789/0 2023-09-30T01:38:13.723 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.711+0000 7f6e83fb8700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e7c05ecb0 con 0x7f6e7c0a0410 2023-09-30T01:38:13.723 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.711+0000 7f6e83fb8700 10 monclient(hunting): start opening mon connection 2023-09-30T01:38:13.723 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.711+0000 7f6e83fb8700 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6e7c098860 con 0x7f6e7c133ee0 2023-09-30T01:38:13.723 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.711+0000 7f6e83fb8700 10 monclient(hunting): start opening mon connection 2023-09-30T01:38:13.723 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.711+0000 7f6e83fb8700 1 -- --> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e7c0984f0 con 0x7f6e7c0a0040 2023-09-30T01:38:13.723 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.711+0000 7f6e83fb8700 10 monclient(hunting): _renew_subs 2023-09-30T01:38:13.724 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.711+0000 7f6e83fb8700 10 monclient(hunting): authenticate will time out at 1165.748916s 2023-09-30T01:38:13.724 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.711+0000 7f6e81553700 1 --2- >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7f6e7c0a0040 0x7f6e7c0a1360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:38:13.724 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.711+0000 7f6e82555700 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f6e7c0a0410 0x7f6e7c0a18a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.88:37716/0 (socket says 172.21.15.88:37716) 2023-09-30T01:38:13.724 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.711+0000 7f6e81d54700 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f6e7c133ee0 0x7f6e7c133670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:38:13.724 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.711+0000 7f6e82555700 1 -- 172.21.15.88:0/2517518855 learned_addr learned my addr 172.21.15.88:0/2517518855 (peer_addr_for_me v1:172.21.15.88:0/0) 2023-09-30T01:38:13.725 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.711+0000 7f6e81553700 10 monclient(hunting): get_auth_request con 0x7f6e7c0a0040 auth_method 0 2023-09-30T01:38:13.725 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.711+0000 7f6e81553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:38:13.725 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.711+0000 7f6e81553700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:38:13.725 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.711+0000 7f6e81553700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:38:13.725 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.711+0000 7f6e81d54700 10 monclient(hunting): get_auth_request con 0x7f6e7c133ee0 auth_method 0 2023-09-30T01:38:13.725 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.711+0000 7f6e81d54700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:38:13.725 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.711+0000 7f6e81d54700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:38:13.725 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.711+0000 7f6e81d54700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:38:13.725 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.711+0000 7f6e81553700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:38:13.725 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.711+0000 7f6e81553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:38:13.725 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.711+0000 7f6e81553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:38:13.726 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.711+0000 7f6e81d54700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:38:13.726 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.711+0000 7f6e81d54700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:38:13.726 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.711+0000 7f6e81d54700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:38:13.726 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.711+0000 7f6e80d52700 1 -- 172.21.15.88:0/2517518855 <== mon.1 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3531363361 0 0) 0x7f6e7c05ecb0 con 0x7f6e7c0a0410 2023-09-30T01:38:13.726 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.711+0000 7f6e80d52700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:38:13.726 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.711+0000 7f6e80d52700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:38:13.726 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.711+0000 7f6e80d52700 10 monclient(hunting): my global_id is 5311 2023-09-30T01:38:13.726 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.711+0000 7f6e80d52700 1 -- 172.21.15.88:0/2517518855 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6e64003570 con 0x7f6e7c0a0410 2023-09-30T01:38:13.726 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.711+0000 7f6e81553700 10 monclient(hunting): handle_auth_done global_id 4827 payload 274 2023-09-30T01:38:13.726 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.711+0000 7f6e81553700 1 -- 172.21.15.88:0/2517518855 >> v2:172.21.15.107:3300/0 conn(0x7f6e7c133ee0 msgr2=0x7f6e7c133670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:38:13.726 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.711+0000 7f6e81553700 1 --2- 172.21.15.88:0/2517518855 >> v2:172.21.15.107:3300/0 conn(0x7f6e7c133ee0 0x7f6e7c133670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:38:13.726 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.711+0000 7f6e81553700 1 -- 172.21.15.88:0/2517518855 >> v1:172.21.15.107:6789/0 conn(0x7f6e7c0a0410 legacy=0x7f6e7c0a18a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:38:13.727 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.711+0000 7f6e81553700 10 monclient: _finish_hunting 0 2023-09-30T01:38:13.727 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.711+0000 7f6e81553700 1 monclient: found mon.noname-g 2023-09-30T01:38:13.727 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.711+0000 7f6e81553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:38:13.727 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.711+0000 7f6e81553700 10 monclient: _send_mon_message to mon.noname-g at v2:172.21.15.107:3301/0 2023-09-30T01:38:13.727 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.711+0000 7f6e81553700 1 -- 172.21.15.88:0/2517518855 --> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e7c1342b0 con 0x7f6e7c0a0040 2023-09-30T01:38:13.727 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.711+0000 7f6e81d54700 1 --2- 172.21.15.88:0/2517518855 >> v2:172.21.15.107:3300/0 conn(0x7f6e7c133ee0 0x7f6e7c133670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:38:13.727 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.711+0000 7f6e81553700 1 --2- 172.21.15.88:0/2517518855 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7f6e7c0a0040 0x7f6e7c0a1360 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f6e700098d0 tx=0x7f6e70009120 comp rx=0 tx=0).ready entity=mon.3 client_cookie=fbc23d602b91f3c6 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:38:13.727 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.711+0000 7f6e80d52700 1 -- 172.21.15.88:0/2517518855 <== mon.3 v2:172.21.15.107:3301/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f6e70005c10 con 0x7f6e7c0a0040 2023-09-30T01:38:13.727 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.711+0000 7f6e80d52700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:38:13.727 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.711+0000 7f6e80d52700 10 monclient: got monmap 1 from mon.noname-g (according to old e1) 2023-09-30T01:38:13.727 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.711+0000 7f6e80d52700 10 monclient: dump: 2023-09-30T01:38:13.728 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:38:13.728 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:38:13.728 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:38:13.728 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:38:13.728 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:38:13.728 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:38:13.728 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:38:13.728 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:38:13.728 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:38:13.728 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:38:13.728 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:38:13.729 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:38:13.729 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:38:13.729 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:38:13.729 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:38:13.729 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:38:13.729 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.711+0000 7f6e80d52700 10 monclient: _finish_auth 0 2023-09-30T01:38:13.729 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.711+0000 7f6e80d52700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:38:13.729 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.711+0000 7f6e80d52700 1 -- 172.21.15.88:0/2517518855 <== mon.3 v2:172.21.15.107:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e70005d90 con 0x7f6e7c0a0040 2023-09-30T01:38:13.729 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.711+0000 7f6e83fb8700 5 monclient: authenticate success, global_id 4827 2023-09-30T01:38:13.729 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.711+0000 7f6e83fb8700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:38:13.729 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.711+0000 7f6e80d52700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:38:13.730 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.711+0000 7f6e83fb8700 10 monclient: get_monmap_and_config success 2023-09-30T01:38:13.730 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.711+0000 7f6e80d52700 1 -- 172.21.15.88:0/2517518855 <== mon.3 v2:172.21.15.107:3301/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f6e70011420 con 0x7f6e7c0a0040 2023-09-30T01:38:13.730 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.711+0000 7f6e83fb8700 4 set_mon_vals no callback set 2023-09-30T01:38:13.730 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.711+0000 7f6e80d52700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:38:13.730 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.711+0000 7f6e80d52700 10 monclient: got monmap 1 from mon.g (according to old e1) 2023-09-30T01:38:13.730 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.711+0000 7f6e80d52700 10 monclient: dump: 2023-09-30T01:38:13.731 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:38:13.731 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:38:13.731 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:38:13.731 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:38:13.731 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:38:13.731 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:38:13.731 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:38:13.731 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:38:13.731 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:38:13.731 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:38:13.732 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:38:13.732 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:38:13.732 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:38:13.732 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:38:13.732 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:38:13.732 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:38:13.732 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.711+0000 7f6e83fb8700 10 monclient: shutdown 2023-09-30T01:38:13.732 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.711+0000 7f6e83fb8700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:38:13.732 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.711+0000 7f6e83fb8700 1 -- 172.21.15.88:0/2517518855 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7f6e7c0a0040 msgr2=0x7f6e7c0a1360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:38:13.732 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.711+0000 7f6e83fb8700 1 --2- 172.21.15.88:0/2517518855 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7f6e7c0a0040 0x7f6e7c0a1360 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f6e700098d0 tx=0x7f6e70009120 comp rx=0 tx=0).stop 2023-09-30T01:38:13.732 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.711+0000 7f6e83fb8700 1 -- 172.21.15.88:0/2517518855 shutdown_connections 2023-09-30T01:38:13.733 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.711+0000 7f6e83fb8700 1 --2- 172.21.15.88:0/2517518855 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7f6e7c0a0040 0x7f6e7c0a1360 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:38:13.733 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.711+0000 7f6e83fb8700 1 --2- 172.21.15.88:0/2517518855 >> v2:172.21.15.107:3300/0 conn(0x7f6e7c133ee0 0x7f6e7c133670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:38:13.733 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.711+0000 7f6e83fb8700 1 -- 172.21.15.88:0/2517518855 >> 172.21.15.88:0/2517518855 conn(0x7f6e7c06e5e0 msgr2=0x7f6e7c06e9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:38:13.733 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.711+0000 7f6e83fb8700 1 -- 172.21.15.88:0/2517518855 shutdown_connections 2023-09-30T01:38:13.733 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.711+0000 7f6e83fb8700 1 -- 172.21.15.88:0/2517518855 wait complete. 2023-09-30T01:38:13.733 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.715+0000 7f6e83fb8700 10 monclient: build_initial_monmap 2023-09-30T01:38:13.733 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.715+0000 7f6e83fb8700 10 monclient: monmap: 2023-09-30T01:38:13.733 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2023-09-30T01:38:13.733 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:38:13.733 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 0.000000 2023-09-30T01:38:13.733 INFO:teuthology.orchestra.run.smithi088.stderr:created 0.000000 2023-09-30T01:38:13.734 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2023-09-30T01:38:13.734 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 1 2023-09-30T01:38:13.734 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.noname-a 2023-09-30T01:38:13.734 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.noname-f 2023-09-30T01:38:13.734 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.noname-b 2023-09-30T01:38:13.734 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.noname-g 2023-09-30T01:38:13.734 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.noname-c 2023-09-30T01:38:13.734 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.noname-h 2023-09-30T01:38:13.734 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.noname-d 2023-09-30T01:38:13.735 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.noname-i 2023-09-30T01:38:13.735 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.noname-e 2023-09-30T01:38:13.735 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:38:13.735 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.715+0000 7f6e83fb8700 1 Processor -- start 2023-09-30T01:38:13.735 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.715+0000 7f6e83fb8700 1 -- start start 2023-09-30T01:38:13.735 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.715+0000 7f6e83fb8700 10 monclient: init 2023-09-30T01:38:13.735 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.715+0000 7f6e83fb8700 10 monclient: _reopen_session rank -1 2023-09-30T01:38:13.735 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.715+0000 7f6e83fb8700 10 monclient: _add_conns ranks=[7,2,0,1,4,8,3,6,5] 2023-09-30T01:38:13.735 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.715+0000 7f6e83fb8700 1 --2- >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7f6e7c0a0040 0x7f6e7c11d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:38:13.735 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.715+0000 7f6e83fb8700 10 monclient(hunting): picked mon.noname-i con 0x7f6e7c0a0040 addr [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] 2023-09-30T01:38:13.736 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.715+0000 7f6e83fb8700 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6e7c0a0410 0x7f6e7c11d690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:38:13.736 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.715+0000 7f6e83fb8700 10 monclient(hunting): picked mon.noname-b con 0x7f6e7c0a0410 addr [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] 2023-09-30T01:38:13.736 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.715+0000 7f6e83fb8700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6e7c133ee0 0x7f6e7c11dbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:38:13.736 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.715+0000 7f6e83fb8700 10 monclient(hunting): picked mon.noname-a con 0x7f6e7c133ee0 addr [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] 2023-09-30T01:38:13.736 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.715+0000 7f6e83fb8700 10 monclient(hunting): start opening mon connection 2023-09-30T01:38:13.736 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.715+0000 7f6e83fb8700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e7c0a0bc0 con 0x7f6e7c133ee0 2023-09-30T01:38:13.736 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.715+0000 7f6e83fb8700 10 monclient(hunting): start opening mon connection 2023-09-30T01:38:13.736 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.715+0000 7f6e83fb8700 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e7c097d90 con 0x7f6e7c0a0410 2023-09-30T01:38:13.736 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.715+0000 7f6e83fb8700 10 monclient(hunting): start opening mon connection 2023-09-30T01:38:13.736 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.715+0000 7f6e83fb8700 1 -- --> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] -- mon_getmap magic: 0 v1 -- 0x7f6e7c1209e0 con 0x7f6e7c0a0040 2023-09-30T01:38:13.736 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.715+0000 7f6e83fb8700 10 monclient(hunting): _renew_subs 2023-09-30T01:38:13.737 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.715+0000 7f6e83fb8700 10 monclient(hunting): authenticate will time out at 1165.754065s 2023-09-30T01:38:13.737 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.715+0000 7f6e81553700 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6e7c0a0410 0x7f6e7c11d690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:38:13.737 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.715+0000 7f6e82555700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6e7c133ee0 0x7f6e7c11dbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:38:13.737 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.715+0000 7f6e81d54700 1 --2- >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7f6e7c0a0040 0x7f6e7c11d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:38:13.737 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.715+0000 7f6e81553700 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6e7c0a0410 0x7f6e7c11d690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3301/0 says I am v2:172.21.15.88:53574/0 (socket says 172.21.15.88:53574) 2023-09-30T01:38:13.737 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.715+0000 7f6e81553700 1 -- 172.21.15.88:0/394546539 learned_addr learned my addr 172.21.15.88:0/394546539 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-09-30T01:38:13.737 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.715+0000 7f6e81553700 10 monclient(hunting): get_auth_request con 0x7f6e7c0a0410 auth_method 0 2023-09-30T01:38:13.737 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.715+0000 7f6e81553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:38:13.737 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.715+0000 7f6e81553700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:38:13.737 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.715+0000 7f6e81553700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:38:13.737 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.715+0000 7f6e82555700 10 monclient(hunting): get_auth_request con 0x7f6e7c133ee0 auth_method 0 2023-09-30T01:38:13.737 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.715+0000 7f6e82555700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:38:13.738 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.715+0000 7f6e82555700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:38:13.738 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.715+0000 7f6e82555700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:38:13.738 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.715+0000 7f6e81d54700 10 monclient(hunting): get_auth_request con 0x7f6e7c0a0040 auth_method 0 2023-09-30T01:38:13.738 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.715+0000 7f6e81d54700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:38:13.738 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.715+0000 7f6e81d54700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:38:13.738 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.715+0000 7f6e81d54700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:38:13.738 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.715+0000 7f6e82555700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:38:13.738 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.715+0000 7f6e82555700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:38:13.738 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.715+0000 7f6e82555700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:38:13.738 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.715+0000 7f6e81553700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:38:13.738 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.715+0000 7f6e81553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:38:13.738 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.715+0000 7f6e81553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:38:13.739 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.715+0000 7f6e81d54700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:38:13.739 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.715+0000 7f6e81d54700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:38:13.739 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.715+0000 7f6e81d54700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:38:13.739 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.715+0000 7f6e82555700 10 monclient(hunting): handle_auth_done global_id 5229 payload 824 2023-09-30T01:38:13.739 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.715+0000 7f6e82555700 1 -- 172.21.15.88:0/394546539 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7f6e7c0a0040 msgr2=0x7f6e7c11d150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:38:13.739 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.715+0000 7f6e82555700 1 --2- 172.21.15.88:0/394546539 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7f6e7c0a0040 0x7f6e7c11d150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:38:13.739 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.715+0000 7f6e82555700 1 -- 172.21.15.88:0/394546539 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6e7c0a0410 msgr2=0x7f6e7c11d690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:38:13.739 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.715+0000 7f6e82555700 1 --2- 172.21.15.88:0/394546539 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6e7c0a0410 0x7f6e7c11d690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:38:13.739 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.715+0000 7f6e82555700 10 monclient: _finish_hunting 0 2023-09-30T01:38:13.740 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.715+0000 7f6e82555700 1 monclient: found mon.noname-a 2023-09-30T01:38:13.740 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.715+0000 7f6e82555700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:38:13.740 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.715+0000 7f6e82555700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.88:3300/0 2023-09-30T01:38:13.740 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.715+0000 7f6e82555700 1 -- 172.21.15.88:0/394546539 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e6c009a20 con 0x7f6e7c133ee0 2023-09-30T01:38:13.740 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.715+0000 7f6e81553700 1 --2- 172.21.15.88:0/394546539 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6e7c0a0410 0x7f6e7c11d690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:38:13.740 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.715+0000 7f6e82555700 1 --2- 172.21.15.88:0/394546539 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6e7c133ee0 0x7f6e7c11dbd0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f6e74001ed0 tx=0x7f6e74007d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:38:13.740 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.715+0000 7f6e79ffb700 1 -- 172.21.15.88:0/394546539 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f6e7400cea0 con 0x7f6e7c133ee0 2023-09-30T01:38:13.740 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.715+0000 7f6e79ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:38:13.740 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.715+0000 7f6e79ffb700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T01:38:13.740 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.715+0000 7f6e79ffb700 10 monclient: dump: 2023-09-30T01:38:13.740 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:38:13.740 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:38:13.740 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:38:13.741 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:38:13.741 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:38:13.741 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:38:13.741 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:38:13.741 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:38:13.741 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:38:13.742 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:38:13.742 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:38:13.742 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:38:13.742 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:38:13.742 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:38:13.742 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:38:13.742 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:38:13.742 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.715+0000 7f6e79ffb700 10 monclient: _finish_auth 0 2023-09-30T01:38:13.742 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.715+0000 7f6e79ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:38:13.742 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.715+0000 7f6e79ffb700 1 -- 172.21.15.88:0/394546539 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e740070d0 con 0x7f6e7c133ee0 2023-09-30T01:38:13.742 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.715+0000 7f6e83fb8700 5 monclient: authenticate success, global_id 5229 2023-09-30T01:38:13.742 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.715+0000 7f6e83fb8700 10 monclient: _renew_subs 2023-09-30T01:38:13.743 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.715+0000 7f6e83fb8700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.88:3300/0 2023-09-30T01:38:13.743 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.715+0000 7f6e83fb8700 1 -- 172.21.15.88:0/394546539 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e700093c0 con 0x7f6e7c133ee0 2023-09-30T01:38:13.743 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.715+0000 7f6e79ffb700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:38:13.743 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.715+0000 7f6e79ffb700 1 -- 172.21.15.88:0/394546539 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f6e740076c0 con 0x7f6e7c133ee0 2023-09-30T01:38:13.743 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.715+0000 7f6e79ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:38:13.743 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.715+0000 7f6e79ffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:38:13.743 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.715+0000 7f6e79ffb700 10 monclient: dump: 2023-09-30T01:38:13.743 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:38:13.743 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:38:13.743 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:38:13.743 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:38:13.744 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:38:13.744 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:38:13.744 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:38:13.744 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:38:13.744 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:38:13.744 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:38:13.744 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:38:13.745 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:38:13.745 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:38:13.745 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:38:13.745 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:38:13.745 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:38:13.745 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.715+0000 7f6e7bfff700 4 set_mon_vals no callback set 2023-09-30T01:38:13.745 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.715+0000 7f6e83fb8700 10 monclient: _renew_subs 2023-09-30T01:38:13.746 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.715+0000 7f6e83fb8700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.88:3300/0 2023-09-30T01:38:13.746 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.715+0000 7f6e83fb8700 1 -- 172.21.15.88:0/394546539 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e7c120420 con 0x7f6e7c133ee0 2023-09-30T01:38:13.746 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.719+0000 7f6e79ffb700 1 -- 172.21.15.88:0/394546539 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 40144+0+0 (secure 0 0 0) 0x7f6e74007240 con 0x7f6e7c133ee0 2023-09-30T01:38:13.746 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.719+0000 7f6e83fb8700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.88:3300/0 2023-09-30T01:38:13.747 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.719+0000 7f6e83fb8700 1 -- 172.21.15.88:0/394546539 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f6e7c071960 con 0x7f6e7c133ee0 2023-09-30T01:38:13.747 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.719+0000 7f6e79ffb700 1 --2- 172.21.15.88:0/394546539 >> v2:172.21.15.107:6812/29926 conn(0x7f6e68033360 0x7f6e68035810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:38:13.747 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.719+0000 7f6e79ffb700 1 -- 172.21.15.88:0/394546539 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 3502+0+0 (secure 0 0 0) 0x7f6e74042390 con 0x7f6e7c133ee0 2023-09-30T01:38:13.747 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.719+0000 7f6e79ffb700 1 --2- 172.21.15.88:0/394546539 >> v2:172.21.15.107:6808/30276 conn(0x7f6e68038e00 0x7f6e6803b210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:38:13.747 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.719+0000 7f6e79ffb700 1 -- 172.21.15.88:0/394546539 --> v2:172.21.15.107:6808/30276 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6e6803b880 con 0x7f6e68038e00 2023-09-30T01:38:13.748 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.719+0000 7f6e79ffb700 1 -- 172.21.15.88:0/394546539 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_get_version_reply(handle=1 version=15) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6e740074f0 con 0x7f6e7c133ee0 2023-09-30T01:38:13.748 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.719+0000 7f6e79ffb700 10 monclient: handle_get_version_reply finishing 1 version 15 2023-09-30T01:38:13.748 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.719+0000 7f6e81d54700 1 --2- 172.21.15.88:0/394546539 >> v2:172.21.15.107:6812/29926 conn(0x7f6e68033360 0x7f6e68035810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:38:13.748 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.719+0000 7f6e81553700 1 --2- 172.21.15.88:0/394546539 >> v2:172.21.15.107:6808/30276 conn(0x7f6e68038e00 0x7f6e6803b210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:38:13.748 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.719+0000 7f6e81d54700 10 monclient: get_auth_request con 0x7f6e68033360 auth_method 0 2023-09-30T01:38:13.748 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.719+0000 7f6e81553700 10 monclient: get_auth_request con 0x7f6e68038e00 auth_method 0 2023-09-30T01:38:13.749 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.719+0000 7f6e81553700 1 --2- 172.21.15.88:0/394546539 >> v2:172.21.15.107:6808/30276 conn(0x7f6e68038e00 0x7f6e6803b210 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:38:13.749 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.719+0000 7f6e81d54700 1 --2- 172.21.15.88:0/394546539 >> v2:172.21.15.107:6812/29926 conn(0x7f6e68033360 0x7f6e68035810 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f6e6c00f650 tx=0x7f6e6c00b040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:38:13.749 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.719+0000 7f6e79ffb700 1 -- 172.21.15.88:0/394546539 <== osd.4 v2:172.21.15.107:6808/30276 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23999 (crc 0 0 0) 0x7f6e6803b880 con 0x7f6e68038e00 2023-09-30T01:38:13.749 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.739+0000 7f6e83fb8700 1 -- 172.21.15.88:0/394546539 --> v2:172.21.15.107:6808/30276 -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f6e7c0a37c0 con 0x7f6e68038e00 2023-09-30T01:38:13.749 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.739+0000 7f6e79ffb700 1 -- 172.21.15.88:0/394546539 <== osd.4 v2:172.21.15.107:6808/30276 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7f6e7c0a37c0 con 0x7f6e68038e00 2023-09-30T01:38:13.750 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-09-30T01:38:13.750 INFO:teuthology.orchestra.run.smithi088.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-09-30T01:38:13.750 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-09-30T01:38:13.751 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.739+0000 7f6e83fb8700 1 -- 172.21.15.88:0/394546539 >> v2:172.21.15.107:6808/30276 conn(0x7f6e68038e00 msgr2=0x7f6e6803b210 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:38:13.752 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.739+0000 7f6e83fb8700 1 --2- 172.21.15.88:0/394546539 >> v2:172.21.15.107:6808/30276 conn(0x7f6e68038e00 0x7f6e6803b210 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:38:13.752 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.739+0000 7f6e83fb8700 1 -- 172.21.15.88:0/394546539 >> v2:172.21.15.107:6812/29926 conn(0x7f6e68033360 msgr2=0x7f6e68035810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:38:13.752 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.739+0000 7f6e83fb8700 1 --2- 172.21.15.88:0/394546539 >> v2:172.21.15.107:6812/29926 conn(0x7f6e68033360 0x7f6e68035810 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f6e6c00f650 tx=0x7f6e6c00b040 comp rx=0 tx=0).stop 2023-09-30T01:38:13.752 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.739+0000 7f6e83fb8700 10 monclient: shutdown 2023-09-30T01:38:13.752 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.739+0000 7f6e83fb8700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:38:13.752 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.739+0000 7f6e83fb8700 1 -- 172.21.15.88:0/394546539 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6e7c133ee0 msgr2=0x7f6e7c11dbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:38:13.752 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.739+0000 7f6e83fb8700 1 --2- 172.21.15.88:0/394546539 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6e7c133ee0 0x7f6e7c11dbd0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f6e74001ed0 tx=0x7f6e74007d10 comp rx=0 tx=0).stop 2023-09-30T01:38:13.752 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.739+0000 7f6e82555700 1 -- 172.21.15.88:0/394546539 reap_dead start 2023-09-30T01:38:13.752 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.739+0000 7f6e83fb8700 1 -- 172.21.15.88:0/394546539 shutdown_connections 2023-09-30T01:38:13.752 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.739+0000 7f6e83fb8700 1 -- 172.21.15.88:0/394546539 >> 172.21.15.88:0/394546539 conn(0x7f6e7c06e5e0 msgr2=0x7f6e7c0a4690 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:38:13.753 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.739+0000 7f6e83fb8700 1 -- 172.21.15.88:0/394546539 shutdown_connections 2023-09-30T01:38:13.753 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.739+0000 7f6e83fb8700 1 -- 172.21.15.88:0/394546539 wait complete. 2023-09-30T01:38:13.754 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 4 2023-09-30T01:38:13.863 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.859+0000 7f6deb59e700 10 monclient: get_monmap_and_config 2023-09-30T01:38:13.863 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.859+0000 7f6deb59e700 10 monclient: build_initial_monmap 2023-09-30T01:38:13.863 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.859+0000 7f6deb59e700 10 monclient: monmap: 2023-09-30T01:38:13.863 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2023-09-30T01:38:13.864 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:38:13.864 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:38:13.863107+0000 2023-09-30T01:38:13.864 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:38:13.863107+0000 2023-09-30T01:38:13.864 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2023-09-30T01:38:13.864 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:38:13.864 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a-legacy 2023-09-30T01:38:13.864 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.107:6789/0 mon.noname-f-legacy 2023-09-30T01:38:13.865 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.noname-b 2023-09-30T01:38:13.865 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.noname-g 2023-09-30T01:38:13.865 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.noname-c 2023-09-30T01:38:13.865 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.noname-h 2023-09-30T01:38:13.865 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.noname-d 2023-09-30T01:38:13.865 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.noname-i 2023-09-30T01:38:13.866 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.noname-e 2023-09-30T01:38:13.866 INFO:teuthology.orchestra.run.smithi088.stderr:9: v2:172.21.15.88:3300/0 mon.noname-a 2023-09-30T01:38:13.866 INFO:teuthology.orchestra.run.smithi088.stderr:10: v2:172.21.15.107:3300/0 mon.noname-f 2023-09-30T01:38:13.867 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:38:13.876 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.859+0000 7f6deb59e700 1 Processor -- start 2023-09-30T01:38:13.876 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.863+0000 7f6deb59e700 1 -- start start 2023-09-30T01:38:13.877 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.863+0000 7f6deb59e700 10 monclient: init 2023-09-30T01:38:13.878 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.863+0000 7f6deb59e700 10 monclient: _reopen_session rank -1 2023-09-30T01:38:13.878 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.863+0000 7f6deb59e700 10 monclient: _add_conns ranks=[0,4,7,2,5,8,1,10,9,3,6] 2023-09-30T01:38:13.878 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.863+0000 7f6deb59e700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f6dec097a40 addr v1:172.21.15.88:6789/0 2023-09-30T01:38:13.878 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.863+0000 7f6deb59e700 1 --2- >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f6dec132e10 0x7f6dec138650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:38:13.878 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.863+0000 7f6deb59e700 10 monclient(hunting): picked mon.noname-c con 0x7f6dec132e10 addr [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] 2023-09-30T01:38:13.878 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.863+0000 7f6deb59e700 1 --2- >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7f6dec1311c0 0x7f6dec13aba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:38:13.878 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.863+0000 7f6deb59e700 10 monclient(hunting): picked mon.noname-i con 0x7f6dec1311c0 addr [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] 2023-09-30T01:38:13.879 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.863+0000 7f6deb59e700 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6dec04bb20 con 0x7f6dec097a40 2023-09-30T01:38:13.879 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.863+0000 7f6deb59e700 10 monclient(hunting): start opening mon connection 2023-09-30T01:38:13.879 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.863+0000 7f6deb59e700 1 -- --> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] -- mon_getmap magic: 0 v1 -- 0x7f6dec09a5d0 con 0x7f6dec132e10 2023-09-30T01:38:13.879 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.863+0000 7f6deb59e700 10 monclient(hunting): start opening mon connection 2023-09-30T01:38:13.879 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.863+0000 7f6deb59e700 1 -- --> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] -- mon_getmap magic: 0 v1 -- 0x7f6dec097e10 con 0x7f6dec1311c0 2023-09-30T01:38:13.879 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.863+0000 7f6deb59e700 10 monclient(hunting): _renew_subs 2023-09-30T01:38:13.879 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.863+0000 7f6deb59e700 10 monclient(hunting): authenticate will time out at 1165.900691s 2023-09-30T01:38:13.879 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.863+0000 7f6dea59c700 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f6dec097a40 0x7f6dec132700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:58028/0 (socket says 172.21.15.88:58028) 2023-09-30T01:38:13.879 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.863+0000 7f6dea59c700 1 -- 172.21.15.88:0/1325669981 learned_addr learned my addr 172.21.15.88:0/1325669981 (peer_addr_for_me v1:172.21.15.88:0/0) 2023-09-30T01:38:13.879 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.863+0000 7f6de9d9b700 1 --2- 172.21.15.88:0/1325669981 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f6dec132e10 0x7f6dec138650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:38:13.879 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.863+0000 7f6de9d9b700 10 monclient(hunting): get_auth_request con 0x7f6dec132e10 auth_method 0 2023-09-30T01:38:13.880 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.863+0000 7f6dead9d700 1 --2- 172.21.15.88:0/1325669981 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7f6dec1311c0 0x7f6dec13aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:38:13.880 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.863+0000 7f6de9d9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:38:13.880 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.863+0000 7f6de9d9b700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:38:13.880 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.863+0000 7f6de9d9b700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:38:13.880 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.863+0000 7f6dead9d700 10 monclient(hunting): get_auth_request con 0x7f6dec1311c0 auth_method 0 2023-09-30T01:38:13.880 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.863+0000 7f6dead9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:38:13.880 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.863+0000 7f6dead9d700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:38:13.880 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.863+0000 7f6dead9d700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:38:13.880 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.863+0000 7f6de9d9b700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:38:13.880 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.863+0000 7f6de9d9b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:38:13.880 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.863+0000 7f6de9d9b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:38:13.881 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.863+0000 7f6dead9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:38:13.881 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.863+0000 7f6dead9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:38:13.881 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.863+0000 7f6dead9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:38:13.881 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.863+0000 7f6de959a700 1 -- 172.21.15.88:0/1325669981 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3113971616 0 0) 0x7f6dec04bb20 con 0x7f6dec097a40 2023-09-30T01:38:13.881 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.863+0000 7f6de959a700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:38:13.881 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.863+0000 7f6de959a700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:38:13.881 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.863+0000 7f6de959a700 10 monclient(hunting): my global_id is 5238 2023-09-30T01:38:13.881 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.863+0000 7f6de959a700 1 -- 172.21.15.88:0/1325669981 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6dd4003570 con 0x7f6dec097a40 2023-09-30T01:38:13.881 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.863+0000 7f6de9d9b700 10 monclient(hunting): handle_auth_done global_id 4990 payload 274 2023-09-30T01:38:13.882 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.863+0000 7f6de9d9b700 1 -- 172.21.15.88:0/1325669981 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7f6dec1311c0 msgr2=0x7f6dec13aba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:38:13.882 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.863+0000 7f6de9d9b700 1 --2- 172.21.15.88:0/1325669981 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7f6dec1311c0 0x7f6dec13aba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:38:13.882 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.863+0000 7f6de9d9b700 1 -- 172.21.15.88:0/1325669981 >> v1:172.21.15.88:6789/0 conn(0x7f6dec097a40 legacy=0x7f6dec132700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:38:13.882 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.863+0000 7f6de9d9b700 10 monclient: _finish_hunting 0 2023-09-30T01:38:13.882 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.863+0000 7f6de9d9b700 1 monclient: found mon.noname-c 2023-09-30T01:38:13.882 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.863+0000 7f6de9d9b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:38:13.882 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.863+0000 7f6de9d9b700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.88:3302/0 2023-09-30T01:38:13.882 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.863+0000 7f6de9d9b700 1 -- 172.21.15.88:0/1325669981 --> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6dec098d40 con 0x7f6dec132e10 2023-09-30T01:38:13.882 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.863+0000 7f6de9d9b700 1 --2- 172.21.15.88:0/1325669981 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f6dec132e10 0x7f6dec138650 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f6ddc009910 tx=0x7f6ddc0091a0 comp rx=0 tx=0).ready entity=mon.4 client_cookie=ad3f9c34215481ee server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:38:13.882 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.863+0000 7f6de959a700 1 -- 172.21.15.88:0/1325669981 <== mon.4 v2:172.21.15.88:3302/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f6ddc011ab0 con 0x7f6dec132e10 2023-09-30T01:38:13.882 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.863+0000 7f6de959a700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:38:13.883 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.863+0000 7f6de959a700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T01:38:13.883 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.863+0000 7f6de959a700 10 monclient: dump: 2023-09-30T01:38:13.883 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:38:13.883 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:38:13.883 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:38:13.883 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:38:13.883 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:38:13.883 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:38:13.883 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:38:13.883 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:38:13.884 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:38:13.884 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:38:13.884 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:38:13.884 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:38:13.884 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:38:13.884 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:38:13.884 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:38:13.884 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:38:13.884 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.863+0000 7f6de959a700 10 monclient: _finish_auth 0 2023-09-30T01:38:13.884 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.863+0000 7f6de959a700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:38:13.885 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.863+0000 7f6deb59e700 5 monclient: authenticate success, global_id 4990 2023-09-30T01:38:13.885 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.863+0000 7f6deb59e700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:38:13.885 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.863+0000 7f6de959a700 1 -- 172.21.15.88:0/1325669981 <== mon.4 v2:172.21.15.88:3302/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ddc011c30 con 0x7f6dec132e10 2023-09-30T01:38:13.885 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.863+0000 7f6de959a700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:38:13.885 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.863+0000 7f6deb59e700 10 monclient: get_monmap_and_config success 2023-09-30T01:38:13.885 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.863+0000 7f6deb59e700 4 set_mon_vals no callback set 2023-09-30T01:38:13.885 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.863+0000 7f6de959a700 1 -- 172.21.15.88:0/1325669981 <== mon.4 v2:172.21.15.88:3302/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f6ddc011da0 con 0x7f6dec132e10 2023-09-30T01:38:13.886 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.863+0000 7f6de959a700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:38:13.886 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.863+0000 7f6de959a700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T01:38:13.886 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.863+0000 7f6de959a700 10 monclient: dump: 2023-09-30T01:38:13.886 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:38:13.886 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:38:13.886 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:38:13.886 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:38:13.886 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:38:13.886 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:38:13.887 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:38:13.887 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:38:13.887 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:38:13.887 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:38:13.887 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:38:13.887 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:38:13.887 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:38:13.887 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:38:13.887 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:38:13.887 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:38:13.888 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.863+0000 7f6deb59e700 10 monclient: shutdown 2023-09-30T01:38:13.888 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.863+0000 7f6deb59e700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:38:13.888 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.863+0000 7f6deb59e700 1 -- 172.21.15.88:0/1325669981 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f6dec132e10 msgr2=0x7f6dec138650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:38:13.888 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.863+0000 7f6deb59e700 1 --2- 172.21.15.88:0/1325669981 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f6dec132e10 0x7f6dec138650 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f6ddc009910 tx=0x7f6ddc0091a0 comp rx=0 tx=0).stop 2023-09-30T01:38:13.888 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.863+0000 7f6deb59e700 1 -- 172.21.15.88:0/1325669981 shutdown_connections 2023-09-30T01:38:13.888 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.863+0000 7f6deb59e700 1 --2- 172.21.15.88:0/1325669981 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7f6dec1311c0 0x7f6dec13aba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:38:13.888 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.863+0000 7f6deb59e700 1 --2- 172.21.15.88:0/1325669981 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f6dec132e10 0x7f6dec138650 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:38:13.888 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.863+0000 7f6deb59e700 1 -- 172.21.15.88:0/1325669981 >> 172.21.15.88:0/1325669981 conn(0x7f6dec06e5e0 msgr2=0x7f6dec06e9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:38:13.888 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.863+0000 7f6deb59e700 1 -- 172.21.15.88:0/1325669981 shutdown_connections 2023-09-30T01:38:13.888 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.863+0000 7f6deb59e700 1 -- 172.21.15.88:0/1325669981 wait complete. 2023-09-30T01:38:13.889 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.863+0000 7f6deb59e700 10 monclient: build_initial_monmap 2023-09-30T01:38:13.889 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.863+0000 7f6deb59e700 10 monclient: monmap: 2023-09-30T01:38:13.889 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2023-09-30T01:38:13.889 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:38:13.889 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 0.000000 2023-09-30T01:38:13.889 INFO:teuthology.orchestra.run.smithi088.stderr:created 0.000000 2023-09-30T01:38:13.889 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2023-09-30T01:38:13.889 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 1 2023-09-30T01:38:13.889 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.noname-a 2023-09-30T01:38:13.890 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.noname-f 2023-09-30T01:38:13.890 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.noname-b 2023-09-30T01:38:13.890 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.noname-g 2023-09-30T01:38:13.890 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.noname-c 2023-09-30T01:38:13.890 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.noname-h 2023-09-30T01:38:13.890 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.noname-d 2023-09-30T01:38:13.890 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.noname-i 2023-09-30T01:38:13.890 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.noname-e 2023-09-30T01:38:13.890 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:38:13.890 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.867+0000 7f6deb59e700 1 Processor -- start 2023-09-30T01:38:13.890 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.867+0000 7f6deb59e700 1 -- start start 2023-09-30T01:38:13.890 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.867+0000 7f6deb59e700 10 monclient: init 2023-09-30T01:38:13.891 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.867+0000 7f6deb59e700 10 monclient: _reopen_session rank -1 2023-09-30T01:38:13.891 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.867+0000 7f6deb59e700 10 monclient: _add_conns ranks=[5,3,2,6,8,1,7,4,0] 2023-09-30T01:38:13.891 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.867+0000 7f6deb59e700 1 --2- >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7f6dec097a40 0x7f6dec11cef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:38:13.891 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.867+0000 7f6deb59e700 10 monclient(hunting): picked mon.noname-h con 0x7f6dec097a40 addr [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] 2023-09-30T01:38:13.891 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.867+0000 7f6deb59e700 1 --2- >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7f6dec1311c0 0x7f6dec11d430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:38:13.891 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.867+0000 7f6deb59e700 10 monclient(hunting): picked mon.noname-g con 0x7f6dec1311c0 addr [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] 2023-09-30T01:38:13.891 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.867+0000 7f6deb59e700 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6dec132e10 0x7f6dec11d970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:38:13.891 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.867+0000 7f6deb59e700 10 monclient(hunting): picked mon.noname-b con 0x7f6dec132e10 addr [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] 2023-09-30T01:38:13.891 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.867+0000 7f6deb59e700 10 monclient(hunting): start opening mon connection 2023-09-30T01:38:13.891 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.867+0000 7f6deb59e700 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6dec099c30 con 0x7f6dec132e10 2023-09-30T01:38:13.891 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.867+0000 7f6deb59e700 10 monclient(hunting): start opening mon connection 2023-09-30T01:38:13.891 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.867+0000 7f6deb59e700 1 -- --> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6dec121110 con 0x7f6dec1311c0 2023-09-30T01:38:13.892 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.867+0000 7f6deb59e700 10 monclient(hunting): start opening mon connection 2023-09-30T01:38:13.892 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.867+0000 7f6deb59e700 1 -- --> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] -- mon_getmap magic: 0 v1 -- 0x7f6dec11fbe0 con 0x7f6dec097a40 2023-09-30T01:38:13.892 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.867+0000 7f6deb59e700 10 monclient(hunting): _renew_subs 2023-09-30T01:38:13.892 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.867+0000 7f6deb59e700 10 monclient(hunting): authenticate will time out at 1165.905137s 2023-09-30T01:38:13.892 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.867+0000 7f6dead9d700 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6dec132e10 0x7f6dec11d970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:38:13.892 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.867+0000 7f6dead9d700 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6dec132e10 0x7f6dec11d970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3301/0 says I am v2:172.21.15.88:53584/0 (socket says 172.21.15.88:53584) 2023-09-30T01:38:13.892 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.867+0000 7f6dead9d700 1 -- 172.21.15.88:0/777782630 learned_addr learned my addr 172.21.15.88:0/777782630 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-09-30T01:38:13.892 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.867+0000 7f6dead9d700 10 monclient(hunting): get_auth_request con 0x7f6dec132e10 auth_method 0 2023-09-30T01:38:13.892 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.867+0000 7f6dead9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:38:13.892 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.867+0000 7f6dead9d700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:38:13.892 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.867+0000 7f6dead9d700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:38:13.892 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.867+0000 7f6de9d9b700 1 --2- 172.21.15.88:0/777782630 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7f6dec1311c0 0x7f6dec11d430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:38:13.893 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.867+0000 7f6dea59c700 1 --2- 172.21.15.88:0/777782630 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7f6dec097a40 0x7f6dec11cef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:38:13.893 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.867+0000 7f6dead9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:38:13.893 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.867+0000 7f6dead9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:38:13.893 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.867+0000 7f6dead9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:38:13.893 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.867+0000 7f6de9d9b700 10 monclient(hunting): get_auth_request con 0x7f6dec1311c0 auth_method 0 2023-09-30T01:38:13.893 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.867+0000 7f6de9d9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:38:13.893 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.867+0000 7f6de9d9b700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:38:13.893 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.867+0000 7f6de9d9b700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:38:13.893 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.867+0000 7f6dea59c700 10 monclient(hunting): get_auth_request con 0x7f6dec097a40 auth_method 0 2023-09-30T01:38:13.893 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.867+0000 7f6dea59c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:38:13.893 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.867+0000 7f6dea59c700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:38:13.893 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.867+0000 7f6dea59c700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:38:13.894 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.867+0000 7f6de9d9b700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:38:13.894 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.867+0000 7f6de9d9b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:38:13.894 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.867+0000 7f6de9d9b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:38:13.894 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.867+0000 7f6dea59c700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:38:13.894 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.867+0000 7f6dea59c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:38:13.894 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.867+0000 7f6dea59c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:38:13.894 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.867+0000 7f6dead9d700 10 monclient(hunting): handle_auth_done global_id 4943 payload 824 2023-09-30T01:38:13.894 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.867+0000 7f6dead9d700 1 -- 172.21.15.88:0/777782630 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7f6dec097a40 msgr2=0x7f6dec11cef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:38:13.894 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.867+0000 7f6dead9d700 1 --2- 172.21.15.88:0/777782630 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7f6dec097a40 0x7f6dec11cef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:38:13.894 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.867+0000 7f6dead9d700 1 -- 172.21.15.88:0/777782630 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7f6dec1311c0 msgr2=0x7f6dec11d430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:38:13.894 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.867+0000 7f6dead9d700 1 --2- 172.21.15.88:0/777782630 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7f6dec1311c0 0x7f6dec11d430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:38:13.894 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.867+0000 7f6dead9d700 10 monclient: _finish_hunting 0 2023-09-30T01:38:13.895 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.867+0000 7f6dead9d700 1 monclient: found mon.noname-b 2023-09-30T01:38:13.895 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.867+0000 7f6dead9d700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:38:13.895 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.867+0000 7f6dead9d700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.88:3301/0 2023-09-30T01:38:13.895 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.867+0000 7f6dead9d700 1 -- 172.21.15.88:0/777782630 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ddc009420 con 0x7f6dec132e10 2023-09-30T01:38:13.895 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.867+0000 7f6dead9d700 1 --2- 172.21.15.88:0/777782630 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6dec132e10 0x7f6dec11d970 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f6de4009e90 tx=0x7f6de400b040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:38:13.895 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.867+0000 7f6dda7fc700 1 -- 172.21.15.88:0/777782630 <== mon.2 v2:172.21.15.88:3301/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f6de4014480 con 0x7f6dec132e10 2023-09-30T01:38:13.895 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.867+0000 7f6dda7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:38:13.895 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.867+0000 7f6dda7fc700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T01:38:13.895 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.867+0000 7f6dda7fc700 10 monclient: dump: 2023-09-30T01:38:13.895 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:38:13.895 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:38:13.895 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:38:13.896 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:38:13.896 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:38:13.896 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:38:13.896 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:38:13.896 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:38:13.896 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:38:13.896 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:38:13.896 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:38:13.896 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:38:13.896 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:38:13.896 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:38:13.897 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:38:13.897 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:38:13.897 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.867+0000 7f6dda7fc700 10 monclient: _finish_auth 0 2023-09-30T01:38:13.897 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.867+0000 7f6dda7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:38:13.897 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.867+0000 7f6dda7fc700 1 -- 172.21.15.88:0/777782630 <== mon.2 v2:172.21.15.88:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6de400f040 con 0x7f6dec132e10 2023-09-30T01:38:13.897 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.867+0000 7f6deb59e700 5 monclient: authenticate success, global_id 4943 2023-09-30T01:38:13.897 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.867+0000 7f6deb59e700 10 monclient: _renew_subs 2023-09-30T01:38:13.897 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.867+0000 7f6deb59e700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.88:3301/0 2023-09-30T01:38:13.897 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.867+0000 7f6deb59e700 1 -- 172.21.15.88:0/777782630 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6dec0996b0 con 0x7f6dec132e10 2023-09-30T01:38:13.898 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.867+0000 7f6dda7fc700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:38:13.898 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.867+0000 7f6dda7fc700 1 -- 172.21.15.88:0/777782630 <== mon.2 v2:172.21.15.88:3301/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f6de4014aa0 con 0x7f6dec132e10 2023-09-30T01:38:13.898 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.867+0000 7f6dda7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:38:13.898 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.867+0000 7f6dda7fc700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T01:38:13.898 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.867+0000 7f6dda7fc700 10 monclient: dump: 2023-09-30T01:38:13.898 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:38:13.898 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:38:13.898 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:38:13.899 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:38:13.899 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:38:13.899 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:38:13.899 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:38:13.899 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:38:13.899 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:38:13.899 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:38:13.899 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:38:13.900 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:38:13.900 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:38:13.900 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:38:13.900 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:38:13.900 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:38:13.900 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.867+0000 7f6deb59e700 10 monclient: _renew_subs 2023-09-30T01:38:13.901 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.867+0000 7f6deb59e700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.88:3301/0 2023-09-30T01:38:13.901 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.867+0000 7f6deb59e700 1 -- 172.21.15.88:0/777782630 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6dec11e350 con 0x7f6dec132e10 2023-09-30T01:38:13.901 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.867+0000 7f6ddbfff700 4 set_mon_vals no callback set 2023-09-30T01:38:13.901 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.867+0000 7f6deb59e700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:38:13.901 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.867+0000 7f6deb59e700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:38:13.901 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.867+0000 7f6deb59e700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.88:3301/0 2023-09-30T01:38:13.901 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.867+0000 7f6deb59e700 1 -- 172.21.15.88:0/777782630 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6dec09e720 con 0x7f6dec132e10 2023-09-30T01:38:13.902 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.871+0000 7f6dda7fc700 1 -- 172.21.15.88:0/777782630 <== mon.2 v2:172.21.15.88:3301/0 4 ==== mgrmap(e 5) v1 ==== 40144+0+0 (secure 0 0 0) 0x7f6de4014c00 con 0x7f6dec132e10 2023-09-30T01:38:13.902 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.871+0000 7f6dda7fc700 1 --2- 172.21.15.88:0/777782630 >> v2:172.21.15.107:6812/29926 conn(0x7f6dcc033360 0x7f6dcc035810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:38:13.902 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.871+0000 7f6dda7fc700 1 -- 172.21.15.88:0/777782630 <== mon.2 v2:172.21.15.88:3301/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 3502+0+0 (secure 0 0 0) 0x7f6de4043fd0 con 0x7f6dec132e10 2023-09-30T01:38:13.902 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.871+0000 7f6dea59c700 1 --2- 172.21.15.88:0/777782630 >> v2:172.21.15.107:6812/29926 conn(0x7f6dcc033360 0x7f6dcc035810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:38:13.902 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.871+0000 7f6dea59c700 10 monclient: get_auth_request con 0x7f6dcc033360 auth_method 0 2023-09-30T01:38:13.902 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.871+0000 7f6dea59c700 1 --2- 172.21.15.88:0/777782630 >> v2:172.21.15.107:6812/29926 conn(0x7f6dcc033360 0x7f6dcc035810 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f6de0007fd0 tx=0x7f6de0009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:38:13.903 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.875+0000 7f6dda7fc700 1 -- 172.21.15.88:0/777782630 <== mon.2 v2:172.21.15.88:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f6de400caa0 con 0x7f6dec132e10 2023-09-30T01:38:13.903 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.875+0000 7f6dda7fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:38:13.903 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:13.875+0000 7f6dda7fc700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:38:14.037 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.035+0000 7f6deb59e700 1 -- 172.21.15.88:0/777782630 --> v2:172.21.15.107:6812/29926 -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "4", "target": ["mon-mgr", ""]}) v1 -- 0x7f6dec071960 con 0x7f6dcc033360 2023-09-30T01:38:14.038 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.035+0000 7f6dda7fc700 1 -- 172.21.15.88:0/777782630 <== mgr.4108 v2:172.21.15.107:6812/29926 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 4 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f6dec071960 con 0x7f6dcc033360 2023-09-30T01:38:14.040 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.039+0000 7f6deb59e700 1 -- 172.21.15.88:0/777782630 >> v2:172.21.15.107:6812/29926 conn(0x7f6dcc033360 msgr2=0x7f6dcc035810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:38:14.040 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.039+0000 7f6deb59e700 1 --2- 172.21.15.88:0/777782630 >> v2:172.21.15.107:6812/29926 conn(0x7f6dcc033360 0x7f6dcc035810 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f6de0007fd0 tx=0x7f6de0009040 comp rx=0 tx=0).stop 2023-09-30T01:38:14.041 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.039+0000 7f6deb59e700 10 monclient: shutdown 2023-09-30T01:38:14.041 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.039+0000 7f6deb59e700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:38:14.041 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.039+0000 7f6deb59e700 1 -- 172.21.15.88:0/777782630 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6dec132e10 msgr2=0x7f6dec11d970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:38:14.041 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.039+0000 7f6deb59e700 1 --2- 172.21.15.88:0/777782630 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6dec132e10 0x7f6dec11d970 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f6de4009e90 tx=0x7f6de400b040 comp rx=0 tx=0).stop 2023-09-30T01:38:14.041 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.039+0000 7f6deb59e700 1 -- 172.21.15.88:0/777782630 shutdown_connections 2023-09-30T01:38:14.041 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.039+0000 7f6deb59e700 1 --2- 172.21.15.88:0/777782630 >> v2:172.21.15.107:6812/29926 conn(0x7f6dcc033360 0x7f6dcc035810 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:38:14.041 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.039+0000 7f6deb59e700 1 --2- 172.21.15.88:0/777782630 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6dec132e10 0x7f6dec11d970 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:38:14.041 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.039+0000 7f6deb59e700 1 --2- 172.21.15.88:0/777782630 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7f6dec1311c0 0x7f6dec11d430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:38:14.041 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.039+0000 7f6deb59e700 1 --2- 172.21.15.88:0/777782630 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7f6dec097a40 0x7f6dec11cef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:38:14.041 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.039+0000 7f6deb59e700 1 -- 172.21.15.88:0/777782630 >> 172.21.15.88:0/777782630 conn(0x7f6dec06e5e0 msgr2=0x7f6dec1339c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:38:14.042 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.039+0000 7f6deb59e700 1 -- 172.21.15.88:0/777782630 shutdown_connections 2023-09-30T01:38:14.042 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.039+0000 7f6deb59e700 1 -- 172.21.15.88:0/777782630 wait complete. 2023-09-30T01:38:14.042 INFO:teuthology.orchestra.run.smithi088.stderr:instructed osd(s) 4 to deep-scrub 2023-09-30T01:38:14.051 INFO:tasks.ceph:Scrubbing osd.5 2023-09-30T01:38:14.052 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 config set osd_debug_deep_scrub_sleep 0 2023-09-30T01:38:14.159 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.155+0000 7fec4e6b5700 10 monclient: get_monmap_and_config 2023-09-30T01:38:14.159 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.155+0000 7fec4e6b5700 10 monclient: build_initial_monmap 2023-09-30T01:38:14.159 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.155+0000 7fec4e6b5700 10 monclient: monmap: 2023-09-30T01:38:14.160 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2023-09-30T01:38:14.160 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:38:14.160 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:38:14.159385+0000 2023-09-30T01:38:14.160 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:38:14.159385+0000 2023-09-30T01:38:14.160 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2023-09-30T01:38:14.160 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:38:14.160 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a-legacy 2023-09-30T01:38:14.160 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.107:6789/0 mon.noname-f-legacy 2023-09-30T01:38:14.160 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.noname-b 2023-09-30T01:38:14.160 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.noname-g 2023-09-30T01:38:14.160 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.noname-c 2023-09-30T01:38:14.160 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.noname-h 2023-09-30T01:38:14.161 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.noname-d 2023-09-30T01:38:14.161 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.noname-i 2023-09-30T01:38:14.161 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.noname-e 2023-09-30T01:38:14.161 INFO:teuthology.orchestra.run.smithi088.stderr:9: v2:172.21.15.88:3300/0 mon.noname-a 2023-09-30T01:38:14.161 INFO:teuthology.orchestra.run.smithi088.stderr:10: v2:172.21.15.107:3300/0 mon.noname-f 2023-09-30T01:38:14.161 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:38:14.166 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.159+0000 7fec4e6b5700 1 Processor -- start 2023-09-30T01:38:14.166 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.159+0000 7fec4e6b5700 1 -- start start 2023-09-30T01:38:14.166 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.159+0000 7fec4e6b5700 10 monclient: init 2023-09-30T01:38:14.166 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.159+0000 7fec4e6b5700 10 monclient: _reopen_session rank -1 2023-09-30T01:38:14.166 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.159+0000 7fec4e6b5700 10 monclient: _add_conns ranks=[2,0,4,1,8,10,3,7,6,9,5] 2023-09-30T01:38:14.166 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.159+0000 7fec4e6b5700 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fec4809d1b0 0x7fec48099dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:38:14.167 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.159+0000 7fec4e6b5700 10 monclient(hunting): picked mon.noname-b con 0x7fec4809d1b0 addr [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] 2023-09-30T01:38:14.167 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.159+0000 7fec4e6b5700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fec4809dbf0 addr v1:172.21.15.88:6789/0 2023-09-30T01:38:14.167 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.159+0000 7fec4e6b5700 1 --2- >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7fec4809e6d0 0x7fec48137c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:38:14.167 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.159+0000 7fec4e6b5700 10 monclient(hunting): picked mon.noname-c con 0x7fec4809e6d0 addr [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] 2023-09-30T01:38:14.167 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.159+0000 7fec4e6b5700 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec4804bb20 con 0x7fec4809dbf0 2023-09-30T01:38:14.167 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.159+0000 7fec4e6b5700 10 monclient(hunting): start opening mon connection 2023-09-30T01:38:14.167 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.159+0000 7fec4e6b5700 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec4809a780 con 0x7fec4809d1b0 2023-09-30T01:38:14.167 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.159+0000 7fec4e6b5700 10 monclient(hunting): start opening mon connection 2023-09-30T01:38:14.167 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.159+0000 7fec4e6b5700 1 -- --> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] -- mon_getmap magic: 0 v1 -- 0x7fec480993f0 con 0x7fec4809e6d0 2023-09-30T01:38:14.167 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.159+0000 7fec4e6b5700 10 monclient(hunting): _renew_subs 2023-09-30T01:38:14.167 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.159+0000 7fec4e6b5700 10 monclient(hunting): authenticate will time out at 1166.197665s 2023-09-30T01:38:14.167 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.159+0000 7fec4d6b3700 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fec4809d1b0 0x7fec48099dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:38:14.168 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.159+0000 7fec4deb4700 1 --2- >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7fec4809e6d0 0x7fec48137c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:38:14.168 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.159+0000 7fec4d6b3700 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fec4809d1b0 0x7fec48099dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3301/0 says I am v2:172.21.15.88:53586/0 (socket says 172.21.15.88:53586) 2023-09-30T01:38:14.168 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.159+0000 7fec4d6b3700 1 -- 172.21.15.88:0/1895449362 learned_addr learned my addr 172.21.15.88:0/1895449362 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-09-30T01:38:14.168 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.159+0000 7fec4d6b3700 10 monclient(hunting): get_auth_request con 0x7fec4809d1b0 auth_method 0 2023-09-30T01:38:14.168 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.159+0000 7fec4d6b3700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:38:14.168 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.159+0000 7fec4d6b3700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:38:14.168 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.159+0000 7fec4d6b3700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:38:14.168 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.159+0000 7fec4deb4700 10 monclient(hunting): get_auth_request con 0x7fec4809e6d0 auth_method 0 2023-09-30T01:38:14.168 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.159+0000 7fec4deb4700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:38:14.168 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.159+0000 7fec4deb4700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:38:14.168 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.159+0000 7fec4deb4700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:38:14.168 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.159+0000 7fec4d6b3700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:38:14.169 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.159+0000 7fec4d6b3700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:38:14.169 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.159+0000 7fec4d6b3700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:38:14.169 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.159+0000 7fec4deb4700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:38:14.169 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.159+0000 7fec4deb4700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:38:14.169 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.159+0000 7fec4deb4700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:38:14.169 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.159+0000 7fec47fff700 1 -- 172.21.15.88:0/1895449362 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2937429520 0 0) 0x7fec4804bb20 con 0x7fec4809dbf0 2023-09-30T01:38:14.169 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.159+0000 7fec4d6b3700 10 monclient(hunting): handle_auth_done global_id 4952 payload 274 2023-09-30T01:38:14.169 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.159+0000 7fec4d6b3700 1 -- 172.21.15.88:0/1895449362 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7fec4809e6d0 msgr2=0x7fec48137c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:38:14.169 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.159+0000 7fec4d6b3700 1 --2- 172.21.15.88:0/1895449362 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7fec4809e6d0 0x7fec48137c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:38:14.169 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.159+0000 7fec4d6b3700 1 -- 172.21.15.88:0/1895449362 >> v1:172.21.15.88:6789/0 conn(0x7fec4809dbf0 legacy=0x7fec4809dfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:38:14.169 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.159+0000 7fec4d6b3700 10 monclient: _finish_hunting 0 2023-09-30T01:38:14.169 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.159+0000 7fec4d6b3700 1 monclient: found mon.noname-b 2023-09-30T01:38:14.170 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.159+0000 7fec4d6b3700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:38:14.170 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.159+0000 7fec4d6b3700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.88:3301/0 2023-09-30T01:38:14.170 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.159+0000 7fec4d6b3700 1 -- 172.21.15.88:0/1895449362 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec4809f0d0 con 0x7fec4809d1b0 2023-09-30T01:38:14.170 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.159+0000 7fec47fff700 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2023-09-30T01:38:14.170 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.159+0000 7fec4deb4700 1 --2- 172.21.15.88:0/1895449362 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7fec4809e6d0 0x7fec48137c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:38:14.170 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.159+0000 7fec4d6b3700 1 --2- 172.21.15.88:0/1895449362 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fec4809d1b0 0x7fec48099dd0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fec38009f50 tx=0x7fec38009840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7c92119f413f7ea1 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:38:14.170 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.159+0000 7fec47fff700 1 -- 172.21.15.88:0/1895449362 <== mon.2 v2:172.21.15.88:3301/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7fec380092e0 con 0x7fec4809d1b0 2023-09-30T01:38:14.170 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.159+0000 7fec47fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:38:14.170 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.159+0000 7fec47fff700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T01:38:14.171 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.159+0000 7fec47fff700 10 monclient: dump: 2023-09-30T01:38:14.171 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:38:14.171 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:38:14.171 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:38:14.171 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:38:14.171 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:38:14.171 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:38:14.171 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:38:14.171 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:38:14.171 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:38:14.172 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:38:14.172 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:38:14.172 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:38:14.172 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:38:14.172 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:38:14.172 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:38:14.172 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:38:14.172 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.159+0000 7fec47fff700 10 monclient: _finish_auth 0 2023-09-30T01:38:14.172 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.159+0000 7fec47fff700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:38:14.173 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.159+0000 7fec4e6b5700 5 monclient: authenticate success, global_id 4952 2023-09-30T01:38:14.173 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.159+0000 7fec4e6b5700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:38:14.173 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.159+0000 7fec47fff700 1 -- 172.21.15.88:0/1895449362 <== mon.2 v2:172.21.15.88:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec3800be80 con 0x7fec4809d1b0 2023-09-30T01:38:14.173 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.159+0000 7fec47fff700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:38:14.173 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.159+0000 7fec4e6b5700 10 monclient: get_monmap_and_config success 2023-09-30T01:38:14.173 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.159+0000 7fec4e6b5700 4 set_mon_vals no callback set 2023-09-30T01:38:14.173 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.159+0000 7fec47fff700 1 -- 172.21.15.88:0/1895449362 <== mon.2 v2:172.21.15.88:3301/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7fec380118e0 con 0x7fec4809d1b0 2023-09-30T01:38:14.173 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.159+0000 7fec47fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:38:14.173 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.159+0000 7fec47fff700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T01:38:14.173 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.159+0000 7fec47fff700 10 monclient: dump: 2023-09-30T01:38:14.173 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:38:14.174 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:38:14.174 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:38:14.174 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:38:14.174 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:38:14.174 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:38:14.174 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:38:14.175 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:38:14.175 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:38:14.175 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:38:14.175 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:38:14.175 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:38:14.175 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:38:14.176 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:38:14.176 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:38:14.176 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:38:14.176 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.159+0000 7fec4e6b5700 10 monclient: shutdown 2023-09-30T01:38:14.176 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.159+0000 7fec4e6b5700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:38:14.176 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.159+0000 7fec4e6b5700 1 -- 172.21.15.88:0/1895449362 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fec4809d1b0 msgr2=0x7fec48099dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:38:14.176 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.159+0000 7fec4e6b5700 1 --2- 172.21.15.88:0/1895449362 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fec4809d1b0 0x7fec48099dd0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fec38009f50 tx=0x7fec38009840 comp rx=0 tx=0).stop 2023-09-30T01:38:14.176 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.159+0000 7fec4e6b5700 1 -- 172.21.15.88:0/1895449362 shutdown_connections 2023-09-30T01:38:14.176 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.159+0000 7fec4e6b5700 1 --2- 172.21.15.88:0/1895449362 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7fec4809e6d0 0x7fec48137c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:38:14.176 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.159+0000 7fec4e6b5700 1 --2- 172.21.15.88:0/1895449362 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fec4809d1b0 0x7fec48099dd0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:38:14.177 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.159+0000 7fec4e6b5700 1 -- 172.21.15.88:0/1895449362 >> 172.21.15.88:0/1895449362 conn(0x7fec4809fe10 msgr2=0x7fec480a0210 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:38:14.177 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.159+0000 7fec4e6b5700 1 -- 172.21.15.88:0/1895449362 shutdown_connections 2023-09-30T01:38:14.177 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.159+0000 7fec4e6b5700 1 -- 172.21.15.88:0/1895449362 wait complete. 2023-09-30T01:38:14.177 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.159+0000 7fec4e6b5700 10 monclient: build_initial_monmap 2023-09-30T01:38:14.177 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.159+0000 7fec4e6b5700 10 monclient: monmap: 2023-09-30T01:38:14.177 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2023-09-30T01:38:14.177 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:38:14.177 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 0.000000 2023-09-30T01:38:14.177 INFO:teuthology.orchestra.run.smithi088.stderr:created 0.000000 2023-09-30T01:38:14.177 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2023-09-30T01:38:14.177 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 1 2023-09-30T01:38:14.178 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.noname-a 2023-09-30T01:38:14.178 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.noname-f 2023-09-30T01:38:14.178 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.noname-b 2023-09-30T01:38:14.178 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.noname-g 2023-09-30T01:38:14.178 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.noname-c 2023-09-30T01:38:14.178 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.noname-h 2023-09-30T01:38:14.178 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.noname-d 2023-09-30T01:38:14.178 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.noname-i 2023-09-30T01:38:14.178 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.noname-e 2023-09-30T01:38:14.178 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:38:14.178 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.163+0000 7fec4e6b5700 1 Processor -- start 2023-09-30T01:38:14.179 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.163+0000 7fec4e6b5700 1 -- start start 2023-09-30T01:38:14.179 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.163+0000 7fec4e6b5700 10 monclient: init 2023-09-30T01:38:14.179 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.163+0000 7fec4e6b5700 10 monclient: _reopen_session rank -1 2023-09-30T01:38:14.179 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.163+0000 7fec4e6b5700 10 monclient: _add_conns ranks=[7,6,8,2,0,4,1,5,3] 2023-09-30T01:38:14.179 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.163+0000 7fec4e6b5700 1 --2- >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7fec4809d1b0 0x7fec4806e030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:38:14.179 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.163+0000 7fec4e6b5700 10 monclient(hunting): picked mon.noname-i con 0x7fec4809d1b0 addr [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] 2023-09-30T01:38:14.179 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.163+0000 7fec4e6b5700 1 --2- >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7fec4809dbf0 0x7fec4812f1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:38:14.179 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.163+0000 7fec4e6b5700 10 monclient(hunting): picked mon.noname-d con 0x7fec4809dbf0 addr [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] 2023-09-30T01:38:14.179 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.163+0000 7fec4e6b5700 1 --2- >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7fec4809e6d0 0x7fec4812f730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:38:14.179 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.163+0000 7fec4e6b5700 10 monclient(hunting): picked mon.noname-e con 0x7fec4809e6d0 addr [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] 2023-09-30T01:38:14.179 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.163+0000 7fec4e6b5700 10 monclient(hunting): start opening mon connection 2023-09-30T01:38:14.180 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.163+0000 7fec4e6b5700 1 -- --> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] -- mon_getmap magic: 0 v1 -- 0x7fec48098930 con 0x7fec4809dbf0 2023-09-30T01:38:14.180 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.163+0000 7fec4e6b5700 10 monclient(hunting): start opening mon connection 2023-09-30T01:38:14.180 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.163+0000 7fec4e6b5700 1 -- --> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] -- mon_getmap magic: 0 v1 -- 0x7fec48098460 con 0x7fec4809d1b0 2023-09-30T01:38:14.180 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.163+0000 7fec4e6b5700 10 monclient(hunting): start opening mon connection 2023-09-30T01:38:14.180 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.163+0000 7fec4e6b5700 1 -- --> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] -- mon_getmap magic: 0 v1 -- 0x7fec4813f3c0 con 0x7fec4809e6d0 2023-09-30T01:38:14.180 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.163+0000 7fec4e6b5700 10 monclient(hunting): _renew_subs 2023-09-30T01:38:14.180 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.163+0000 7fec4e6b5700 10 monclient(hunting): authenticate will time out at 1166.200819s 2023-09-30T01:38:14.180 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.163+0000 7fec4deb4700 1 --2- >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7fec4809e6d0 0x7fec4812f730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:38:14.180 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.163+0000 7fec4deb4700 1 --2- >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7fec4809e6d0 0x7fec4812f730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3304/0 says I am v2:172.21.15.88:35742/0 (socket says 172.21.15.88:35742) 2023-09-30T01:38:14.180 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.163+0000 7fec4deb4700 1 -- 172.21.15.88:0/740341207 learned_addr learned my addr 172.21.15.88:0/740341207 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-09-30T01:38:14.180 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.163+0000 7fec4deb4700 10 monclient(hunting): get_auth_request con 0x7fec4809e6d0 auth_method 0 2023-09-30T01:38:14.180 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.163+0000 7fec4deb4700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:38:14.181 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.163+0000 7fec4deb4700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:38:14.181 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.163+0000 7fec4deb4700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:38:14.181 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.163+0000 7fec4ceb2700 1 --2- 172.21.15.88:0/740341207 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7fec4809dbf0 0x7fec4812f1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:38:14.181 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.163+0000 7fec4ceb2700 10 monclient(hunting): get_auth_request con 0x7fec4809dbf0 auth_method 0 2023-09-30T01:38:14.181 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.163+0000 7fec4ceb2700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:38:14.181 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.163+0000 7fec4ceb2700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:38:14.181 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.163+0000 7fec4ceb2700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:38:14.181 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.163+0000 7fec4deb4700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:38:14.182 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.163+0000 7fec4deb4700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:38:14.182 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.163+0000 7fec4deb4700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:38:14.182 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.163+0000 7fec4d6b3700 1 --2- 172.21.15.88:0/740341207 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7fec4809d1b0 0x7fec4806e030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:38:14.182 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.163+0000 7fec4ceb2700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:38:14.182 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.163+0000 7fec4ceb2700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:38:14.182 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.163+0000 7fec4ceb2700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:38:14.182 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.163+0000 7fec4d6b3700 10 monclient(hunting): get_auth_request con 0x7fec4809d1b0 auth_method 0 2023-09-30T01:38:14.182 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.163+0000 7fec4d6b3700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:38:14.182 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.163+0000 7fec4d6b3700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:38:14.182 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.163+0000 7fec4d6b3700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:38:14.182 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.163+0000 7fec4deb4700 10 monclient(hunting): handle_auth_done global_id 4931 payload 824 2023-09-30T01:38:14.183 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.163+0000 7fec4deb4700 1 -- 172.21.15.88:0/740341207 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7fec4809d1b0 msgr2=0x7fec4806e030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:38:14.183 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.163+0000 7fec4deb4700 1 --2- 172.21.15.88:0/740341207 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7fec4809d1b0 0x7fec4806e030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:38:14.183 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.163+0000 7fec4deb4700 1 -- 172.21.15.88:0/740341207 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7fec4809dbf0 msgr2=0x7fec4812f1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:38:14.183 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.163+0000 7fec4deb4700 1 --2- 172.21.15.88:0/740341207 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7fec4809dbf0 0x7fec4812f1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:38:14.183 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.163+0000 7fec4deb4700 10 monclient: _finish_hunting 0 2023-09-30T01:38:14.183 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.163+0000 7fec4deb4700 1 monclient: found mon.noname-e 2023-09-30T01:38:14.183 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.163+0000 7fec4deb4700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:38:14.183 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.163+0000 7fec4deb4700 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.88:3304/0 2023-09-30T01:38:14.183 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.163+0000 7fec4deb4700 1 -- 172.21.15.88:0/740341207 --> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec40009420 con 0x7fec4809e6d0 2023-09-30T01:38:14.183 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.163+0000 7fec4deb4700 1 --2- 172.21.15.88:0/740341207 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7fec4809e6d0 0x7fec4812f730 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fec4000ec50 tx=0x7fec400091a0 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:38:14.183 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.163+0000 7fec457fa700 1 -- 172.21.15.88:0/740341207 <== mon.8 v2:172.21.15.88:3304/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7fec4000ccd0 con 0x7fec4809e6d0 2023-09-30T01:38:14.184 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.163+0000 7fec457fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:38:14.184 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.163+0000 7fec457fa700 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2023-09-30T01:38:14.184 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.163+0000 7fec457fa700 10 monclient: dump: 2023-09-30T01:38:14.184 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:38:14.184 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:38:14.184 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:38:14.184 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:38:14.184 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:38:14.184 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:38:14.184 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:38:14.184 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:38:14.185 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:38:14.185 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:38:14.185 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:38:14.185 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:38:14.185 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:38:14.185 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:38:14.185 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:38:14.185 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:38:14.185 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.163+0000 7fec457fa700 10 monclient: _finish_auth 0 2023-09-30T01:38:14.185 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.163+0000 7fec457fa700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:38:14.185 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.163+0000 7fec4e6b5700 5 monclient: authenticate success, global_id 4931 2023-09-30T01:38:14.185 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.163+0000 7fec4e6b5700 10 monclient: _renew_subs 2023-09-30T01:38:14.186 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.163+0000 7fec4e6b5700 10 monclient: _send_mon_message to mon.e at v2:172.21.15.88:3304/0 2023-09-30T01:38:14.186 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.163+0000 7fec4e6b5700 1 -- 172.21.15.88:0/740341207 --> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec38009ae0 con 0x7fec4809e6d0 2023-09-30T01:38:14.186 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.163+0000 7fec457fa700 1 -- 172.21.15.88:0/740341207 <== mon.8 v2:172.21.15.88:3304/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec4000ce50 con 0x7fec4809e6d0 2023-09-30T01:38:14.186 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.163+0000 7fec457fa700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:38:14.186 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.163+0000 7fec4e6b5700 10 monclient: _renew_subs 2023-09-30T01:38:14.186 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.163+0000 7fec457fa700 1 -- 172.21.15.88:0/740341207 <== mon.8 v2:172.21.15.88:3304/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7fec400138f0 con 0x7fec4809e6d0 2023-09-30T01:38:14.186 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.163+0000 7fec4e6b5700 10 monclient: _send_mon_message to mon.e at v2:172.21.15.88:3304/0 2023-09-30T01:38:14.186 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.163+0000 7fec4e6b5700 1 -- 172.21.15.88:0/740341207 --> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec4806ec00 con 0x7fec4809e6d0 2023-09-30T01:38:14.186 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.163+0000 7fec457fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:38:14.186 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.163+0000 7fec477fe700 4 set_mon_vals no callback set 2023-09-30T01:38:14.187 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.163+0000 7fec457fa700 10 monclient: got monmap 1 from mon.e (according to old e1) 2023-09-30T01:38:14.187 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.163+0000 7fec457fa700 10 monclient: dump: 2023-09-30T01:38:14.187 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:38:14.187 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:38:14.187 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:38:14.187 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:38:14.187 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:38:14.187 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:38:14.187 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:38:14.187 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:38:14.187 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:38:14.187 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:38:14.188 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:38:14.188 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:38:14.188 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:38:14.188 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:38:14.188 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:38:14.188 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:38:14.190 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.163+0000 7fec4e6b5700 10 monclient: _send_mon_message to mon.e at v2:172.21.15.88:3304/0 2023-09-30T01:38:14.190 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.163+0000 7fec4e6b5700 1 -- 172.21.15.88:0/740341207 --> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fec4811c400 con 0x7fec4809e6d0 2023-09-30T01:38:14.190 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.163+0000 7fec457fa700 1 -- 172.21.15.88:0/740341207 <== mon.8 v2:172.21.15.88:3304/0 4 ==== mgrmap(e 5) v1 ==== 40144+0+0 (secure 0 0 0) 0x7fec40013470 con 0x7fec4809e6d0 2023-09-30T01:38:14.190 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.167+0000 7fec457fa700 1 --2- 172.21.15.88:0/740341207 >> v2:172.21.15.107:6812/29926 conn(0x7fec34033290 0x7fec34035740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:38:14.190 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.167+0000 7fec457fa700 1 -- 172.21.15.88:0/740341207 <== mon.8 v2:172.21.15.88:3304/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 3502+0+0 (secure 0 0 0) 0x7fec40043690 con 0x7fec4809e6d0 2023-09-30T01:38:14.191 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.167+0000 7fec457fa700 1 --2- 172.21.15.88:0/740341207 >> v2:172.21.15.107:6800/30275 conn(0x7fec34038d30 0x7fec3403b140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:38:14.191 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.167+0000 7fec457fa700 1 -- 172.21.15.88:0/740341207 --> v2:172.21.15.107:6800/30275 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fec3403b7b0 con 0x7fec34038d30 2023-09-30T01:38:14.191 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.167+0000 7fec457fa700 1 -- 172.21.15.88:0/740341207 <== mon.8 v2:172.21.15.88:3304/0 6 ==== mon_get_version_reply(handle=1 version=15) v2 ==== 24+0+0 (secure 0 0 0) 0x7fec40045d20 con 0x7fec4809e6d0 2023-09-30T01:38:14.191 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.167+0000 7fec457fa700 10 monclient: handle_get_version_reply finishing 1 version 15 2023-09-30T01:38:14.191 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.167+0000 7fec4d6b3700 1 --2- 172.21.15.88:0/740341207 >> v2:172.21.15.107:6812/29926 conn(0x7fec34033290 0x7fec34035740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:38:14.191 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.167+0000 7fec4d6b3700 10 monclient: get_auth_request con 0x7fec34033290 auth_method 0 2023-09-30T01:38:14.191 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.167+0000 7fec4ceb2700 1 --2- 172.21.15.88:0/740341207 >> v2:172.21.15.107:6800/30275 conn(0x7fec34038d30 0x7fec3403b140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:38:14.191 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.167+0000 7fec4ceb2700 10 monclient: get_auth_request con 0x7fec34038d30 auth_method 0 2023-09-30T01:38:14.191 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.167+0000 7fec4d6b3700 1 --2- 172.21.15.88:0/740341207 >> v2:172.21.15.107:6812/29926 conn(0x7fec34033290 0x7fec34035740 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fec38009f20 tx=0x7fec3801a000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:38:14.192 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.167+0000 7fec4ceb2700 1 --2- 172.21.15.88:0/740341207 >> v2:172.21.15.107:6800/30275 conn(0x7fec34038d30 0x7fec3403b140 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:38:14.192 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.167+0000 7fec457fa700 1 -- 172.21.15.88:0/740341207 <== osd.5 v2:172.21.15.107:6800/30275 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23999 (crc 0 0 0) 0x7fec3403b7b0 con 0x7fec34038d30 2023-09-30T01:38:14.192 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.187+0000 7fec4e6b5700 1 -- 172.21.15.88:0/740341207 --> v2:172.21.15.107:6800/30275 -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7fec4811c960 con 0x7fec34038d30 2023-09-30T01:38:14.192 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.187+0000 7fec457fa700 1 -- 172.21.15.88:0/740341207 <== osd.5 v2:172.21.15.107:6800/30275 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7fec4811c960 con 0x7fec34038d30 2023-09-30T01:38:14.192 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-09-30T01:38:14.193 INFO:teuthology.orchestra.run.smithi088.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-09-30T01:38:14.193 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-09-30T01:38:14.194 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.187+0000 7fec4e6b5700 1 -- 172.21.15.88:0/740341207 >> v2:172.21.15.107:6800/30275 conn(0x7fec34038d30 msgr2=0x7fec3403b140 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:38:14.194 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.187+0000 7fec4e6b5700 1 --2- 172.21.15.88:0/740341207 >> v2:172.21.15.107:6800/30275 conn(0x7fec34038d30 0x7fec3403b140 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:38:14.194 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.187+0000 7fec4e6b5700 1 -- 172.21.15.88:0/740341207 >> v2:172.21.15.107:6812/29926 conn(0x7fec34033290 msgr2=0x7fec34035740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:38:14.194 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.187+0000 7fec4e6b5700 1 --2- 172.21.15.88:0/740341207 >> v2:172.21.15.107:6812/29926 conn(0x7fec34033290 0x7fec34035740 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fec38009f20 tx=0x7fec3801a000 comp rx=0 tx=0).stop 2023-09-30T01:38:14.194 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.187+0000 7fec4e6b5700 10 monclient: shutdown 2023-09-30T01:38:14.195 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.187+0000 7fec4e6b5700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:38:14.195 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.187+0000 7fec4e6b5700 1 -- 172.21.15.88:0/740341207 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7fec4809e6d0 msgr2=0x7fec4812f730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:38:14.195 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.187+0000 7fec4e6b5700 1 --2- 172.21.15.88:0/740341207 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7fec4809e6d0 0x7fec4812f730 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fec4000ec50 tx=0x7fec400091a0 comp rx=0 tx=0).stop 2023-09-30T01:38:14.195 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.191+0000 7fec4deb4700 1 -- 172.21.15.88:0/740341207 reap_dead start 2023-09-30T01:38:14.195 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.191+0000 7fec4e6b5700 1 -- 172.21.15.88:0/740341207 shutdown_connections 2023-09-30T01:38:14.195 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.191+0000 7fec4e6b5700 1 -- 172.21.15.88:0/740341207 >> 172.21.15.88:0/740341207 conn(0x7fec4809fe10 msgr2=0x7fec480a12a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:38:14.195 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.191+0000 7fec4e6b5700 1 -- 172.21.15.88:0/740341207 shutdown_connections 2023-09-30T01:38:14.195 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.191+0000 7fec4e6b5700 1 -- 172.21.15.88:0/740341207 wait complete. 2023-09-30T01:38:14.204 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 5 2023-09-30T01:38:14.312 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.311+0000 7fa0908d7700 10 monclient: get_monmap_and_config 2023-09-30T01:38:14.313 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.311+0000 7fa0908d7700 10 monclient: build_initial_monmap 2023-09-30T01:38:14.313 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.311+0000 7fa0908d7700 10 monclient: monmap: 2023-09-30T01:38:14.313 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2023-09-30T01:38:14.313 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:38:14.313 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:38:14.312623+0000 2023-09-30T01:38:14.313 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:38:14.312623+0000 2023-09-30T01:38:14.313 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2023-09-30T01:38:14.313 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:38:14.313 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a-legacy 2023-09-30T01:38:14.314 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.107:6789/0 mon.noname-f-legacy 2023-09-30T01:38:14.314 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.noname-b 2023-09-30T01:38:14.314 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.noname-g 2023-09-30T01:38:14.314 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.noname-c 2023-09-30T01:38:14.314 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.noname-h 2023-09-30T01:38:14.314 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.noname-d 2023-09-30T01:38:14.314 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.noname-i 2023-09-30T01:38:14.314 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.noname-e 2023-09-30T01:38:14.314 INFO:teuthology.orchestra.run.smithi088.stderr:9: v2:172.21.15.88:3300/0 mon.noname-a 2023-09-30T01:38:14.314 INFO:teuthology.orchestra.run.smithi088.stderr:10: v2:172.21.15.107:3300/0 mon.noname-f 2023-09-30T01:38:14.315 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:38:14.325 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.311+0000 7fa0908d7700 1 Processor -- start 2023-09-30T01:38:14.325 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.311+0000 7fa0908d7700 1 -- start start 2023-09-30T01:38:14.325 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.311+0000 7fa0908d7700 10 monclient: init 2023-09-30T01:38:14.326 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.311+0000 7fa0908d7700 10 monclient: _reopen_session rank -1 2023-09-30T01:38:14.326 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.311+0000 7fa0908d7700 10 monclient: _add_conns ranks=[6,10,7,9,1,3,8,0,5,2,4] 2023-09-30T01:38:14.326 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.311+0000 7fa0908d7700 1 --2- >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7fa08c097a40 0x7fa08c132700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:38:14.326 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.311+0000 7fa0908d7700 10 monclient(hunting): picked mon.noname-d con 0x7fa08c097a40 addr [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] 2023-09-30T01:38:14.326 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.311+0000 7fa0908d7700 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fa08c1311c0 0x7fa08c132c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:38:14.326 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.311+0000 7fa0908d7700 10 monclient(hunting): picked mon.noname-f con 0x7fa08c1311c0 addr v2:172.21.15.107:3300/0 2023-09-30T01:38:14.326 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.311+0000 7fa0908d7700 1 --2- >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7fa08c1332e0 0x7fa08c133740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:38:14.326 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.311+0000 7fa0908d7700 10 monclient(hunting): picked mon.noname-i con 0x7fa08c1332e0 addr [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] 2023-09-30T01:38:14.326 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.311+0000 7fa0908d7700 10 monclient(hunting): start opening mon connection 2023-09-30T01:38:14.326 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.311+0000 7fa0908d7700 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa08c131590 con 0x7fa08c1311c0 2023-09-30T01:38:14.326 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.311+0000 7fa0908d7700 10 monclient(hunting): start opening mon connection 2023-09-30T01:38:14.327 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.311+0000 7fa0908d7700 1 -- --> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] -- mon_getmap magic: 0 v1 -- 0x7fa08c09a5d0 con 0x7fa08c097a40 2023-09-30T01:38:14.327 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.311+0000 7fa0908d7700 10 monclient(hunting): start opening mon connection 2023-09-30T01:38:14.327 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.311+0000 7fa0908d7700 1 -- --> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] -- mon_getmap magic: 0 v1 -- 0x7fa08c097e10 con 0x7fa08c1332e0 2023-09-30T01:38:14.327 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.311+0000 7fa0908d7700 10 monclient(hunting): _renew_subs 2023-09-30T01:38:14.327 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.311+0000 7fa0908d7700 10 monclient(hunting): authenticate will time out at 1166.350250s 2023-09-30T01:38:14.327 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.311+0000 7fa08ad9d700 1 --2- >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7fa08c097a40 0x7fa08c132700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:38:14.327 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.311+0000 7fa08ad9d700 1 --2- >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7fa08c097a40 0x7fa08c132700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3303/0 says I am v2:172.21.15.88:49820/0 (socket says 172.21.15.88:49820) 2023-09-30T01:38:14.327 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.311+0000 7fa08ad9d700 1 -- 172.21.15.88:0/3736386530 learned_addr learned my addr 172.21.15.88:0/3736386530 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-09-30T01:38:14.327 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.311+0000 7fa08ad9d700 10 monclient(hunting): get_auth_request con 0x7fa08c097a40 auth_method 0 2023-09-30T01:38:14.327 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.311+0000 7fa08ad9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:38:14.328 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.311+0000 7fa08ad9d700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:38:14.328 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.311+0000 7fa08ad9d700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:38:14.328 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.311+0000 7fa08ad9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:38:14.328 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.311+0000 7fa08ad9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:38:14.328 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.311+0000 7fa08ad9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:38:14.328 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.311+0000 7fa08b59e700 1 --2- 172.21.15.88:0/3736386530 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7fa08c1332e0 0x7fa08c133740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:38:14.328 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.311+0000 7fa08b59e700 10 monclient(hunting): get_auth_request con 0x7fa08c1332e0 auth_method 0 2023-09-30T01:38:14.329 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.311+0000 7fa08b59e700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:38:14.329 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.311+0000 7fa08b59e700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:38:14.329 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.311+0000 7fa08b59e700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:38:14.329 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.311+0000 7fa08ad9d700 10 monclient(hunting): handle_auth_done global_id 4857 payload 274 2023-09-30T01:38:14.329 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.311+0000 7fa08a59c700 1 --2- 172.21.15.88:0/3736386530 >> v2:172.21.15.107:3300/0 conn(0x7fa08c1311c0 0x7fa08c132c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:38:14.329 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.311+0000 7fa08ad9d700 1 -- 172.21.15.88:0/3736386530 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7fa08c1332e0 msgr2=0x7fa08c133740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:38:14.329 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.311+0000 7fa08ad9d700 1 --2- 172.21.15.88:0/3736386530 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7fa08c1332e0 0x7fa08c133740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:38:14.329 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.311+0000 7fa08ad9d700 1 -- 172.21.15.88:0/3736386530 >> v2:172.21.15.107:3300/0 conn(0x7fa08c1311c0 msgr2=0x7fa08c132c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:38:14.329 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.311+0000 7fa08ad9d700 1 --2- 172.21.15.88:0/3736386530 >> v2:172.21.15.107:3300/0 conn(0x7fa08c1311c0 0x7fa08c132c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:38:14.329 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.311+0000 7fa08ad9d700 10 monclient: _finish_hunting 0 2023-09-30T01:38:14.330 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.311+0000 7fa08ad9d700 1 monclient: found mon.noname-d 2023-09-30T01:38:14.330 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.311+0000 7fa08ad9d700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:38:14.330 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.311+0000 7fa08ad9d700 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.88:3303/0 2023-09-30T01:38:14.330 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.311+0000 7fa08ad9d700 1 -- 172.21.15.88:0/3736386530 --> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa08c04bb20 con 0x7fa08c097a40 2023-09-30T01:38:14.330 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.311+0000 7fa08a59c700 10 monclient: get_auth_request con 0x7fa08c1311c0 auth_method 0 2023-09-30T01:38:14.330 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.311+0000 7fa08a59c700 1 --2- 172.21.15.88:0/3736386530 >> v2:172.21.15.107:3300/0 conn(0x7fa08c1311c0 0x7fa08c132c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-09-30T01:38:14.330 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.311+0000 7fa08ad9d700 1 --2- 172.21.15.88:0/3736386530 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7fa08c097a40 0x7fa08c132700 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fa074009f20 tx=0x7fa074009840 comp rx=0 tx=0).ready entity=mon.6 client_cookie=fd9c8ed7ec5da8b7 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:38:14.330 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.311+0000 7fa089d9b700 1 -- 172.21.15.88:0/3736386530 <== mon.6 v2:172.21.15.88:3303/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7fa07400e070 con 0x7fa08c097a40 2023-09-30T01:38:14.330 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.311+0000 7fa089d9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:38:14.330 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.311+0000 7fa089d9b700 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2023-09-30T01:38:14.330 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.311+0000 7fa089d9b700 10 monclient: dump: 2023-09-30T01:38:14.330 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:38:14.331 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:38:14.331 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:38:14.331 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:38:14.331 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:38:14.331 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:38:14.331 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:38:14.331 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:38:14.331 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:38:14.331 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:38:14.331 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:38:14.331 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:38:14.332 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:38:14.332 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:38:14.332 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:38:14.332 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:38:14.332 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.311+0000 7fa089d9b700 10 monclient: _finish_auth 0 2023-09-30T01:38:14.332 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.311+0000 7fa089d9b700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:38:14.332 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.311+0000 7fa0908d7700 5 monclient: authenticate success, global_id 4857 2023-09-30T01:38:14.332 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.311+0000 7fa0908d7700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:38:14.332 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.311+0000 7fa089d9b700 1 -- 172.21.15.88:0/3736386530 <== mon.6 v2:172.21.15.88:3303/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa07400ba80 con 0x7fa08c097a40 2023-09-30T01:38:14.333 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.311+0000 7fa089d9b700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:38:14.333 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.311+0000 7fa0908d7700 10 monclient: get_monmap_and_config success 2023-09-30T01:38:14.333 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.311+0000 7fa0908d7700 4 set_mon_vals no callback set 2023-09-30T01:38:14.333 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.311+0000 7fa089d9b700 1 -- 172.21.15.88:0/3736386530 <== mon.6 v2:172.21.15.88:3303/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7fa0740092e0 con 0x7fa08c097a40 2023-09-30T01:38:14.333 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.311+0000 7fa089d9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:38:14.333 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.311+0000 7fa089d9b700 10 monclient: got monmap 1 from mon.d (according to old e1) 2023-09-30T01:38:14.333 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.311+0000 7fa089d9b700 10 monclient: dump: 2023-09-30T01:38:14.333 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:38:14.333 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:38:14.333 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:38:14.333 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:38:14.333 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:38:14.333 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:38:14.334 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:38:14.334 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:38:14.334 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:38:14.334 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:38:14.334 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:38:14.334 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:38:14.334 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:38:14.334 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:38:14.334 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:38:14.334 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:38:14.334 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.315+0000 7fa0908d7700 10 monclient: shutdown 2023-09-30T01:38:14.334 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.315+0000 7fa0908d7700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:38:14.335 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.315+0000 7fa0908d7700 1 -- 172.21.15.88:0/3736386530 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7fa08c097a40 msgr2=0x7fa08c132700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:38:14.335 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.315+0000 7fa0908d7700 1 --2- 172.21.15.88:0/3736386530 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7fa08c097a40 0x7fa08c132700 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fa074009f20 tx=0x7fa074009840 comp rx=0 tx=0).stop 2023-09-30T01:38:14.335 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.315+0000 7fa0908d7700 1 -- 172.21.15.88:0/3736386530 shutdown_connections 2023-09-30T01:38:14.335 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.315+0000 7fa0908d7700 1 --2- 172.21.15.88:0/3736386530 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7fa08c1332e0 0x7fa08c133740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:38:14.335 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.315+0000 7fa0908d7700 1 --2- 172.21.15.88:0/3736386530 >> v2:172.21.15.107:3300/0 conn(0x7fa08c1311c0 0x7fa08c132c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:38:14.335 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.315+0000 7fa0908d7700 1 --2- 172.21.15.88:0/3736386530 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7fa08c097a40 0x7fa08c132700 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:38:14.335 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.315+0000 7fa0908d7700 1 -- 172.21.15.88:0/3736386530 >> 172.21.15.88:0/3736386530 conn(0x7fa08c06e5e0 msgr2=0x7fa08c06e9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:38:14.335 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.315+0000 7fa0908d7700 1 -- 172.21.15.88:0/3736386530 shutdown_connections 2023-09-30T01:38:14.335 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.315+0000 7fa0908d7700 1 -- 172.21.15.88:0/3736386530 wait complete. 2023-09-30T01:38:14.335 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.315+0000 7fa0908d7700 10 monclient: build_initial_monmap 2023-09-30T01:38:14.335 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.315+0000 7fa0908d7700 10 monclient: monmap: 2023-09-30T01:38:14.335 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2023-09-30T01:38:14.336 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:38:14.336 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 0.000000 2023-09-30T01:38:14.336 INFO:teuthology.orchestra.run.smithi088.stderr:created 0.000000 2023-09-30T01:38:14.336 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2023-09-30T01:38:14.336 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 1 2023-09-30T01:38:14.336 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.noname-a 2023-09-30T01:38:14.336 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.noname-f 2023-09-30T01:38:14.336 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.noname-b 2023-09-30T01:38:14.336 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.noname-g 2023-09-30T01:38:14.336 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.noname-c 2023-09-30T01:38:14.336 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.noname-h 2023-09-30T01:38:14.336 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.noname-d 2023-09-30T01:38:14.336 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.noname-i 2023-09-30T01:38:14.337 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.noname-e 2023-09-30T01:38:14.337 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:38:14.337 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.315+0000 7fa0908d7700 1 Processor -- start 2023-09-30T01:38:14.337 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.315+0000 7fa0908d7700 1 -- start start 2023-09-30T01:38:14.337 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.315+0000 7fa0908d7700 10 monclient: init 2023-09-30T01:38:14.337 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.319+0000 7fa0908d7700 10 monclient: _reopen_session rank -1 2023-09-30T01:38:14.337 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.319+0000 7fa0908d7700 10 monclient: _add_conns ranks=[1,0,8,2,6,3,5,7,4] 2023-09-30T01:38:14.337 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.319+0000 7fa0908d7700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa08c097a40 0x7fa08c0a1d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:38:14.337 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.319+0000 7fa0908d7700 10 monclient(hunting): picked mon.noname-f con 0x7fa08c097a40 addr [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] 2023-09-30T01:38:14.337 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.319+0000 7fa0908d7700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa08c1311c0 0x7fa08c0a2290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:38:14.337 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.319+0000 7fa0908d7700 10 monclient(hunting): picked mon.noname-a con 0x7fa08c1311c0 addr [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] 2023-09-30T01:38:14.337 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.319+0000 7fa0908d7700 1 --2- >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7fa08c1332e0 0x7fa08c0a27d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:38:14.338 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.319+0000 7fa0908d7700 10 monclient(hunting): picked mon.noname-e con 0x7fa08c1332e0 addr [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] 2023-09-30T01:38:14.338 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.319+0000 7fa0908d7700 10 monclient(hunting): start opening mon connection 2023-09-30T01:38:14.338 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.319+0000 7fa0908d7700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa08c11ec70 con 0x7fa08c1311c0 2023-09-30T01:38:14.338 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.319+0000 7fa0908d7700 10 monclient(hunting): start opening mon connection 2023-09-30T01:38:14.338 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.319+0000 7fa0908d7700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa08c099ca0 con 0x7fa08c097a40 2023-09-30T01:38:14.338 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.319+0000 7fa0908d7700 10 monclient(hunting): start opening mon connection 2023-09-30T01:38:14.338 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.319+0000 7fa0908d7700 1 -- --> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] -- mon_getmap magic: 0 v1 -- 0x7fa08c0996b0 con 0x7fa08c1332e0 2023-09-30T01:38:14.338 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.319+0000 7fa0908d7700 10 monclient(hunting): _renew_subs 2023-09-30T01:38:14.338 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.319+0000 7fa0908d7700 10 monclient(hunting): authenticate will time out at 1166.356387s 2023-09-30T01:38:14.339 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.319+0000 7fa08a59c700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa08c1311c0 0x7fa08c0a2290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:38:14.339 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.319+0000 7fa08a59c700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa08c1311c0 0x7fa08c0a2290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:48194/0 (socket says 172.21.15.88:48194) 2023-09-30T01:38:14.339 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.319+0000 7fa08a59c700 1 -- 172.21.15.88:0/3915745093 learned_addr learned my addr 172.21.15.88:0/3915745093 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-09-30T01:38:14.339 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.319+0000 7fa08a59c700 10 monclient(hunting): get_auth_request con 0x7fa08c1311c0 auth_method 0 2023-09-30T01:38:14.339 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.319+0000 7fa08a59c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:38:14.339 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.319+0000 7fa08a59c700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:38:14.339 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.319+0000 7fa08a59c700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:38:14.339 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.319+0000 7fa08b59e700 1 --2- 172.21.15.88:0/3915745093 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7fa08c1332e0 0x7fa08c0a27d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:38:14.340 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.319+0000 7fa08ad9d700 1 --2- 172.21.15.88:0/3915745093 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa08c097a40 0x7fa08c0a1d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:38:14.340 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.319+0000 7fa08b59e700 10 monclient(hunting): get_auth_request con 0x7fa08c1332e0 auth_method 0 2023-09-30T01:38:14.340 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.319+0000 7fa08b59e700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:38:14.340 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.319+0000 7fa08b59e700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:38:14.340 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.319+0000 7fa08b59e700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:38:14.340 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.319+0000 7fa08a59c700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:38:14.340 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.319+0000 7fa08a59c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:38:14.340 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.319+0000 7fa08a59c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:38:14.340 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.319+0000 7fa08ad9d700 10 monclient(hunting): get_auth_request con 0x7fa08c097a40 auth_method 0 2023-09-30T01:38:14.340 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.319+0000 7fa08ad9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:38:14.340 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.319+0000 7fa08ad9d700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:38:14.340 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.319+0000 7fa08ad9d700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:38:14.341 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.319+0000 7fa08b59e700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:38:14.341 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.319+0000 7fa08b59e700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:38:14.341 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.319+0000 7fa08b59e700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:38:14.341 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.319+0000 7fa08ad9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:38:14.341 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.319+0000 7fa08ad9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:38:14.341 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.319+0000 7fa08ad9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:38:14.341 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.319+0000 7fa08a59c700 10 monclient(hunting): handle_auth_done global_id 5256 payload 824 2023-09-30T01:38:14.341 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.319+0000 7fa08a59c700 1 -- 172.21.15.88:0/3915745093 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7fa08c1332e0 msgr2=0x7fa08c0a27d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:38:14.341 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.319+0000 7fa08a59c700 1 --2- 172.21.15.88:0/3915745093 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7fa08c1332e0 0x7fa08c0a27d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:38:14.341 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.319+0000 7fa08a59c700 1 -- 172.21.15.88:0/3915745093 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa08c097a40 msgr2=0x7fa08c0a1d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:38:14.341 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.319+0000 7fa08a59c700 1 --2- 172.21.15.88:0/3915745093 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa08c097a40 0x7fa08c0a1d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:38:14.342 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.319+0000 7fa08a59c700 10 monclient: _finish_hunting 0 2023-09-30T01:38:14.342 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.319+0000 7fa08a59c700 1 monclient: found mon.noname-a 2023-09-30T01:38:14.342 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.319+0000 7fa08a59c700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:38:14.342 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.319+0000 7fa08a59c700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.88:3300/0 2023-09-30T01:38:14.342 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.319+0000 7fa08a59c700 1 -- 172.21.15.88:0/3915745093 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa074009ae0 con 0x7fa08c1311c0 2023-09-30T01:38:14.342 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.319+0000 7fa08a59c700 1 --2- 172.21.15.88:0/3915745093 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa08c1311c0 0x7fa08c0a2290 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fa08c099840 tx=0x7fa084008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:38:14.342 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.319+0000 7fa082ffd700 1 -- 172.21.15.88:0/3915745093 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7fa084011a80 con 0x7fa08c1311c0 2023-09-30T01:38:14.342 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.319+0000 7fa082ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:38:14.342 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.319+0000 7fa082ffd700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T01:38:14.342 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.319+0000 7fa082ffd700 10 monclient: dump: 2023-09-30T01:38:14.342 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:38:14.342 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:38:14.343 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:38:14.343 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:38:14.343 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:38:14.343 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:38:14.343 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:38:14.343 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:38:14.343 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:38:14.343 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:38:14.343 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:38:14.343 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:38:14.343 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:38:14.343 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:38:14.344 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:38:14.344 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:38:14.344 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.319+0000 7fa082ffd700 10 monclient: _finish_auth 0 2023-09-30T01:38:14.344 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.319+0000 7fa082ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:38:14.344 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.319+0000 7fa0908d7700 5 monclient: authenticate success, global_id 5256 2023-09-30T01:38:14.344 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.319+0000 7fa0908d7700 10 monclient: _renew_subs 2023-09-30T01:38:14.344 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.319+0000 7fa0908d7700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.88:3300/0 2023-09-30T01:38:14.344 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.319+0000 7fa082ffd700 1 -- 172.21.15.88:0/3915745093 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa084011c20 con 0x7fa08c1311c0 2023-09-30T01:38:14.344 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.319+0000 7fa0908d7700 1 -- 172.21.15.88:0/3915745093 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa08c11f510 con 0x7fa08c1311c0 2023-09-30T01:38:14.344 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.319+0000 7fa082ffd700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:38:14.344 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.319+0000 7fa082ffd700 1 -- 172.21.15.88:0/3915745093 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7fa084011d90 con 0x7fa08c1311c0 2023-09-30T01:38:14.344 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.319+0000 7fa082ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:38:14.344 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.319+0000 7fa08959a700 4 set_mon_vals no callback set 2023-09-30T01:38:14.345 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.319+0000 7fa082ffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:38:14.345 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.319+0000 7fa082ffd700 10 monclient: dump: 2023-09-30T01:38:14.345 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:38:14.345 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:38:14.345 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:38:14.345 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:38:14.345 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:38:14.345 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:38:14.345 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:38:14.345 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:38:14.345 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:38:14.345 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:38:14.346 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:38:14.346 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:38:14.346 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:38:14.346 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:38:14.346 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:38:14.346 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:38:14.346 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.319+0000 7fa0908d7700 10 monclient: _renew_subs 2023-09-30T01:38:14.346 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.319+0000 7fa0908d7700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.88:3300/0 2023-09-30T01:38:14.346 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.319+0000 7fa0908d7700 1 -- 172.21.15.88:0/3915745093 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa08c130d20 con 0x7fa08c1311c0 2023-09-30T01:38:14.346 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.319+0000 7fa0908d7700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:38:14.347 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.319+0000 7fa082ffd700 1 -- 172.21.15.88:0/3915745093 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 40144+0+0 (secure 0 0 0) 0x7fa084011600 con 0x7fa08c1311c0 2023-09-30T01:38:14.347 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.319+0000 7fa0908d7700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:38:14.347 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.319+0000 7fa0908d7700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.88:3300/0 2023-09-30T01:38:14.347 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.319+0000 7fa0908d7700 1 -- 172.21.15.88:0/3915745093 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa08c071960 con 0x7fa08c1311c0 2023-09-30T01:38:14.347 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.319+0000 7fa082ffd700 1 --2- 172.21.15.88:0/3915745093 >> v2:172.21.15.107:6812/29926 conn(0x7fa070033360 0x7fa070035810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:38:14.347 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.319+0000 7fa082ffd700 1 -- 172.21.15.88:0/3915745093 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 3502+0+0 (secure 0 0 0) 0x7fa0840432f0 con 0x7fa08c1311c0 2023-09-30T01:38:14.347 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.319+0000 7fa08ad9d700 1 --2- 172.21.15.88:0/3915745093 >> v2:172.21.15.107:6812/29926 conn(0x7fa070033360 0x7fa070035810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:38:14.347 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.319+0000 7fa08ad9d700 10 monclient: get_auth_request con 0x7fa070033360 auth_method 0 2023-09-30T01:38:14.347 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.323+0000 7fa08ad9d700 1 --2- 172.21.15.88:0/3915745093 >> v2:172.21.15.107:6812/29926 conn(0x7fa070033360 0x7fa070035810 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fa074009f20 tx=0x7fa074018000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:38:14.348 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.327+0000 7fa082ffd700 1 -- 172.21.15.88:0/3915745093 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fa0840118b0 con 0x7fa08c1311c0 2023-09-30T01:38:14.348 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.327+0000 7fa082ffd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:38:14.348 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.327+0000 7fa082ffd700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:38:14.489 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.487+0000 7fa0908d7700 1 -- 172.21.15.88:0/3915745093 --> v2:172.21.15.107:6812/29926 -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "5", "target": ["mon-mgr", ""]}) v1 -- 0x7fa08c11dd30 con 0x7fa070033360 2023-09-30T01:38:14.490 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.487+0000 7fa082ffd700 1 -- 172.21.15.88:0/3915745093 <== mgr.4108 v2:172.21.15.107:6812/29926 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 5 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7fa08c11dd30 con 0x7fa070033360 2023-09-30T01:38:14.494 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.491+0000 7fa0908d7700 1 -- 172.21.15.88:0/3915745093 >> v2:172.21.15.107:6812/29926 conn(0x7fa070033360 msgr2=0x7fa070035810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:38:14.494 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.491+0000 7fa0908d7700 1 --2- 172.21.15.88:0/3915745093 >> v2:172.21.15.107:6812/29926 conn(0x7fa070033360 0x7fa070035810 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fa074009f20 tx=0x7fa074018000 comp rx=0 tx=0).stop 2023-09-30T01:38:14.494 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.491+0000 7fa0908d7700 10 monclient: shutdown 2023-09-30T01:38:14.495 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.491+0000 7fa0908d7700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:38:14.495 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.491+0000 7fa0908d7700 1 -- 172.21.15.88:0/3915745093 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa08c1311c0 msgr2=0x7fa08c0a2290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:38:14.495 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.491+0000 7fa0908d7700 1 --2- 172.21.15.88:0/3915745093 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa08c1311c0 0x7fa08c0a2290 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fa08c099840 tx=0x7fa084008040 comp rx=0 tx=0).stop 2023-09-30T01:38:14.495 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.491+0000 7fa0908d7700 1 -- 172.21.15.88:0/3915745093 shutdown_connections 2023-09-30T01:38:14.495 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.491+0000 7fa0908d7700 1 --2- 172.21.15.88:0/3915745093 >> v2:172.21.15.107:6812/29926 conn(0x7fa070033360 0x7fa070035810 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:38:14.495 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.491+0000 7fa0908d7700 1 --2- 172.21.15.88:0/3915745093 >> [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] conn(0x7fa08c1332e0 0x7fa08c0a27d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:38:14.495 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.491+0000 7fa0908d7700 1 --2- 172.21.15.88:0/3915745093 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa08c1311c0 0x7fa08c0a2290 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:38:14.495 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.491+0000 7fa0908d7700 1 --2- 172.21.15.88:0/3915745093 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa08c097a40 0x7fa08c0a1d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:38:14.495 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.491+0000 7fa0908d7700 1 -- 172.21.15.88:0/3915745093 >> 172.21.15.88:0/3915745093 conn(0x7fa08c06e5e0 msgr2=0x7fa08c132950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:38:14.495 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.491+0000 7fa0908d7700 1 -- 172.21.15.88:0/3915745093 shutdown_connections 2023-09-30T01:38:14.495 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.491+0000 7fa0908d7700 1 -- 172.21.15.88:0/3915745093 wait complete. 2023-09-30T01:38:14.496 INFO:teuthology.orchestra.run.smithi088.stderr:instructed osd(s) 5 to deep-scrub 2023-09-30T01:38:14.512 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-09-30T01:38:14.612 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.611+0000 7f0a47089700 10 monclient: get_monmap_and_config 2023-09-30T01:38:14.612 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.611+0000 7f0a47089700 10 monclient: build_initial_monmap 2023-09-30T01:38:14.612 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.611+0000 7f0a47089700 10 monclient: monmap: 2023-09-30T01:38:14.612 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2023-09-30T01:38:14.613 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:38:14.613 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:38:14.611929+0000 2023-09-30T01:38:14.613 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:38:14.611929+0000 2023-09-30T01:38:14.613 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2023-09-30T01:38:14.613 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:38:14.613 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a-legacy 2023-09-30T01:38:14.613 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.107:6789/0 mon.noname-f-legacy 2023-09-30T01:38:14.613 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.noname-b 2023-09-30T01:38:14.613 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.noname-g 2023-09-30T01:38:14.614 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.noname-c 2023-09-30T01:38:14.614 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.noname-h 2023-09-30T01:38:14.614 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.noname-d 2023-09-30T01:38:14.614 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.noname-i 2023-09-30T01:38:14.614 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.noname-e 2023-09-30T01:38:14.614 INFO:teuthology.orchestra.run.smithi088.stderr:9: v2:172.21.15.88:3300/0 mon.noname-a 2023-09-30T01:38:14.614 INFO:teuthology.orchestra.run.smithi088.stderr:10: v2:172.21.15.107:3300/0 mon.noname-f 2023-09-30T01:38:14.614 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:38:14.623 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.611+0000 7f0a47089700 1 Processor -- start 2023-09-30T01:38:14.623 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.611+0000 7f0a47089700 1 -- start start 2023-09-30T01:38:14.624 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.611+0000 7f0a47089700 10 monclient: init 2023-09-30T01:38:14.624 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.611+0000 7f0a47089700 10 monclient: _reopen_session rank -1 2023-09-30T01:38:14.624 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.611+0000 7f0a47089700 10 monclient: _add_conns ranks=[2,7,6,8,4,5,1,3,0,9,10] 2023-09-30T01:38:14.624 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.611+0000 7f0a47089700 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0a400966a0 0x7f0a40096a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:38:14.624 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.611+0000 7f0a47089700 10 monclient(hunting): picked mon.noname-b con 0x7f0a400966a0 addr [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] 2023-09-30T01:38:14.624 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.611+0000 7f0a47089700 1 --2- >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7f0a40097e40 0x7f0a4009ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:38:14.624 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.611+0000 7f0a47089700 10 monclient(hunting): picked mon.noname-i con 0x7f0a40097e40 addr [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] 2023-09-30T01:38:14.624 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.611+0000 7f0a47089700 1 --2- >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7f0a40098210 0x7f0a4009e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:38:14.624 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.611+0000 7f0a47089700 10 monclient(hunting): picked mon.noname-d con 0x7f0a40098210 addr [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] 2023-09-30T01:38:14.624 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.611+0000 7f0a47089700 10 monclient(hunting): start opening mon connection 2023-09-30T01:38:14.625 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.611+0000 7f0a47089700 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a40097740 con 0x7f0a400966a0 2023-09-30T01:38:14.626 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.611+0000 7f0a47089700 10 monclient(hunting): start opening mon connection 2023-09-30T01:38:14.626 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.611+0000 7f0a47089700 1 -- --> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] -- mon_getmap magic: 0 v1 -- 0x7f0a40063510 con 0x7f0a40098210 2023-09-30T01:38:14.626 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.611+0000 7f0a47089700 10 monclient(hunting): start opening mon connection 2023-09-30T01:38:14.626 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.611+0000 7f0a47089700 1 -- --> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] -- mon_getmap magic: 0 v1 -- 0x7f0a40063750 con 0x7f0a40097e40 2023-09-30T01:38:14.627 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.611+0000 7f0a47089700 10 monclient(hunting): _renew_subs 2023-09-30T01:38:14.627 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.611+0000 7f0a47089700 10 monclient(hunting): authenticate will time out at 1166.649352s 2023-09-30T01:38:14.627 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.611+0000 7f0a45626700 1 --2- >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7f0a40098210 0x7f0a4009e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:38:14.627 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.611+0000 7f0a44e25700 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0a400966a0 0x7f0a40096a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:38:14.627 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.611+0000 7f0a44e25700 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0a400966a0 0x7f0a40096a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3301/0 says I am v2:172.21.15.88:53590/0 (socket says 172.21.15.88:53590) 2023-09-30T01:38:14.627 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.611+0000 7f0a45626700 1 --2- >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7f0a40098210 0x7f0a4009e2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3303/0 says I am v2:172.21.15.88:49832/0 (socket says 172.21.15.88:49832) 2023-09-30T01:38:14.627 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.611+0000 7f0a44e25700 1 -- 172.21.15.88:0/4209588927 learned_addr learned my addr 172.21.15.88:0/4209588927 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-09-30T01:38:14.627 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.611+0000 7f0a44e25700 10 monclient(hunting): get_auth_request con 0x7f0a400966a0 auth_method 0 2023-09-30T01:38:14.627 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.611+0000 7f0a44e25700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:38:14.627 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.611+0000 7f0a44e25700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:38:14.628 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.611+0000 7f0a44e25700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:38:14.628 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.611+0000 7f0a45626700 10 monclient(hunting): get_auth_request con 0x7f0a40098210 auth_method 0 2023-09-30T01:38:14.628 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.611+0000 7f0a45626700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:38:14.628 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.611+0000 7f0a45626700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:38:14.628 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.611+0000 7f0a45626700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:38:14.628 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.611+0000 7f0a44e25700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:38:14.628 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.611+0000 7f0a44e25700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:38:14.629 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.611+0000 7f0a3ffff700 1 --2- 172.21.15.88:0/4209588927 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7f0a40097e40 0x7f0a4009ddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:38:14.629 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.611+0000 7f0a44e25700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:38:14.629 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.611+0000 7f0a45626700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:38:14.629 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.611+0000 7f0a45626700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:38:14.629 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.611+0000 7f0a45626700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:38:14.629 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.611+0000 7f0a3ffff700 10 monclient(hunting): get_auth_request con 0x7f0a40097e40 auth_method 0 2023-09-30T01:38:14.629 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.611+0000 7f0a3ffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:38:14.630 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.611+0000 7f0a3ffff700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:38:14.630 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.611+0000 7f0a3ffff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:38:14.630 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.611+0000 7f0a3ffff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:38:14.630 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.611+0000 7f0a3ffff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:38:14.630 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.611+0000 7f0a3ffff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:38:14.630 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.611+0000 7f0a44e25700 10 monclient(hunting): handle_auth_done global_id 4961 payload 274 2023-09-30T01:38:14.630 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.611+0000 7f0a44e25700 1 -- 172.21.15.88:0/4209588927 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7f0a40097e40 msgr2=0x7f0a4009ddb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:38:14.630 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.611+0000 7f0a44e25700 1 --2- 172.21.15.88:0/4209588927 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7f0a40097e40 0x7f0a4009ddb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:38:14.630 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.611+0000 7f0a44e25700 1 -- 172.21.15.88:0/4209588927 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7f0a40098210 msgr2=0x7f0a4009e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:38:14.630 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.611+0000 7f0a44e25700 1 --2- 172.21.15.88:0/4209588927 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7f0a40098210 0x7f0a4009e2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:38:14.630 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.611+0000 7f0a44e25700 10 monclient: _finish_hunting 0 2023-09-30T01:38:14.631 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.611+0000 7f0a44e25700 1 monclient: found mon.noname-b 2023-09-30T01:38:14.631 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.611+0000 7f0a44e25700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:38:14.631 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.611+0000 7f0a44e25700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.88:3301/0 2023-09-30T01:38:14.631 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.611+0000 7f0a44e25700 1 -- 172.21.15.88:0/4209588927 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a4005ecb0 con 0x7f0a400966a0 2023-09-30T01:38:14.631 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.611+0000 7f0a45626700 1 --2- 172.21.15.88:0/4209588927 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7f0a40098210 0x7f0a4009e2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:38:14.631 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.611+0000 7f0a44e25700 1 --2- 172.21.15.88:0/4209588927 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0a400966a0 0x7f0a40096a70 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f0a38009f50 tx=0x7f0a38009840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d8522ecc95b79afe server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:38:14.631 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.611+0000 7f0a3f7fe700 1 -- 172.21.15.88:0/4209588927 <== mon.2 v2:172.21.15.88:3301/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f0a3800e070 con 0x7f0a400966a0 2023-09-30T01:38:14.631 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.611+0000 7f0a3f7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:38:14.631 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.611+0000 7f0a3f7fe700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T01:38:14.631 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.611+0000 7f0a3f7fe700 10 monclient: dump: 2023-09-30T01:38:14.631 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:38:14.632 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:38:14.632 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:38:14.632 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:38:14.632 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:38:14.632 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:38:14.632 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:38:14.632 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:38:14.632 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:38:14.632 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:38:14.632 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:38:14.632 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:38:14.632 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:38:14.632 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:38:14.633 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:38:14.633 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:38:14.633 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.611+0000 7f0a3f7fe700 10 monclient: _finish_auth 0 2023-09-30T01:38:14.633 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.611+0000 7f0a3f7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:38:14.633 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.611+0000 7f0a3f7fe700 1 -- 172.21.15.88:0/4209588927 <== mon.2 v2:172.21.15.88:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a3800bdd0 con 0x7f0a400966a0 2023-09-30T01:38:14.633 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.611+0000 7f0a47089700 5 monclient: authenticate success, global_id 4961 2023-09-30T01:38:14.633 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.611+0000 7f0a47089700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:38:14.633 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.611+0000 7f0a3f7fe700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:38:14.633 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.611+0000 7f0a47089700 10 monclient: get_monmap_and_config success 2023-09-30T01:38:14.633 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.611+0000 7f0a3f7fe700 1 -- 172.21.15.88:0/4209588927 <== mon.2 v2:172.21.15.88:3301/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f0a380092e0 con 0x7f0a400966a0 2023-09-30T01:38:14.633 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.611+0000 7f0a3f7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:38:14.633 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.611+0000 7f0a47089700 4 set_mon_vals no callback set 2023-09-30T01:38:14.634 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.611+0000 7f0a3f7fe700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T01:38:14.634 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.611+0000 7f0a3f7fe700 10 monclient: dump: 2023-09-30T01:38:14.634 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:38:14.634 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:38:14.634 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:38:14.634 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:38:14.634 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:38:14.634 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:38:14.634 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:38:14.634 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:38:14.635 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:38:14.635 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:38:14.635 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:38:14.635 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:38:14.635 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:38:14.635 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:38:14.635 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:38:14.635 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:38:14.635 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.611+0000 7f0a47089700 10 monclient: shutdown 2023-09-30T01:38:14.635 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.611+0000 7f0a47089700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:38:14.635 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.611+0000 7f0a47089700 1 -- 172.21.15.88:0/4209588927 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0a400966a0 msgr2=0x7f0a40096a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:38:14.635 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.611+0000 7f0a47089700 1 --2- 172.21.15.88:0/4209588927 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0a400966a0 0x7f0a40096a70 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f0a38009f50 tx=0x7f0a38009840 comp rx=0 tx=0).stop 2023-09-30T01:38:14.636 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.611+0000 7f0a47089700 1 -- 172.21.15.88:0/4209588927 shutdown_connections 2023-09-30T01:38:14.636 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.611+0000 7f0a47089700 1 --2- 172.21.15.88:0/4209588927 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7f0a40098210 0x7f0a4009e2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:38:14.636 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.611+0000 7f0a47089700 1 --2- 172.21.15.88:0/4209588927 >> [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] conn(0x7f0a40097e40 0x7f0a4009ddb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:38:14.636 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.611+0000 7f0a47089700 1 --2- 172.21.15.88:0/4209588927 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0a400966a0 0x7f0a40096a70 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:38:14.636 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.611+0000 7f0a47089700 1 -- 172.21.15.88:0/4209588927 >> 172.21.15.88:0/4209588927 conn(0x7f0a400a3440 msgr2=0x7f0a400a3840 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:38:14.636 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.611+0000 7f0a47089700 1 -- 172.21.15.88:0/4209588927 shutdown_connections 2023-09-30T01:38:14.636 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.611+0000 7f0a47089700 1 -- 172.21.15.88:0/4209588927 wait complete. 2023-09-30T01:38:14.636 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.615+0000 7f0a47089700 10 monclient: build_initial_monmap 2023-09-30T01:38:14.636 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.615+0000 7f0a47089700 10 monclient: monmap: 2023-09-30T01:38:14.637 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2023-09-30T01:38:14.637 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:38:14.637 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 0.000000 2023-09-30T01:38:14.637 INFO:teuthology.orchestra.run.smithi088.stderr:created 0.000000 2023-09-30T01:38:14.637 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2023-09-30T01:38:14.637 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 1 2023-09-30T01:38:14.637 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.noname-a 2023-09-30T01:38:14.637 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.noname-f 2023-09-30T01:38:14.637 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.noname-b 2023-09-30T01:38:14.637 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.noname-g 2023-09-30T01:38:14.637 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.noname-c 2023-09-30T01:38:14.638 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.noname-h 2023-09-30T01:38:14.638 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.noname-d 2023-09-30T01:38:14.638 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.noname-i 2023-09-30T01:38:14.638 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.noname-e 2023-09-30T01:38:14.638 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:38:14.638 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.615+0000 7f0a47089700 1 Processor -- start 2023-09-30T01:38:14.638 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.615+0000 7f0a47089700 1 -- start start 2023-09-30T01:38:14.638 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.615+0000 7f0a47089700 10 monclient: init 2023-09-30T01:38:14.638 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.615+0000 7f0a47089700 10 monclient: _reopen_session rank -1 2023-09-30T01:38:14.638 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.615+0000 7f0a47089700 10 monclient: _add_conns ranks=[1,0,3,4,7,5,8,6,2] 2023-09-30T01:38:14.638 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.615+0000 7f0a47089700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0a400966a0 0x7f0a40142c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:38:14.639 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.615+0000 7f0a47089700 10 monclient(hunting): picked mon.noname-f con 0x7f0a400966a0 addr [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] 2023-09-30T01:38:14.639 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.615+0000 7f0a47089700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0a40097e40 0x7f0a40143150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:38:14.639 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.615+0000 7f0a47089700 10 monclient(hunting): picked mon.noname-a con 0x7f0a40097e40 addr [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] 2023-09-30T01:38:14.639 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.615+0000 7f0a47089700 1 --2- >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7f0a40098210 0x7f0a4008df60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:38:14.639 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.615+0000 7f0a47089700 10 monclient(hunting): picked mon.noname-g con 0x7f0a40098210 addr [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] 2023-09-30T01:38:14.639 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.615+0000 7f0a47089700 10 monclient(hunting): start opening mon connection 2023-09-30T01:38:14.639 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.615+0000 7f0a47089700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a40099830 con 0x7f0a40097e40 2023-09-30T01:38:14.639 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.615+0000 7f0a47089700 10 monclient(hunting): start opening mon connection 2023-09-30T01:38:14.639 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.615+0000 7f0a47089700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a4009f780 con 0x7f0a400966a0 2023-09-30T01:38:14.639 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.615+0000 7f0a47089700 10 monclient(hunting): start opening mon connection 2023-09-30T01:38:14.640 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.615+0000 7f0a47089700 1 -- --> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a40090860 con 0x7f0a40098210 2023-09-30T01:38:14.640 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.615+0000 7f0a47089700 10 monclient(hunting): _renew_subs 2023-09-30T01:38:14.640 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.615+0000 7f0a47089700 10 monclient(hunting): authenticate will time out at 1166.653775s 2023-09-30T01:38:14.640 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.615+0000 7f0a3ffff700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0a40097e40 0x7f0a40143150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:38:14.640 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.615+0000 7f0a3ffff700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0a40097e40 0x7f0a40143150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:48204/0 (socket says 172.21.15.88:48204) 2023-09-30T01:38:14.640 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.615+0000 7f0a3ffff700 1 -- 172.21.15.88:0/344455043 learned_addr learned my addr 172.21.15.88:0/344455043 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-09-30T01:38:14.640 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.615+0000 7f0a3ffff700 10 monclient(hunting): get_auth_request con 0x7f0a40097e40 auth_method 0 2023-09-30T01:38:14.640 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.615+0000 7f0a3ffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:38:14.640 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.615+0000 7f0a3ffff700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:38:14.640 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.615+0000 7f0a3ffff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:38:14.641 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.615+0000 7f0a44e25700 1 --2- 172.21.15.88:0/344455043 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0a400966a0 0x7f0a40142c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:38:14.641 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.615+0000 7f0a45626700 1 --2- 172.21.15.88:0/344455043 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7f0a40098210 0x7f0a4008df60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:38:14.641 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.615+0000 7f0a44e25700 10 monclient(hunting): get_auth_request con 0x7f0a400966a0 auth_method 0 2023-09-30T01:38:14.641 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.615+0000 7f0a44e25700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:38:14.641 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.615+0000 7f0a44e25700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:38:14.641 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.615+0000 7f0a44e25700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:38:14.641 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.615+0000 7f0a3ffff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:38:14.641 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.615+0000 7f0a3ffff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:38:14.641 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.615+0000 7f0a3ffff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:38:14.641 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.615+0000 7f0a45626700 10 monclient(hunting): get_auth_request con 0x7f0a40098210 auth_method 0 2023-09-30T01:38:14.642 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.615+0000 7f0a45626700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:38:14.642 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.615+0000 7f0a45626700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:38:14.642 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.615+0000 7f0a45626700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:38:14.642 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.615+0000 7f0a44e25700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:38:14.642 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.615+0000 7f0a44e25700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:38:14.643 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.615+0000 7f0a44e25700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:38:14.643 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.615+0000 7f0a45626700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:38:14.643 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.615+0000 7f0a45626700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:38:14.643 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.615+0000 7f0a45626700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:38:14.643 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.615+0000 7f0a3ffff700 10 monclient(hunting): handle_auth_done global_id 5265 payload 824 2023-09-30T01:38:14.643 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.615+0000 7f0a3ffff700 1 -- 172.21.15.88:0/344455043 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7f0a40098210 msgr2=0x7f0a4008df60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:38:14.643 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.615+0000 7f0a3ffff700 1 --2- 172.21.15.88:0/344455043 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7f0a40098210 0x7f0a4008df60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:38:14.643 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.615+0000 7f0a3ffff700 1 -- 172.21.15.88:0/344455043 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0a400966a0 msgr2=0x7f0a40142c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:38:14.643 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.615+0000 7f0a3ffff700 1 --2- 172.21.15.88:0/344455043 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0a400966a0 0x7f0a40142c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:38:14.643 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.615+0000 7f0a3ffff700 10 monclient: _finish_hunting 0 2023-09-30T01:38:14.644 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.615+0000 7f0a3ffff700 1 monclient: found mon.noname-a 2023-09-30T01:38:14.644 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.615+0000 7f0a3ffff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:38:14.644 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.615+0000 7f0a3ffff700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.88:3300/0 2023-09-30T01:38:14.644 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.615+0000 7f0a3ffff700 1 -- 172.21.15.88:0/344455043 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a30009420 con 0x7f0a40097e40 2023-09-30T01:38:14.644 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.615+0000 7f0a3ffff700 1 --2- 172.21.15.88:0/344455043 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0a40097e40 0x7f0a40143150 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f0a3400ec80 tx=0x7f0a340091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:38:14.644 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.615+0000 7f0a3cff9700 1 -- 172.21.15.88:0/344455043 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f0a340096b0 con 0x7f0a40097e40 2023-09-30T01:38:14.644 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.615+0000 7f0a3cff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:38:14.644 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.615+0000 7f0a3cff9700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T01:38:14.644 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.615+0000 7f0a3cff9700 10 monclient: dump: 2023-09-30T01:38:14.644 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:38:14.645 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:38:14.645 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:38:14.645 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:38:14.645 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:38:14.645 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:38:14.645 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:38:14.645 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:38:14.645 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:38:14.646 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:38:14.646 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:38:14.646 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:38:14.646 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:38:14.646 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:38:14.646 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:38:14.646 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:38:14.646 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.615+0000 7f0a3cff9700 10 monclient: _finish_auth 0 2023-09-30T01:38:14.646 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.615+0000 7f0a3cff9700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:38:14.646 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.615+0000 7f0a47089700 5 monclient: authenticate success, global_id 5265 2023-09-30T01:38:14.646 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.615+0000 7f0a3cff9700 1 -- 172.21.15.88:0/344455043 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a3400ccd0 con 0x7f0a40097e40 2023-09-30T01:38:14.647 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.615+0000 7f0a3cff9700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:38:14.647 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.615+0000 7f0a47089700 10 monclient: _renew_subs 2023-09-30T01:38:14.647 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.615+0000 7f0a3cff9700 1 -- 172.21.15.88:0/344455043 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f0a3400ce40 con 0x7f0a40097e40 2023-09-30T01:38:14.647 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.615+0000 7f0a47089700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.88:3300/0 2023-09-30T01:38:14.647 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.615+0000 7f0a47089700 1 -- 172.21.15.88:0/344455043 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a38009ae0 con 0x7f0a40097e40 2023-09-30T01:38:14.647 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.615+0000 7f0a3e7fc700 4 set_mon_vals no callback set 2023-09-30T01:38:14.647 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.615+0000 7f0a3cff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:38:14.647 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.615+0000 7f0a3cff9700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:38:14.647 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.615+0000 7f0a3cff9700 10 monclient: dump: 2023-09-30T01:38:14.647 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:38:14.648 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:38:14.648 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:38:14.648 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:38:14.648 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:38:14.648 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:38:14.648 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:38:14.648 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:38:14.648 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:38:14.648 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:38:14.648 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:38:14.649 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:38:14.649 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:38:14.649 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:38:14.649 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:38:14.649 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:38:14.649 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.615+0000 7f0a47089700 10 monclient: _renew_subs 2023-09-30T01:38:14.649 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.615+0000 7f0a47089700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.88:3300/0 2023-09-30T01:38:14.649 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.615+0000 7f0a47089700 1 -- 172.21.15.88:0/344455043 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a40064b80 con 0x7f0a40097e40 2023-09-30T01:38:14.649 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.619+0000 7f0a3cff9700 1 -- 172.21.15.88:0/344455043 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 40144+0+0 (secure 0 0 0) 0x7f0a3400c850 con 0x7f0a40097e40 2023-09-30T01:38:14.649 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.619+0000 7f0a47089700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:38:14.650 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.619+0000 7f0a47089700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:38:14.650 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.619+0000 7f0a47089700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.88:3300/0 2023-09-30T01:38:14.650 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.619+0000 7f0a47089700 1 -- 172.21.15.88:0/344455043 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0a40091e30 con 0x7f0a40097e40 2023-09-30T01:38:14.650 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.619+0000 7f0a3cff9700 1 --2- 172.21.15.88:0/344455043 >> v2:172.21.15.107:6812/29926 conn(0x7f0a2c033360 0x7f0a2c035810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:38:14.651 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.619+0000 7f0a3cff9700 1 -- 172.21.15.88:0/344455043 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 3502+0+0 (secure 0 0 0) 0x7f0a34043620 con 0x7f0a40097e40 2023-09-30T01:38:14.651 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.619+0000 7f0a44e25700 1 --2- 172.21.15.88:0/344455043 >> v2:172.21.15.107:6812/29926 conn(0x7f0a2c033360 0x7f0a2c035810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:38:14.653 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.619+0000 7f0a44e25700 10 monclient: get_auth_request con 0x7f0a2c033360 auth_method 0 2023-09-30T01:38:14.653 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.619+0000 7f0a44e25700 1 --2- 172.21.15.88:0/344455043 >> v2:172.21.15.107:6812/29926 conn(0x7f0a2c033360 0x7f0a2c035810 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f0a38009f50 tx=0x7f0a38018000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:38:14.654 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.627+0000 7f0a3cff9700 1 -- 172.21.15.88:0/344455043 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f0a40091e30 con 0x7f0a40097e40 2023-09-30T01:38:14.654 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.627+0000 7f0a3cff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:38:14.654 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.627+0000 7f0a3cff9700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:38:14.789 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.787+0000 7f0a47089700 1 -- 172.21.15.88:0/344455043 --> v2:172.21.15.107:6812/29926 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f0a40091e30 con 0x7f0a2c033360 2023-09-30T01:38:14.793 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.791+0000 7f0a3cff9700 1 -- 172.21.15.88:0/344455043 <== mgr.4108 v2:172.21.15.107:6812/29926 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+53271 (secure 0 0 0) 0x7f0a40091e30 con 0x7f0a2c033360 2023-09-30T01:38:14.794 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-09-30T01:38:14.796 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.795+0000 7f0a47089700 1 -- 172.21.15.88:0/344455043 >> v2:172.21.15.107:6812/29926 conn(0x7f0a2c033360 msgr2=0x7f0a2c035810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:38:14.796 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.795+0000 7f0a47089700 1 --2- 172.21.15.88:0/344455043 >> v2:172.21.15.107:6812/29926 conn(0x7f0a2c033360 0x7f0a2c035810 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f0a38009f50 tx=0x7f0a38018000 comp rx=0 tx=0).stop 2023-09-30T01:38:14.796 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.795+0000 7f0a47089700 10 monclient: shutdown 2023-09-30T01:38:14.797 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.795+0000 7f0a47089700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:38:14.797 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.795+0000 7f0a47089700 1 -- 172.21.15.88:0/344455043 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0a40097e40 msgr2=0x7f0a40143150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:38:14.797 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.795+0000 7f0a47089700 1 --2- 172.21.15.88:0/344455043 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0a40097e40 0x7f0a40143150 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f0a3400ec80 tx=0x7f0a340091a0 comp rx=0 tx=0).stop 2023-09-30T01:38:14.797 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.795+0000 7f0a47089700 1 -- 172.21.15.88:0/344455043 shutdown_connections 2023-09-30T01:38:14.797 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.795+0000 7f0a47089700 1 --2- 172.21.15.88:0/344455043 >> v2:172.21.15.107:6812/29926 conn(0x7f0a2c033360 0x7f0a2c035810 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:38:14.797 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.795+0000 7f0a47089700 1 --2- 172.21.15.88:0/344455043 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7f0a40098210 0x7f0a4008df60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:38:14.797 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.795+0000 7f0a47089700 1 --2- 172.21.15.88:0/344455043 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0a40097e40 0x7f0a40143150 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:38:14.797 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.795+0000 7f0a47089700 1 --2- 172.21.15.88:0/344455043 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0a400966a0 0x7f0a40142c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:38:14.797 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.795+0000 7f0a47089700 1 -- 172.21.15.88:0/344455043 >> 172.21.15.88:0/344455043 conn(0x7f0a400a3440 msgr2=0x7f0a400a1db0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:38:14.798 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.795+0000 7f0a47089700 1 -- 172.21.15.88:0/344455043 shutdown_connections 2023-09-30T01:38:14.798 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:14.795+0000 7f0a47089700 1 -- 172.21.15.88:0/344455043 wait complete. 2023-09-30T01:38:14.798 INFO:teuthology.orchestra.run.smithi088.stderr:dumped all 2023-09-30T01:38:14.808 INFO:teuthology.orchestra.run.smithi088.stdout:{"pg_ready":true,"pg_map":{"version":167,"stamp":"2023-09-30T01:38:13.744758+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":32704,"kb_used_data":1808,"kb_used_omap":0,"kb_used_meta":30848,"kb_avail":566198336,"statfs":{"total":579820584960,"available":579787096064,"internally_reserved":0,"allocated":1851392,"data_stored":1284542,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":31588352},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003546"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-09-30T01:33:03.296691+0000","last_change":"2023-09-30T01:33:03.292315+0000","last_active":"2023-09-30T01:33:03.296691+0000","last_peered":"2023-09-30T01:33:03.296691+0000","last_clean":"2023-09-30T01:33:03.296691+0000","last_became_active":"2023-09-30T01:33:03.291889+0000","last_became_peered":"2023-09-30T01:33:03.291889+0000","last_unstale":"2023-09-30T01:33:03.296691+0000","last_undegraded":"2023-09-30T01:33:03.296691+0000","last_fullsized":"2023-09-30T01:33:03.296691+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T01:33:02.252599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T01:33:02.252599+0000","last_clean_scrub_stamp":"2023-09-30T01:33:02.252599+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T11:44:00.574379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-09-30T01:33:03.297822+0000","last_change":"2023-09-30T01:33:03.295081+0000","last_active":"2023-09-30T01:33:03.297822+0000","last_peered":"2023-09-30T01:33:03.297822+0000","last_clean":"2023-09-30T01:33:03.297822+0000","last_became_active":"2023-09-30T01:33:03.294535+0000","last_became_peered":"2023-09-30T01:33:03.294535+0000","last_unstale":"2023-09-30T01:33:03.297822+0000","last_undegraded":"2023-09-30T01:33:03.297822+0000","last_fullsized":"2023-09-30T01:33:03.297822+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T01:33:02.252599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T01:33:02.252599+0000","last_clean_scrub_stamp":"2023-09-30T01:33:02.252599+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T05:49:49.642723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-09-30T01:33:03.300117+0000","last_change":"2023-09-30T01:33:03.297392+0000","last_active":"2023-09-30T01:33:03.300117+0000","last_peered":"2023-09-30T01:33:03.300117+0000","last_clean":"2023-09-30T01:33:03.300117+0000","last_became_active":"2023-09-30T01:33:03.296959+0000","last_became_peered":"2023-09-30T01:33:03.296959+0000","last_unstale":"2023-09-30T01:33:03.300117+0000","last_undegraded":"2023-09-30T01:33:03.300117+0000","last_fullsized":"2023-09-30T01:33:03.300117+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T01:33:02.252599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T01:33:02.252599+0000","last_clean_scrub_stamp":"2023-09-30T01:33:02.252599+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T11:57:13.112607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-09-30T01:33:03.296932+0000","last_change":"2023-09-30T01:33:03.295478+0000","last_active":"2023-09-30T01:33:03.296932+0000","last_peered":"2023-09-30T01:33:03.296932+0000","last_clean":"2023-09-30T01:33:03.296932+0000","last_became_active":"2023-09-30T01:33:03.295106+0000","last_became_peered":"2023-09-30T01:33:03.295106+0000","last_unstale":"2023-09-30T01:33:03.296932+0000","last_undegraded":"2023-09-30T01:33:03.296932+0000","last_fullsized":"2023-09-30T01:33:03.296932+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T01:33:02.252599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T01:33:02.252599+0000","last_clean_scrub_stamp":"2023-09-30T01:33:02.252599+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T07:34:36.123333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"15'117","reported_seq":159,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-09-30T01:33:05.391246+0000","last_change":"2023-09-30T01:33:03.296872+0000","last_active":"2023-09-30T01:33:05.391246+0000","last_peered":"2023-09-30T01:33:05.391246+0000","last_clean":"2023-09-30T01:33:05.391246+0000","last_became_active":"2023-09-30T01:33:03.295200+0000","last_became_peered":"2023-09-30T01:33:03.295200+0000","last_unstale":"2023-09-30T01:33:05.391246+0000","last_undegraded":"2023-09-30T01:33:05.391246+0000","last_fullsized":"2023-09-30T01:33:05.391246+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T01:33:02.252599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T01:33:02.252599+0000","last_clean_scrub_stamp":"2023-09-30T01:33:02.252599+0000","objects_scrubbed":0,"log_size":117,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T01:51:03.655484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-09-30T01:33:03.296258+0000","last_change":"2023-09-30T01:33:03.293385+0000","last_active":"2023-09-30T01:33:03.296258+0000","last_peered":"2023-09-30T01:33:03.296258+0000","last_clean":"2023-09-30T01:33:03.296258+0000","last_became_active":"2023-09-30T01:33:03.293004+0000","last_became_peered":"2023-09-30T01:33:03.293004+0000","last_unstale":"2023-09-30T01:33:03.296258+0000","last_undegraded":"2023-09-30T01:33:03.296258+0000","last_fullsized":"2023-09-30T01:33:03.296258+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T01:33:02.252599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T01:33:02.252599+0000","last_clean_scrub_stamp":"2023-09-30T01:33:02.252599+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T03:32:32.404638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-09-30T01:33:03.299431+0000","last_change":"2023-09-30T01:33:03.296945+0000","last_active":"2023-09-30T01:33:03.299431+0000","last_peered":"2023-09-30T01:33:03.299431+0000","last_clean":"2023-09-30T01:33:03.299431+0000","last_became_active":"2023-09-30T01:33:03.296465+0000","last_became_peered":"2023-09-30T01:33:03.296465+0000","last_unstale":"2023-09-30T01:33:03.299431+0000","last_undegraded":"2023-09-30T01:33:03.299431+0000","last_fullsized":"2023-09-30T01:33:03.299431+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T01:33:02.252599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T01:33:02.252599+0000","last_clean_scrub_stamp":"2023-09-30T01:33:02.252599+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T05:32:13.855128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-09-30T01:33:03.296569+0000","last_change":"2023-09-30T01:33:03.291527+0000","last_active":"2023-09-30T01:33:03.296569+0000","last_peered":"2023-09-30T01:33:03.296569+0000","last_clean":"2023-09-30T01:33:03.296569+0000","last_became_active":"2023-09-30T01:33:03.291084+0000","last_became_peered":"2023-09-30T01:33:03.291084+0000","last_unstale":"2023-09-30T01:33:03.296569+0000","last_undegraded":"2023-09-30T01:33:03.296569+0000","last_fullsized":"2023-09-30T01:33:03.296569+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T01:33:02.252599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T01:33:02.252599+0000","last_clean_scrub_stamp":"2023-09-30T01:33:02.252599+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T12:10:57.124090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-09-30T01:33:03.299303+0000","last_change":"2023-09-30T01:33:03.296313+0000","last_active":"2023-09-30T01:33:03.299303+0000","last_peered":"2023-09-30T01:33:03.299303+0000","last_clean":"2023-09-30T01:33:03.299303+0000","last_became_active":"2023-09-30T01:33:03.294604+0000","last_became_peered":"2023-09-30T01:33:03.294604+0000","last_unstale":"2023-09-30T01:33:03.299303+0000","last_undegraded":"2023-09-30T01:33:03.299303+0000","last_fullsized":"2023-09-30T01:33:03.299303+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T01:33:02.252599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T01:33:02.252599+0000","last_clean_scrub_stamp":"2023-09-30T01:33:02.252599+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T06:00:29.060679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":1057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":6}],"osd_stats":[{"osd":5,"up_from":11,"seq":47244640320,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5128,"kb_used_data":128,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":94366712,"statfs":{"total":96636764160,"available":96631513088,"internally_reserved":0,"allocated":131072,"data_stored":37781,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 01:38:05 2023","interfaces":[{"interface":"back","average":{"1min":0.76700000000000002,"5min":0.85799999999999998,"15min":0.79400000000000004},"min":{"1min":0.46999999999999997,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":1.028,"5min":8.109,"15min":8.109},"last":0.82499999999999996},{"interface":"front","average":{"1min":0.82199999999999995,"5min":0.88600000000000001,"15min":0.85999999999999999},"min":{"1min":0.57699999999999996,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":1.0009999999999999,"5min":8.3780000000000001,"15min":8.3780000000000001},"last":0.60599999999999998}]},{"osd":1,"last update":"Sat Sep 30 01:38:05 2023","interfaces":[{"interface":"back","average":{"1min":0.751,"5min":0.78300000000000003,"15min":0.77700000000000002},"min":{"1min":0.628,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":0.96999999999999997,"5min":1.087,"15min":1.087},"last":0.751},{"interface":"front","average":{"1min":0.82899999999999996,"5min":0.81399999999999995,"15min":0.81699999999999995},"min":{"1min":0.67100000000000004,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.97899999999999998,"5min":1.5289999999999999,"15min":1.5289999999999999},"last":0.69099999999999995}]},{"osd":2,"last update":"Sat Sep 30 01:38:05 2023","interfaces":[{"interface":"back","average":{"1min":0.83799999999999997,"5min":0.878,"15min":0.86799999999999999},"min":{"1min":0.67200000000000004,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":1.0820000000000001,"5min":4.0759999999999996,"15min":4.0759999999999996},"last":0.79400000000000004},{"interface":"front","average":{"1min":0.872,"5min":0.91500000000000004,"15min":0.86799999999999999},"min":{"1min":0.65400000000000003,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":1.0629999999999999,"5min":8.3109999999999999,"15min":8.3109999999999999},"last":0.91500000000000004}]},{"osd":3,"last update":"Sat Sep 30 01:38:05 2023","interfaces":[{"interface":"back","average":{"1min":0.751,"5min":0.751,"15min":0.78700000000000003},"min":{"1min":0.51600000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.95699999999999996,"5min":1.446,"15min":1.446},"last":0.65300000000000002},{"interface":"front","average":{"1min":0.98899999999999999,"5min":0.94299999999999995,"15min":0.95899999999999996},"min":{"1min":0.84299999999999997,"5min":0.53900000000000003,"15min":0.53900000000000003},"max":{"1min":1.1299999999999999,"5min":1.4790000000000001,"15min":1.4790000000000001},"last":0.996}]},{"osd":4,"last update":"Sat Sep 30 01:38:05 2023","interfaces":[{"interface":"back","average":{"1min":0.85299999999999998,"5min":0.88400000000000001,"15min":0.84199999999999997},"min":{"1min":0.626,"5min":0.441,"15min":0.441},"max":{"1min":1.3109999999999999,"5min":8.2289999999999992,"15min":8.2289999999999992},"last":0.94499999999999995},{"interface":"front","average":{"1min":0.85199999999999998,"5min":0.82199999999999995,"15min":0.83399999999999996},"min":{"1min":0.58999999999999997,"5min":0.52100000000000002,"15min":0.52100000000000002},"max":{"1min":1,"5min":1.2310000000000001,"15min":1.2310000000000001},"last":0.876}]}]},{"osd":4,"up_from":11,"seq":47244640320,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5128,"kb_used_data":128,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":94366712,"statfs":{"total":96636764160,"available":96631513088,"internally_reserved":0,"allocated":131072,"data_stored":37781,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 01:37:10 2023","interfaces":[{"interface":"back","average":{"1min":0.82399999999999995,"5min":0.77300000000000002,"15min":0.76900000000000002},"min":{"1min":0.59299999999999997,"5min":0.52500000000000002,"15min":0.52500000000000002},"max":{"1min":1.0169999999999999,"5min":1.0169999999999999,"15min":1.0169999999999999},"last":0.75800000000000001},{"interface":"front","average":{"1min":0.76500000000000001,"5min":0.80000000000000004,"15min":0.80000000000000004},"min":{"1min":0.58899999999999997,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":1.022,"5min":1.6439999999999999,"15min":1.6439999999999999},"last":0.81899999999999995}]},{"osd":1,"last update":"Sat Sep 30 01:37:10 2023","interfaces":[{"interface":"back","average":{"1min":0.77100000000000002,"5min":0.78700000000000003,"15min":0.78100000000000003},"min":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"max":{"1min":1.028,"5min":1.028,"15min":1.028},"last":1.014},{"interface":"front","average":{"1min":0.81200000000000006,"5min":0.79400000000000004,"15min":0.78500000000000003},"min":{"1min":0.59799999999999998,"5min":0.55800000000000005,"15min":0.55800000000000005},"max":{"1min":0.95999999999999996,"5min":1.548,"15min":1.548},"last":0.96499999999999997}]},{"osd":2,"last update":"Sat Sep 30 01:37:10 2023","interfaces":[{"interface":"back","average":{"1min":0.84499999999999997,"5min":0.84899999999999998,"15min":0.86499999999999999},"min":{"1min":0.71299999999999997,"5min":0.625,"15min":0.625},"max":{"1min":1.0449999999999999,"5min":1.1950000000000001,"15min":1.1950000000000001},"last":0.78600000000000003},{"interface":"front","average":{"1min":0.90200000000000002,"5min":0.92000000000000004,"15min":0.93400000000000005},"min":{"1min":0.67500000000000004,"5min":0.64800000000000002,"15min":0.64800000000000002},"max":{"1min":1.0469999999999999,"5min":1.165,"15min":1.165},"last":0.93500000000000005}]},{"osd":3,"last update":"Sat Sep 30 01:37:10 2023","interfaces":[{"interface":"back","average":{"1min":0.72799999999999998,"5min":0.73199999999999998,"15min":0.73799999999999999},"min":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"max":{"1min":0.996,"5min":1.5,"15min":1.5},"last":0.65000000000000002},{"interface":"front","average":{"1min":0.80800000000000005,"5min":0.80100000000000005,"15min":0.81699999999999995},"min":{"1min":0.61099999999999999,"5min":0.54700000000000004,"15min":0.54700000000000004},"max":{"1min":1.1020000000000001,"5min":1.405,"15min":1.405},"last":0.85899999999999999}]},{"osd":5,"last update":"Sat Sep 30 01:37:10 2023","interfaces":[{"interface":"back","average":{"1min":0.80900000000000005,"5min":0.79000000000000004,"15min":0.77900000000000003},"min":{"1min":0.56000000000000005,"5min":0.54800000000000004,"15min":0.54800000000000004},"max":{"1min":1.0860000000000001,"5min":1.7270000000000001,"15min":1.7270000000000001},"last":0.71799999999999997},{"interface":"front","average":{"1min":0.90600000000000003,"5min":0.90300000000000002,"15min":0.90500000000000003},"min":{"1min":0.80900000000000005,"5min":0.66800000000000004,"15min":0.66800000000000004},"max":{"1min":1.089,"5min":1.0960000000000001,"15min":1.0960000000000001},"last":0.89800000000000002}]}]},{"osd":3,"up_from":11,"seq":47244640321,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6096,"kb_used_data":648,"kb_used_omap":0,"kb_used_meta":5440,"kb_avail":94365744,"statfs":{"total":96636764160,"available":96630521856,"internally_reserved":0,"allocated":663552,"data_stored":566709,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5570560},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 01:38:05 2023","interfaces":[{"interface":"back","average":{"1min":0.81200000000000006,"5min":0.79200000000000004,"15min":0.77600000000000002},"min":{"1min":0.60899999999999999,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":0.95299999999999996,"5min":1.052,"15min":1.052},"last":1.0189999999999999},{"interface":"front","average":{"1min":0.77400000000000002,"5min":0.75900000000000001,"15min":0.73799999999999999},"min":{"1min":0.59799999999999998,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":1.018,"5min":1.206,"15min":1.206},"last":0.61599999999999999}]},{"osd":1,"last update":"Sat Sep 30 01:38:05 2023","interfaces":[{"interface":"back","average":{"1min":0.84899999999999998,"5min":0.83499999999999996,"15min":0.83099999999999996},"min":{"1min":0.70899999999999996,"5min":0.56599999999999995,"15min":0.56599999999999995},"max":{"1min":1.083,"5min":1.159,"15min":1.159},"last":0.93100000000000005},{"interface":"front","average":{"1min":0.85699999999999998,"5min":0.79700000000000004,"15min":0.76800000000000002},"min":{"1min":0.60099999999999998,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":1.1140000000000001,"5min":1.137,"15min":1.137},"last":0.95599999999999996}]},{"osd":2,"last update":"Sat Sep 30 01:38:05 2023","interfaces":[{"interface":"back","average":{"1min":0.91700000000000004,"5min":0.89900000000000002,"15min":0.93700000000000006},"min":{"1min":0.64500000000000002,"5min":0.60599999999999998,"15min":0.60599999999999998},"max":{"1min":1.0860000000000001,"5min":1.296,"15min":1.296},"last":0.67200000000000004},{"interface":"front","average":{"1min":0.84699999999999998,"5min":0.88900000000000001,"15min":0.91900000000000004},"min":{"1min":0.65900000000000003,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":1.1559999999999999,"5min":1.46,"15min":1.46},"last":0.77200000000000002}]},{"osd":4,"last update":"Sat Sep 30 01:38:05 2023","interfaces":[{"interface":"back","average":{"1min":0.71199999999999997,"5min":0.73299999999999998,"15min":0.73299999999999998},"min":{"1min":0.55200000000000005,"5min":0.51300000000000001,"15min":0.51300000000000001},"max":{"1min":0.95599999999999996,"5min":1.3220000000000001,"15min":1.3220000000000001},"last":0.81599999999999995},{"interface":"front","average":{"1min":0.82599999999999996,"5min":0.82899999999999996,"15min":0.872},"min":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":1.179,"5min":1.179,"15min":1.179},"last":0.84199999999999997}]},{"osd":5,"last update":"Sat Sep 30 01:38:05 2023","interfaces":[{"interface":"back","average":{"1min":0.82999999999999996,"5min":0.81399999999999995,"15min":0.82799999999999996},"min":{"1min":0.64400000000000002,"5min":0.61099999999999999,"15min":0.61099999999999999},"max":{"1min":1.1439999999999999,"5min":1.3320000000000001,"15min":1.3320000000000001},"last":0.73199999999999998},{"interface":"front","average":{"1min":0.78100000000000003,"5min":0.82599999999999996,"15min":0.86399999999999999},"min":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"max":{"1min":1.054,"5min":1.387,"15min":1.387},"last":0.88100000000000001}]}]},{"osd":2,"up_from":11,"seq":47244640322,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5128,"kb_used_data":128,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":94366712,"statfs":{"total":96636764160,"available":96631513088,"internally_reserved":0,"allocated":131072,"data_stored":37781,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 01:38:07 2023","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.57999999999999996,"15min":0.56599999999999995},"min":{"1min":0.46999999999999997,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.79100000000000004,"5min":0.95099999999999996,"15min":0.95099999999999996},"last":0.55000000000000004},{"interface":"front","average":{"1min":0.66800000000000004,"5min":0.68000000000000005,"15min":0.65800000000000003},"min":{"1min":0.53600000000000003,"5min":0.439,"15min":0.439},"max":{"1min":0.96599999999999997,"5min":2.2029999999999998,"15min":2.2029999999999998},"last":0.77900000000000003}]},{"osd":1,"last update":"Sat Sep 30 01:38:07 2023","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.68400000000000005,"15min":0.70099999999999996},"min":{"1min":0.437,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.85599999999999998,"5min":2.2490000000000001,"15min":2.2490000000000001},"last":0.69099999999999995},{"interface":"front","average":{"1min":0.68799999999999994,"5min":0.65200000000000002,"15min":0.65700000000000003},"min":{"1min":0.46500000000000002,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.80700000000000005,"5min":1.0409999999999999,"15min":1.0409999999999999},"last":0.60999999999999999}]},{"osd":3,"last update":"Sat Sep 30 01:38:07 2023","interfaces":[{"interface":"back","average":{"1min":0.79600000000000004,"5min":0.82099999999999995,"15min":0.83299999999999996},"min":{"1min":0.59399999999999997,"5min":0.59399999999999997,"15min":0.59399999999999997},"max":{"1min":1.0309999999999999,"5min":1.343,"15min":1.343},"last":0.81799999999999995},{"interface":"front","average":{"1min":0.875,"5min":0.91200000000000003,"15min":0.92200000000000004},"min":{"1min":0.63,"5min":0.60499999999999998,"15min":0.60499999999999998},"max":{"1min":1.083,"5min":2.4390000000000001,"15min":2.4390000000000001},"last":0.92100000000000004}]},{"osd":4,"last update":"Sat Sep 30 01:38:07 2023","interfaces":[{"interface":"back","average":{"1min":0.82899999999999996,"5min":0.88300000000000001,"15min":0.90300000000000002},"min":{"1min":0.67400000000000004,"5min":0.64300000000000002,"15min":0.64300000000000002},"max":{"1min":1.0349999999999999,"5min":2.3679999999999999,"15min":2.3679999999999999},"last":0.878},{"interface":"front","average":{"1min":0.878,"5min":0.85399999999999998,"15min":0.84999999999999998},"min":{"1min":0.67300000000000004,"5min":0.505,"15min":0.505},"max":{"1min":1.0169999999999999,"5min":1.401,"15min":1.401},"last":0.85699999999999998}]},{"osd":5,"last update":"Sat Sep 30 01:38:07 2023","interfaces":[{"interface":"back","average":{"1min":0.88100000000000001,"5min":0.86499999999999999,"15min":0.86699999999999999},"min":{"1min":0.70099999999999996,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":1.085,"5min":1.3959999999999999,"15min":1.3959999999999999},"last":0.98099999999999998},{"interface":"front","average":{"1min":0.88500000000000001,"5min":0.90600000000000003,"15min":0.92400000000000004},"min":{"1min":0.65900000000000003,"5min":0.61799999999999999,"15min":0.61799999999999999},"max":{"1min":1.091,"5min":1.625,"15min":1.625},"last":0.95099999999999996}]}]},{"osd":1,"up_from":11,"seq":47244640321,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6096,"kb_used_data":648,"kb_used_omap":0,"kb_used_meta":5440,"kb_avail":94365744,"statfs":{"total":96636764160,"available":96630521856,"internally_reserved":0,"allocated":663552,"data_stored":566709,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5570560},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 01:38:08 2023","interfaces":[{"interface":"back","average":{"1min":0.59199999999999997,"5min":0.57699999999999996,"15min":0.57299999999999995},"min":{"1min":0.38400000000000001,"5min":0.317,"15min":0.317},"max":{"1min":0.90500000000000003,"5min":0.92400000000000004,"15min":0.92400000000000004},"last":0.57999999999999996},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.64500000000000002,"15min":0.64900000000000002},"min":{"1min":0.50900000000000001,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.72699999999999998,"5min":2.085,"15min":2.085},"last":0.63800000000000001}]},{"osd":2,"last update":"Sat Sep 30 01:38:08 2023","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.625,"15min":0.66300000000000003},"min":{"1min":0.34399999999999997,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.80200000000000005,"5min":1.159,"15min":1.159},"last":0.70099999999999996},{"interface":"front","average":{"1min":0.64800000000000002,"5min":0.68799999999999994,"15min":0.69699999999999995},"min":{"1min":0.40500000000000003,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.79500000000000004,"5min":1.9930000000000001,"15min":1.9930000000000001},"last":0.79400000000000004}]},{"osd":3,"last update":"Sat Sep 30 01:38:08 2023","interfaces":[{"interface":"back","average":{"1min":0.81299999999999994,"5min":0.81999999999999995,"15min":0.82399999999999995},"min":{"1min":0.65000000000000002,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":1.018,"5min":1.1499999999999999,"15min":1.1499999999999999},"last":0.83299999999999996},{"interface":"front","average":{"1min":0.85299999999999998,"5min":0.83499999999999996,"15min":0.84099999999999997},"min":{"1min":0.66800000000000004,"5min":0.61699999999999999,"15min":0.61699999999999999},"max":{"1min":1.0960000000000001,"5min":1.2929999999999999,"15min":1.2929999999999999},"last":1.0960000000000001}]},{"osd":4,"last update":"Sat Sep 30 01:38:08 2023","interfaces":[{"interface":"back","average":{"1min":0.84899999999999998,"5min":0.85499999999999998,"15min":0.878},"min":{"1min":0.64300000000000002,"5min":0.53600000000000003,"15min":0.53600000000000003},"max":{"1min":0.98999999999999999,"5min":1.415,"15min":1.415},"last":0.98999999999999999},{"interface":"front","average":{"1min":0.874,"5min":0.872,"15min":0.88600000000000001},"min":{"1min":0.70499999999999996,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":0.99199999999999999,"5min":1.381,"15min":1.381},"last":0.96099999999999997}]},{"osd":5,"last update":"Sat Sep 30 01:38:08 2023","interfaces":[{"interface":"back","average":{"1min":0.85599999999999998,"5min":0.89600000000000002,"15min":0.90200000000000002},"min":{"1min":0.625,"5min":0.625,"15min":0.625},"max":{"1min":1.03,"5min":1.395,"15min":1.395},"last":1.03},{"interface":"front","average":{"1min":0.96199999999999997,"5min":0.95699999999999996,"15min":0.96299999999999997},"min":{"1min":0.67500000000000004,"5min":0.56299999999999994,"15min":0.56299999999999994},"max":{"1min":1.1379999999999999,"5min":1.4179999999999999,"15min":1.4179999999999999},"last":0.93400000000000005}]}]},{"osd":0,"up_from":11,"seq":47244640322,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5128,"kb_used_data":128,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":94366712,"statfs":{"total":96636764160,"available":96631513088,"internally_reserved":0,"allocated":131072,"data_stored":37781,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Sep 30 01:38:07 2023","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.57799999999999996,"15min":0.59099999999999997},"min":{"1min":0.433,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.75,"5min":0.92900000000000005,"15min":0.92900000000000005},"last":0.56100000000000005},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.63700000000000001,"15min":0.63900000000000001},"min":{"1min":0.45400000000000001,"5min":0.314,"15min":0.314},"max":{"1min":0.81999999999999995,"5min":0.98599999999999999,"15min":0.98599999999999999},"last":0.52900000000000003}]},{"osd":2,"last update":"Sat Sep 30 01:38:07 2023","interfaces":[{"interface":"back","average":{"1min":0.60599999999999998,"5min":0.627,"15min":0.61899999999999999},"min":{"1min":0.41199999999999998,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.86299999999999999,"5min":0.93799999999999994,"15min":0.93799999999999994},"last":0.64900000000000002},{"interface":"front","average":{"1min":0.66300000000000003,"5min":0.71399999999999997,"15min":0.70599999999999996},"min":{"1min":0.45100000000000001,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":0.83699999999999997,"5min":2.069,"15min":2.069},"last":0.59599999999999997}]},{"osd":3,"last update":"Sat Sep 30 01:38:07 2023","interfaces":[{"interface":"back","average":{"1min":0.83199999999999996,"5min":0.83699999999999997,"15min":0.85399999999999998},"min":{"1min":0.67700000000000005,"5min":0.60299999999999998,"15min":0.60299999999999998},"max":{"1min":1.081,"5min":1.1319999999999999,"15min":1.1319999999999999},"last":0.73999999999999999},{"interface":"front","average":{"1min":0.82699999999999996,"5min":0.872,"15min":0.876},"min":{"1min":0.66200000000000003,"5min":0.62,"15min":0.62},"max":{"1min":1.024,"5min":2.4870000000000001,"15min":2.4870000000000001},"last":0.79400000000000004}]},{"osd":4,"last update":"Sat Sep 30 01:38:07 2023","interfaces":[{"interface":"back","average":{"1min":0.83899999999999997,"5min":0.90600000000000003,"15min":0.92000000000000004},"min":{"1min":0.69299999999999995,"5min":0.66300000000000003,"15min":0.66300000000000003},"max":{"1min":1.0720000000000001,"5min":2.589,"15min":2.589},"last":0.83099999999999996},{"interface":"front","average":{"1min":0.82399999999999995,"5min":0.85799999999999998,"15min":0.90200000000000002},"min":{"1min":0.67900000000000005,"5min":0.54900000000000004,"15min":0.54900000000000004},"max":{"1min":1.0189999999999999,"5min":1.413,"15min":1.413},"last":0.86499999999999999}]},{"osd":5,"last update":"Sat Sep 30 01:38:07 2023","interfaces":[{"interface":"back","average":{"1min":0.85999999999999999,"5min":0.89300000000000002,"15min":0.92400000000000004},"min":{"1min":0.68300000000000005,"5min":0.58799999999999997,"15min":0.58799999999999997},"max":{"1min":1.1220000000000001,"5min":1.3200000000000001,"15min":1.3200000000000001},"last":0.68300000000000005},{"interface":"front","average":{"1min":0.88900000000000001,"5min":0.94799999999999995,"15min":0.96399999999999997},"min":{"1min":0.749,"5min":0.73999999999999999,"15min":0.73999999999999999},"max":{"1min":1.0609999999999999,"5min":2.4279999999999999,"15min":2.4279999999999999},"last":0.91500000000000004}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-09-30T01:38:14.812 INFO:tasks.ceph:pgid 1.7 last_scrub_stamp 2023-09-30T01:33:02.252599+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=30, tm_hour=1, tm_min=33, tm_sec=2, tm_wday=5, tm_yday=273, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=30, tm_hour=1, tm_min=38, tm_sec=10, tm_wday=5, tm_yday=273, tm_isdst=0) 2023-09-30T01:38:14.812 INFO:tasks.ceph:pgid 1.6 last_scrub_stamp 2023-09-30T01:33:02.252599+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=30, tm_hour=1, tm_min=33, tm_sec=2, tm_wday=5, tm_yday=273, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=30, tm_hour=1, tm_min=38, tm_sec=10, tm_wday=5, tm_yday=273, tm_isdst=0) 2023-09-30T01:38:14.813 INFO:tasks.ceph:pgid 1.5 last_scrub_stamp 2023-09-30T01:33:02.252599+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=30, tm_hour=1, tm_min=33, tm_sec=2, tm_wday=5, tm_yday=273, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=30, tm_hour=1, tm_min=38, tm_sec=10, tm_wday=5, tm_yday=273, tm_isdst=0) 2023-09-30T01:38:14.813 INFO:tasks.ceph:pgid 1.4 last_scrub_stamp 2023-09-30T01:33:02.252599+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=30, tm_hour=1, tm_min=33, tm_sec=2, tm_wday=5, tm_yday=273, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=30, tm_hour=1, tm_min=38, tm_sec=10, tm_wday=5, tm_yday=273, tm_isdst=0) 2023-09-30T01:38:14.813 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2023-09-30T01:33:02.252599+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=30, tm_hour=1, tm_min=33, tm_sec=2, tm_wday=5, tm_yday=273, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=30, tm_hour=1, tm_min=38, tm_sec=10, tm_wday=5, tm_yday=273, tm_isdst=0) 2023-09-30T01:38:14.813 INFO:tasks.ceph:pgid 1.3 last_scrub_stamp 2023-09-30T01:33:02.252599+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=30, tm_hour=1, tm_min=33, tm_sec=2, tm_wday=5, tm_yday=273, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=30, tm_hour=1, tm_min=38, tm_sec=10, tm_wday=5, tm_yday=273, tm_isdst=0) 2023-09-30T01:38:14.813 INFO:tasks.ceph:pgid 1.2 last_scrub_stamp 2023-09-30T01:33:02.252599+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=30, tm_hour=1, tm_min=33, tm_sec=2, tm_wday=5, tm_yday=273, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=30, tm_hour=1, tm_min=38, tm_sec=10, tm_wday=5, tm_yday=273, tm_isdst=0) 2023-09-30T01:38:14.813 INFO:tasks.ceph:pgid 1.1 last_scrub_stamp 2023-09-30T01:33:02.252599+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=30, tm_hour=1, tm_min=33, tm_sec=2, tm_wday=5, tm_yday=273, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=30, tm_hour=1, tm_min=38, tm_sec=10, tm_wday=5, tm_yday=273, tm_isdst=0) 2023-09-30T01:38:14.813 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2023-09-30T01:33:02.252599+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=30, tm_hour=1, tm_min=33, tm_sec=2, tm_wday=5, tm_yday=273, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=30, tm_hour=1, tm_min=38, tm_sec=10, tm_wday=5, tm_yday=273, tm_isdst=0) 2023-09-30T01:38:14.814 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-09-30T01:38:34.815 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-09-30T01:38:34.932 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:34.930+0000 7f33d2cb4700 10 monclient: get_monmap_and_config 2023-09-30T01:38:34.933 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:34.930+0000 7f33d2cb4700 10 monclient: build_initial_monmap 2023-09-30T01:38:34.933 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:34.930+0000 7f33d2cb4700 10 monclient: monmap: 2023-09-30T01:38:34.933 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2023-09-30T01:38:34.933 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:38:34.933 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:38:34.932115+0000 2023-09-30T01:38:34.933 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:38:34.932115+0000 2023-09-30T01:38:34.933 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2023-09-30T01:38:34.933 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:38:34.933 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a-legacy 2023-09-30T01:38:34.933 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.107:6789/0 mon.noname-f-legacy 2023-09-30T01:38:34.933 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.noname-b 2023-09-30T01:38:34.934 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.noname-g 2023-09-30T01:38:34.934 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.noname-c 2023-09-30T01:38:34.934 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.noname-h 2023-09-30T01:38:34.934 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.noname-d 2023-09-30T01:38:34.934 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.noname-i 2023-09-30T01:38:34.934 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.noname-e 2023-09-30T01:38:34.934 INFO:teuthology.orchestra.run.smithi088.stderr:9: v2:172.21.15.88:3300/0 mon.noname-a 2023-09-30T01:38:34.934 INFO:teuthology.orchestra.run.smithi088.stderr:10: v2:172.21.15.107:3300/0 mon.noname-f 2023-09-30T01:38:34.934 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:38:34.934 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:34.930+0000 7f33d2cb4700 1 Processor -- start 2023-09-30T01:38:34.934 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:34.930+0000 7f33d2cb4700 1 -- start start 2023-09-30T01:38:34.934 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:34.930+0000 7f33d2cb4700 10 monclient: init 2023-09-30T01:38:34.937 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:34.930+0000 7f33d2cb4700 10 monclient: _reopen_session rank -1 2023-09-30T01:38:34.937 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:34.930+0000 7f33d2cb4700 10 monclient: _add_conns ranks=[1,6,2,8,10,4,7,0,9,5,3] 2023-09-30T01:38:34.937 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:34.930+0000 7f33d2cb4700 10 monclient(hunting): picked mon.noname-f-legacy con 0x7f33cc097a40 addr v1:172.21.15.107:6789/0 2023-09-30T01:38:34.938 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:34.930+0000 7f33d2cb4700 1 --2- >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7f33cc132e10 0x7f33cc138650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:38:34.938 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:34.930+0000 7f33d2cb4700 10 monclient(hunting): picked mon.noname-d con 0x7f33cc132e10 addr [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] 2023-09-30T01:38:34.938 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:34.930+0000 7f33d2cb4700 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f33cc1311c0 0x7f33cc13aba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:38:34.938 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:34.930+0000 7f33d2cb4700 10 monclient(hunting): picked mon.noname-b con 0x7f33cc1311c0 addr [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] 2023-09-30T01:38:34.938 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:34.930+0000 7f33d2cb4700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33cc04bb20 con 0x7f33cc097a40 2023-09-30T01:38:34.938 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:34.930+0000 7f33d2cb4700 10 monclient(hunting): start opening mon connection 2023-09-30T01:38:34.938 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:34.930+0000 7f33d2cb4700 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33cc09a5d0 con 0x7f33cc1311c0 2023-09-30T01:38:34.938 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:34.930+0000 7f33d2cb4700 10 monclient(hunting): start opening mon connection 2023-09-30T01:38:34.938 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:34.930+0000 7f33d2cb4700 1 -- --> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] -- mon_getmap magic: 0 v1 -- 0x7f33cc097e10 con 0x7f33cc132e10 2023-09-30T01:38:34.938 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:34.930+0000 7f33d2cb4700 10 monclient(hunting): _renew_subs 2023-09-30T01:38:34.938 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:34.930+0000 7f33d2cb4700 10 monclient(hunting): authenticate will time out at 1186.969613s 2023-09-30T01:38:34.939 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:34.930+0000 7f33d14b1700 1 --2- >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7f33cc132e10 0x7f33cc138650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:38:34.939 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:34.930+0000 7f33d14b1700 1 --2- >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7f33cc132e10 0x7f33cc138650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3303/0 says I am v2:172.21.15.88:40018/0 (socket says 172.21.15.88:40018) 2023-09-30T01:38:34.939 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:34.930+0000 7f33d14b1700 1 -- 172.21.15.88:0/3309259688 learned_addr learned my addr 172.21.15.88:0/3309259688 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-09-30T01:38:34.939 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:34.930+0000 7f33d14b1700 10 monclient(hunting): get_auth_request con 0x7f33cc132e10 auth_method 0 2023-09-30T01:38:34.939 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:34.930+0000 7f33d14b1700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:38:34.939 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:34.930+0000 7f33d14b1700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:38:34.939 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:34.930+0000 7f33d14b1700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:38:34.939 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:34.930+0000 7f33d24b3700 1 --2- 172.21.15.88:0/3309259688 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f33cc1311c0 0x7f33cc13aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:38:34.939 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:34.930+0000 7f33d24b3700 10 monclient(hunting): get_auth_request con 0x7f33cc1311c0 auth_method 0 2023-09-30T01:38:34.939 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:34.930+0000 7f33d24b3700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:38:34.939 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:34.930+0000 7f33d24b3700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:38:34.939 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:34.930+0000 7f33d24b3700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:38:34.940 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:34.930+0000 7f33d14b1700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:38:34.940 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:34.930+0000 7f33d14b1700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:38:34.940 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:34.930+0000 7f33d14b1700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:38:34.940 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:34.930+0000 7f33d24b3700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:38:34.940 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:34.930+0000 7f33d24b3700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:38:34.940 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:34.930+0000 7f33d24b3700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:38:34.940 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:34.930+0000 7f33d14b1700 10 monclient(hunting): handle_auth_done global_id 4875 payload 274 2023-09-30T01:38:34.940 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:34.930+0000 7f33d14b1700 1 -- 172.21.15.88:0/3309259688 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f33cc1311c0 msgr2=0x7f33cc13aba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:38:34.940 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:34.930+0000 7f33d14b1700 1 --2- 172.21.15.88:0/3309259688 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f33cc1311c0 0x7f33cc13aba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:38:34.940 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:34.930+0000 7f33d14b1700 1 -- 172.21.15.88:0/3309259688 >> v1:172.21.15.107:6789/0 conn(0x7f33cc097a40 legacy=0x7f33cc132700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:38:34.940 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:34.930+0000 7f33d14b1700 10 monclient: _finish_hunting 0 2023-09-30T01:38:34.940 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:34.930+0000 7f33d14b1700 1 monclient: found mon.noname-d 2023-09-30T01:38:34.941 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:34.930+0000 7f33d14b1700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:38:34.941 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:34.930+0000 7f33d14b1700 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.88:3303/0 2023-09-30T01:38:34.941 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:34.930+0000 7f33d14b1700 1 -- 172.21.15.88:0/3309259688 --> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33cc098d40 con 0x7f33cc132e10 2023-09-30T01:38:34.941 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:34.930+0000 7f33d14b1700 1 --2- 172.21.15.88:0/3309259688 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7f33cc132e10 0x7f33cc138650 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f33c8009f50 tx=0x7f33c8009840 comp rx=0 tx=0).ready entity=mon.6 client_cookie=e8776c714f06ec40 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:38:34.941 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:34.930+0000 7f33d0cb0700 1 -- 172.21.15.88:0/3309259688 <== mon.6 v2:172.21.15.88:3303/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f33c800e070 con 0x7f33cc132e10 2023-09-30T01:38:34.941 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:34.930+0000 7f33d0cb0700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:38:34.941 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:34.930+0000 7f33d0cb0700 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2023-09-30T01:38:34.941 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:34.930+0000 7f33d0cb0700 10 monclient: dump: 2023-09-30T01:38:34.941 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:38:34.941 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:38:34.942 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:38:34.942 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:38:34.942 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:38:34.942 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:38:34.942 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:38:34.942 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:38:34.942 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:38:34.942 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:38:34.942 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:38:34.942 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:38:34.942 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:38:34.942 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:38:34.942 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:38:34.943 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:38:34.943 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:34.930+0000 7f33d0cb0700 10 monclient: _finish_auth 0 2023-09-30T01:38:34.943 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:34.930+0000 7f33d0cb0700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:38:34.943 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:34.930+0000 7f33d0cb0700 1 -- 172.21.15.88:0/3309259688 <== mon.6 v2:172.21.15.88:3303/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f33c800be80 con 0x7f33cc132e10 2023-09-30T01:38:34.943 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:34.930+0000 7f33d2cb4700 5 monclient: authenticate success, global_id 4875 2023-09-30T01:38:34.943 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:34.930+0000 7f33d2cb4700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:38:34.943 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:34.930+0000 7f33d0cb0700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:38:34.943 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:34.930+0000 7f33d2cb4700 10 monclient: get_monmap_and_config success 2023-09-30T01:38:34.943 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:34.930+0000 7f33d0cb0700 1 -- 172.21.15.88:0/3309259688 <== mon.6 v2:172.21.15.88:3303/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f33c80092e0 con 0x7f33cc132e10 2023-09-30T01:38:34.943 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:34.930+0000 7f33d0cb0700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:38:34.943 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:34.930+0000 7f33d2cb4700 4 set_mon_vals no callback set 2023-09-30T01:38:34.943 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:34.930+0000 7f33d0cb0700 10 monclient: got monmap 1 from mon.d (according to old e1) 2023-09-30T01:38:34.944 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:34.930+0000 7f33d0cb0700 10 monclient: dump: 2023-09-30T01:38:34.944 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:38:34.944 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:38:34.944 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:38:34.944 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:38:34.944 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:38:34.944 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:38:34.944 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:38:34.944 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:38:34.944 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:38:34.944 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:38:34.944 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:38:34.945 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:38:34.945 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:38:34.945 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:38:34.945 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:38:34.945 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:38:34.945 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:34.934+0000 7f33d2cb4700 10 monclient: shutdown 2023-09-30T01:38:34.945 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:34.934+0000 7f33d2cb4700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:38:34.945 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:34.934+0000 7f33d2cb4700 1 -- 172.21.15.88:0/3309259688 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7f33cc132e10 msgr2=0x7f33cc138650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:38:34.945 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:34.934+0000 7f33d2cb4700 1 --2- 172.21.15.88:0/3309259688 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7f33cc132e10 0x7f33cc138650 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f33c8009f50 tx=0x7f33c8009840 comp rx=0 tx=0).stop 2023-09-30T01:38:34.945 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:34.934+0000 7f33d2cb4700 1 -- 172.21.15.88:0/3309259688 shutdown_connections 2023-09-30T01:38:34.945 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:34.934+0000 7f33d2cb4700 1 --2- 172.21.15.88:0/3309259688 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7f33cc132e10 0x7f33cc138650 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:38:34.945 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:34.934+0000 7f33d2cb4700 1 --2- 172.21.15.88:0/3309259688 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f33cc1311c0 0x7f33cc13aba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:38:34.946 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:34.934+0000 7f33d2cb4700 1 -- 172.21.15.88:0/3309259688 >> 172.21.15.88:0/3309259688 conn(0x7f33cc06e5e0 msgr2=0x7f33cc06e9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:38:34.946 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:34.934+0000 7f33d2cb4700 1 -- 172.21.15.88:0/3309259688 shutdown_connections 2023-09-30T01:38:34.946 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:34.934+0000 7f33d2cb4700 1 -- 172.21.15.88:0/3309259688 wait complete. 2023-09-30T01:38:34.946 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:34.934+0000 7f33d2cb4700 10 monclient: build_initial_monmap 2023-09-30T01:38:34.946 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:34.934+0000 7f33d2cb4700 10 monclient: monmap: 2023-09-30T01:38:34.946 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2023-09-30T01:38:34.946 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:38:34.946 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 0.000000 2023-09-30T01:38:34.946 INFO:teuthology.orchestra.run.smithi088.stderr:created 0.000000 2023-09-30T01:38:34.946 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2023-09-30T01:38:34.946 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 1 2023-09-30T01:38:34.946 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.noname-a 2023-09-30T01:38:34.946 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.noname-f 2023-09-30T01:38:34.947 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.noname-b 2023-09-30T01:38:34.947 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.noname-g 2023-09-30T01:38:34.947 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.noname-c 2023-09-30T01:38:34.947 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.noname-h 2023-09-30T01:38:34.947 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.noname-d 2023-09-30T01:38:34.947 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.noname-i 2023-09-30T01:38:34.947 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.noname-e 2023-09-30T01:38:34.947 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:38:34.952 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:34.934+0000 7f33d2cb4700 1 Processor -- start 2023-09-30T01:38:34.952 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:34.934+0000 7f33d2cb4700 1 -- start start 2023-09-30T01:38:34.952 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:34.934+0000 7f33d2cb4700 10 monclient: init 2023-09-30T01:38:34.952 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:34.934+0000 7f33d2cb4700 10 monclient: _reopen_session rank -1 2023-09-30T01:38:34.952 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:34.934+0000 7f33d2cb4700 10 monclient: _add_conns ranks=[4,3,5,8,7,0,6,2,1] 2023-09-30T01:38:34.952 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:34.934+0000 7f33d2cb4700 1 --2- >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f33cc097a40 0x7f33cc0a20f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:38:34.952 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:34.934+0000 7f33d2cb4700 10 monclient(hunting): picked mon.noname-c con 0x7f33cc097a40 addr [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] 2023-09-30T01:38:34.952 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:34.934+0000 7f33d2cb4700 1 --2- >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7f33cc1311c0 0x7f33cc130120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:38:34.952 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:34.934+0000 7f33d2cb4700 10 monclient(hunting): picked mon.noname-g con 0x7f33cc1311c0 addr [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] 2023-09-30T01:38:34.953 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:34.934+0000 7f33d2cb4700 1 --2- >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7f33cc132e10 0x7f33cc11cd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:38:34.953 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:34.934+0000 7f33d2cb4700 10 monclient(hunting): picked mon.noname-h con 0x7f33cc132e10 addr [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] 2023-09-30T01:38:34.953 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:34.934+0000 7f33d2cb4700 10 monclient(hunting): start opening mon connection 2023-09-30T01:38:34.953 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:34.934+0000 7f33d2cb4700 1 -- --> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33cc099c30 con 0x7f33cc1311c0 2023-09-30T01:38:34.953 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:34.934+0000 7f33d2cb4700 10 monclient(hunting): start opening mon connection 2023-09-30T01:38:34.953 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:34.934+0000 7f33d2cb4700 1 -- --> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] -- mon_getmap magic: 0 v1 -- 0x7f33cc0996b0 con 0x7f33cc097a40 2023-09-30T01:38:34.953 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:34.934+0000 7f33d2cb4700 10 monclient(hunting): start opening mon connection 2023-09-30T01:38:34.953 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:34.934+0000 7f33d2cb4700 1 -- --> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] -- mon_getmap magic: 0 v1 -- 0x7f33cc11df80 con 0x7f33cc132e10 2023-09-30T01:38:34.953 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:34.934+0000 7f33d2cb4700 10 monclient(hunting): _renew_subs 2023-09-30T01:38:34.953 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:34.934+0000 7f33d2cb4700 10 monclient(hunting): authenticate will time out at 1186.975461s 2023-09-30T01:38:34.953 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:34.934+0000 7f33d1cb2700 1 --2- >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f33cc097a40 0x7f33cc0a20f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:38:34.953 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:34.934+0000 7f33d1cb2700 1 --2- >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f33cc097a40 0x7f33cc0a20f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3302/0 says I am v2:172.21.15.88:53036/0 (socket says 172.21.15.88:53036) 2023-09-30T01:38:34.954 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:34.934+0000 7f33d14b1700 1 --2- >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7f33cc1311c0 0x7f33cc130120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:38:34.954 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:34.934+0000 7f33d1cb2700 1 -- 172.21.15.88:0/3674368722 learned_addr learned my addr 172.21.15.88:0/3674368722 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-09-30T01:38:34.954 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:34.934+0000 7f33d1cb2700 10 monclient(hunting): get_auth_request con 0x7f33cc097a40 auth_method 0 2023-09-30T01:38:34.954 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:34.934+0000 7f33d1cb2700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:38:34.954 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:34.934+0000 7f33d24b3700 1 --2- 172.21.15.88:0/3674368722 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7f33cc132e10 0x7f33cc11cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:38:34.954 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:34.934+0000 7f33d1cb2700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:38:34.954 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:34.934+0000 7f33d1cb2700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:38:34.954 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:34.934+0000 7f33d14b1700 10 monclient(hunting): get_auth_request con 0x7f33cc1311c0 auth_method 0 2023-09-30T01:38:34.954 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:34.934+0000 7f33d14b1700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:38:34.954 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:34.934+0000 7f33d14b1700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:38:34.954 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:34.934+0000 7f33d14b1700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:38:34.954 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:34.934+0000 7f33d24b3700 10 monclient(hunting): get_auth_request con 0x7f33cc132e10 auth_method 0 2023-09-30T01:38:34.955 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:34.934+0000 7f33d24b3700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:38:34.955 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:34.934+0000 7f33d24b3700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:38:34.955 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:34.934+0000 7f33d24b3700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:38:34.955 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:34.938+0000 7f33d1cb2700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:38:34.955 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:34.938+0000 7f33d1cb2700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:38:34.955 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:34.938+0000 7f33d1cb2700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:38:34.955 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:34.938+0000 7f33d14b1700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:38:34.955 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:34.938+0000 7f33d14b1700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:38:34.955 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:34.938+0000 7f33d14b1700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:38:34.955 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:34.938+0000 7f33d24b3700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:38:34.955 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:34.938+0000 7f33d24b3700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:38:34.955 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:34.938+0000 7f33d24b3700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:38:34.956 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:34.938+0000 7f33d1cb2700 10 monclient(hunting): handle_auth_done global_id 5008 payload 824 2023-09-30T01:38:34.956 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:34.938+0000 7f33d1cb2700 1 -- 172.21.15.88:0/3674368722 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7f33cc132e10 msgr2=0x7f33cc11cd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:38:34.956 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:34.938+0000 7f33d1cb2700 1 --2- 172.21.15.88:0/3674368722 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7f33cc132e10 0x7f33cc11cd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:38:34.956 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:34.938+0000 7f33d1cb2700 1 -- 172.21.15.88:0/3674368722 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7f33cc1311c0 msgr2=0x7f33cc130120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:38:34.956 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:34.938+0000 7f33d1cb2700 1 --2- 172.21.15.88:0/3674368722 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7f33cc1311c0 0x7f33cc130120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:38:34.956 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:34.938+0000 7f33d1cb2700 10 monclient: _finish_hunting 0 2023-09-30T01:38:34.956 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:34.938+0000 7f33d1cb2700 1 monclient: found mon.noname-c 2023-09-30T01:38:34.956 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:34.938+0000 7f33d1cb2700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:38:34.956 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:34.938+0000 7f33d1cb2700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.88:3302/0 2023-09-30T01:38:34.956 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:34.938+0000 7f33d1cb2700 1 -- 172.21.15.88:0/3674368722 --> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33c8009ae0 con 0x7f33cc097a40 2023-09-30T01:38:34.956 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:34.938+0000 7f33d24b3700 1 --2- 172.21.15.88:0/3674368722 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7f33cc132e10 0x7f33cc11cd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:38:34.956 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:34.938+0000 7f33d1cb2700 1 --2- 172.21.15.88:0/3674368722 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f33cc097a40 0x7f33cc0a20f0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f33bc00c790 tx=0x7f33bc007f40 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:38:34.957 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:34.938+0000 7f33c1ffb700 1 -- 172.21.15.88:0/3674368722 <== mon.4 v2:172.21.15.88:3302/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f33bc011a10 con 0x7f33cc097a40 2023-09-30T01:38:34.957 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:34.938+0000 7f33c1ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:38:34.957 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:34.938+0000 7f33c1ffb700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T01:38:34.957 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:34.938+0000 7f33c1ffb700 10 monclient: dump: 2023-09-30T01:38:34.957 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:38:34.957 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:38:34.957 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:38:34.957 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:38:34.957 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:38:34.957 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:38:34.957 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:38:34.957 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:38:34.958 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:38:34.958 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:38:34.958 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:38:34.958 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:38:34.958 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:38:34.958 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:38:34.958 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:38:34.958 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:38:34.958 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:34.938+0000 7f33c1ffb700 10 monclient: _finish_auth 0 2023-09-30T01:38:34.958 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:34.938+0000 7f33c1ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:38:34.958 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:34.938+0000 7f33c1ffb700 1 -- 172.21.15.88:0/3674368722 <== mon.4 v2:172.21.15.88:3302/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f33bc011b90 con 0x7f33cc097a40 2023-09-30T01:38:34.958 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:34.938+0000 7f33d2cb4700 5 monclient: authenticate success, global_id 5008 2023-09-30T01:38:34.959 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:34.938+0000 7f33d2cb4700 10 monclient: _renew_subs 2023-09-30T01:38:34.959 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:34.938+0000 7f33d2cb4700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.88:3302/0 2023-09-30T01:38:34.959 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:34.938+0000 7f33d2cb4700 1 -- 172.21.15.88:0/3674368722 --> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f33cc11e370 con 0x7f33cc097a40 2023-09-30T01:38:34.959 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:34.938+0000 7f33c1ffb700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:38:34.959 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:34.938+0000 7f33c1ffb700 1 -- 172.21.15.88:0/3674368722 <== mon.4 v2:172.21.15.88:3302/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7f33bc011d00 con 0x7f33cc097a40 2023-09-30T01:38:34.959 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:34.938+0000 7f33c1ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:38:34.959 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:34.938+0000 7f33c1ffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T01:38:34.959 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:34.938+0000 7f33c1ffb700 10 monclient: dump: 2023-09-30T01:38:34.959 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:38:34.959 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:38:34.959 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:38:34.959 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:38:34.960 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:38:34.960 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:38:34.960 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:38:34.960 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:38:34.960 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:38:34.960 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:38:34.960 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:38:34.960 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:38:34.960 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:38:34.960 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:38:34.960 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:38:34.960 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:38:34.960 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:34.938+0000 7f33c3fff700 4 set_mon_vals no callback set 2023-09-30T01:38:34.961 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:34.938+0000 7f33d2cb4700 10 monclient: _renew_subs 2023-09-30T01:38:34.961 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:34.938+0000 7f33d2cb4700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.88:3302/0 2023-09-30T01:38:34.961 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:34.938+0000 7f33d2cb4700 1 -- 172.21.15.88:0/3674368722 --> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f33cc11f260 con 0x7f33cc097a40 2023-09-30T01:38:34.961 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:34.938+0000 7f33d2cb4700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:38:34.961 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:34.938+0000 7f33d2cb4700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:38:34.961 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:34.938+0000 7f33d2cb4700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.88:3302/0 2023-09-30T01:38:34.961 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:34.938+0000 7f33d2cb4700 1 -- 172.21.15.88:0/3674368722 --> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f33cc0a59b0 con 0x7f33cc097a40 2023-09-30T01:38:34.961 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:34.938+0000 7f33c1ffb700 1 -- 172.21.15.88:0/3674368722 <== mon.4 v2:172.21.15.88:3302/0 4 ==== mgrmap(e 5) v1 ==== 40144+0+0 (secure 0 0 0) 0x7f33bc011590 con 0x7f33cc097a40 2023-09-30T01:38:34.961 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:34.938+0000 7f33c1ffb700 1 --2- 172.21.15.88:0/3674368722 >> v2:172.21.15.107:6812/29926 conn(0x7f33b0033360 0x7f33b0035810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:38:34.961 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:34.938+0000 7f33c1ffb700 1 -- 172.21.15.88:0/3674368722 <== mon.4 v2:172.21.15.88:3302/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 3502+0+0 (secure 0 0 0) 0x7f33bc042380 con 0x7f33cc097a40 2023-09-30T01:38:34.961 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:34.938+0000 7f33d14b1700 1 --2- 172.21.15.88:0/3674368722 >> v2:172.21.15.107:6812/29926 conn(0x7f33b0033360 0x7f33b0035810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:38:34.961 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:34.938+0000 7f33d14b1700 10 monclient: get_auth_request con 0x7f33b0033360 auth_method 0 2023-09-30T01:38:34.962 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:34.942+0000 7f33d14b1700 1 --2- 172.21.15.88:0/3674368722 >> v2:172.21.15.107:6812/29926 conn(0x7f33b0033360 0x7f33b0035810 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f33c8009f20 tx=0x7f33c8018000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:38:34.962 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:34.946+0000 7f33c1ffb700 1 -- 172.21.15.88:0/3674368722 <== mon.4 v2:172.21.15.88:3302/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f33cc0a59b0 con 0x7f33cc097a40 2023-09-30T01:38:34.962 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:34.946+0000 7f33c1ffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:38:34.962 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:34.946+0000 7f33c1ffb700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:38:35.109 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:35.106+0000 7f33d2cb4700 1 -- 172.21.15.88:0/3674368722 --> v2:172.21.15.107:6812/29926 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f33cc071960 con 0x7f33b0033360 2023-09-30T01:38:35.114 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:35.110+0000 7f33c1ffb700 1 -- 172.21.15.88:0/3674368722 <== mgr.4108 v2:172.21.15.107:6812/29926 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+53604 (secure 0 0 0) 0x7f33cc071960 con 0x7f33b0033360 2023-09-30T01:38:35.116 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-09-30T01:38:35.117 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:35.114+0000 7f33d2cb4700 1 -- 172.21.15.88:0/3674368722 >> v2:172.21.15.107:6812/29926 conn(0x7f33b0033360 msgr2=0x7f33b0035810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:38:35.117 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:35.114+0000 7f33d2cb4700 1 --2- 172.21.15.88:0/3674368722 >> v2:172.21.15.107:6812/29926 conn(0x7f33b0033360 0x7f33b0035810 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f33c8009f20 tx=0x7f33c8018000 comp rx=0 tx=0).stop 2023-09-30T01:38:35.117 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:35.114+0000 7f33d2cb4700 10 monclient: shutdown 2023-09-30T01:38:35.117 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:35.114+0000 7f33d2cb4700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:38:35.117 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:35.114+0000 7f33d2cb4700 1 -- 172.21.15.88:0/3674368722 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f33cc097a40 msgr2=0x7f33cc0a20f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:38:35.117 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:35.114+0000 7f33d2cb4700 1 --2- 172.21.15.88:0/3674368722 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f33cc097a40 0x7f33cc0a20f0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f33bc00c790 tx=0x7f33bc007f40 comp rx=0 tx=0).stop 2023-09-30T01:38:35.117 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:35.114+0000 7f33d2cb4700 1 -- 172.21.15.88:0/3674368722 shutdown_connections 2023-09-30T01:38:35.118 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:35.114+0000 7f33d2cb4700 1 --2- 172.21.15.88:0/3674368722 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7f33cc132e10 0x7f33cc11cd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:38:35.118 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:35.114+0000 7f33d2cb4700 1 --2- 172.21.15.88:0/3674368722 >> [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] conn(0x7f33cc1311c0 0x7f33cc130120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:38:35.118 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:35.114+0000 7f33d2cb4700 1 --2- 172.21.15.88:0/3674368722 >> v2:172.21.15.107:6812/29926 conn(0x7f33b0033360 0x7f33b0035810 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:38:35.118 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:35.114+0000 7f33d2cb4700 1 --2- 172.21.15.88:0/3674368722 >> [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] conn(0x7f33cc097a40 0x7f33cc0a20f0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:38:35.118 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:35.114+0000 7f33d2cb4700 1 -- 172.21.15.88:0/3674368722 >> 172.21.15.88:0/3674368722 conn(0x7f33cc06e5e0 msgr2=0x7f33cc134cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:38:35.118 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:35.114+0000 7f33d2cb4700 1 -- 172.21.15.88:0/3674368722 shutdown_connections 2023-09-30T01:38:35.118 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:35.114+0000 7f33d2cb4700 1 -- 172.21.15.88:0/3674368722 wait complete. 2023-09-30T01:38:35.118 INFO:teuthology.orchestra.run.smithi088.stderr:dumped all 2023-09-30T01:38:35.129 INFO:teuthology.orchestra.run.smithi088.stdout:{"pg_ready":true,"pg_map":{"version":177,"stamp":"2023-09-30T01:38:33.750769+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":119,"ondisk_log_size":119,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":32704,"kb_used_data":1808,"kb_used_omap":0,"kb_used_meta":30848,"kb_avail":566198336,"statfs":{"total":579820584960,"available":579787096064,"internally_reserved":0,"allocated":1851392,"data_stored":1284542,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":31588352},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003453"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":20,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-09-30T01:38:18.238414+0000","last_change":"2023-09-30T01:38:18.238346+0000","last_active":"2023-09-30T01:38:18.238414+0000","last_peered":"2023-09-30T01:38:18.238414+0000","last_clean":"2023-09-30T01:38:18.238414+0000","last_became_active":"2023-09-30T01:33:03.291889+0000","last_became_peered":"2023-09-30T01:33:03.291889+0000","last_unstale":"2023-09-30T01:38:18.238414+0000","last_undegraded":"2023-09-30T01:38:18.238414+0000","last_fullsized":"2023-09-30T01:38:18.238414+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T01:38:18.238104+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T01:38:18.238104+0000","last_clean_scrub_stamp":"2023-09-30T01:38:18.238104+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T10:23:48.574218+0000","scrub_duration":0.0038717399999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":20,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-09-30T01:38:16.682956+0000","last_change":"2023-09-30T01:38:16.682824+0000","last_active":"2023-09-30T01:38:16.682956+0000","last_peered":"2023-09-30T01:38:16.682956+0000","last_clean":"2023-09-30T01:38:16.682956+0000","last_became_active":"2023-09-30T01:33:03.294535+0000","last_became_peered":"2023-09-30T01:33:03.294535+0000","last_unstale":"2023-09-30T01:38:16.682956+0000","last_undegraded":"2023-09-30T01:38:16.682956+0000","last_fullsized":"2023-09-30T01:38:16.682956+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T01:38:16.682591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T01:38:16.682591+0000","last_clean_scrub_stamp":"2023-09-30T01:38:16.682591+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T12:37:47.892737+0000","scrub_duration":0.0043724580000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":20,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-09-30T01:38:18.216664+0000","last_change":"2023-09-30T01:38:18.216602+0000","last_active":"2023-09-30T01:38:18.216664+0000","last_peered":"2023-09-30T01:38:18.216664+0000","last_clean":"2023-09-30T01:38:18.216664+0000","last_became_active":"2023-09-30T01:33:03.296959+0000","last_became_peered":"2023-09-30T01:33:03.296959+0000","last_unstale":"2023-09-30T01:38:18.216664+0000","last_undegraded":"2023-09-30T01:38:18.216664+0000","last_fullsized":"2023-09-30T01:38:18.216664+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T01:38:18.216368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T01:38:18.216368+0000","last_clean_scrub_stamp":"2023-09-30T01:38:18.216368+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T04:44:33.110718+0000","scrub_duration":0.0043104579999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":20,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-09-30T01:38:14.532020+0000","last_change":"2023-09-30T01:38:14.531966+0000","last_active":"2023-09-30T01:38:14.532020+0000","last_peered":"2023-09-30T01:38:14.532020+0000","last_clean":"2023-09-30T01:38:14.532020+0000","last_became_active":"2023-09-30T01:33:03.295106+0000","last_became_peered":"2023-09-30T01:33:03.295106+0000","last_unstale":"2023-09-30T01:38:14.532020+0000","last_undegraded":"2023-09-30T01:38:14.532020+0000","last_fullsized":"2023-09-30T01:38:14.532020+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T01:38:14.531806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T01:38:14.531806+0000","last_clean_scrub_stamp":"2023-09-30T01:38:14.531806+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T12:24:33.625991+0000","scrub_duration":0.003076695,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"15'119","reported_seq":172,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-09-30T01:38:19.604204+0000","last_change":"2023-09-30T01:38:19.604161+0000","last_active":"2023-09-30T01:38:19.604204+0000","last_peered":"2023-09-30T01:38:19.604204+0000","last_clean":"2023-09-30T01:38:19.604204+0000","last_became_active":"2023-09-30T01:33:03.295200+0000","last_became_peered":"2023-09-30T01:33:03.295200+0000","last_unstale":"2023-09-30T01:38:19.604204+0000","last_undegraded":"2023-09-30T01:38:19.604204+0000","last_fullsized":"2023-09-30T01:38:19.604204+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"15'119","last_scrub_stamp":"2023-09-30T01:38:19.604022+0000","last_deep_scrub":"15'119","last_deep_scrub_stamp":"2023-09-30T01:38:19.604022+0000","last_clean_scrub_stamp":"2023-09-30T01:38:19.604022+0000","objects_scrubbed":2,"log_size":119,"ondisk_log_size":119,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T02:33:45.689584+0000","scrub_duration":0.013153786000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":20,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-09-30T01:38:15.494534+0000","last_change":"2023-09-30T01:38:15.494467+0000","last_active":"2023-09-30T01:38:15.494534+0000","last_peered":"2023-09-30T01:38:15.494534+0000","last_clean":"2023-09-30T01:38:15.494534+0000","last_became_active":"2023-09-30T01:33:03.293004+0000","last_became_peered":"2023-09-30T01:33:03.293004+0000","last_unstale":"2023-09-30T01:38:15.494534+0000","last_undegraded":"2023-09-30T01:38:15.494534+0000","last_fullsized":"2023-09-30T01:38:15.494534+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T01:38:15.494227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T01:38:15.494227+0000","last_clean_scrub_stamp":"2023-09-30T01:38:15.494227+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T06:27:08.522831+0000","scrub_duration":0.0038460579999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":20,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-09-30T01:38:15.174500+0000","last_change":"2023-09-30T01:38:15.174439+0000","last_active":"2023-09-30T01:38:15.174500+0000","last_peered":"2023-09-30T01:38:15.174500+0000","last_clean":"2023-09-30T01:38:15.174500+0000","last_became_active":"2023-09-30T01:33:03.296465+0000","last_became_peered":"2023-09-30T01:33:03.296465+0000","last_unstale":"2023-09-30T01:38:15.174500+0000","last_undegraded":"2023-09-30T01:38:15.174500+0000","last_fullsized":"2023-09-30T01:38:15.174500+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T01:38:15.174188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T01:38:15.174188+0000","last_clean_scrub_stamp":"2023-09-30T01:38:15.174188+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T03:36:39.076919+0000","scrub_duration":0.0043215249999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":20,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-09-30T01:38:28.224407+0000","last_change":"2023-09-30T01:38:28.224345+0000","last_active":"2023-09-30T01:38:28.224407+0000","last_peered":"2023-09-30T01:38:28.224407+0000","last_clean":"2023-09-30T01:38:28.224407+0000","last_became_active":"2023-09-30T01:33:03.291084+0000","last_became_peered":"2023-09-30T01:33:03.291084+0000","last_unstale":"2023-09-30T01:38:28.224407+0000","last_undegraded":"2023-09-30T01:38:28.224407+0000","last_fullsized":"2023-09-30T01:38:28.224407+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T01:38:28.224092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T01:38:28.224092+0000","last_clean_scrub_stamp":"2023-09-30T01:38:28.224092+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T11:14:14.122667+0000","scrub_duration":0.0044656510000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":20,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-09-30T01:38:17.639848+0000","last_change":"2023-09-30T01:38:17.639772+0000","last_active":"2023-09-30T01:38:17.639848+0000","last_peered":"2023-09-30T01:38:17.639848+0000","last_clean":"2023-09-30T01:38:17.639848+0000","last_became_active":"2023-09-30T01:33:03.294604+0000","last_became_peered":"2023-09-30T01:33:03.294604+0000","last_unstale":"2023-09-30T01:38:17.639848+0000","last_undegraded":"2023-09-30T01:38:17.639848+0000","last_fullsized":"2023-09-30T01:38:17.639848+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T01:38:17.639493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T01:38:17.639493+0000","last_clean_scrub_stamp":"2023-09-30T01:38:17.639493+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T04:13:03.843786+0000","scrub_duration":0.0042747039999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":1057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":119,"ondisk_log_size":119,"up":2,"acting":2,"num_store_stats":2},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":6}],"osd_stats":[{"osd":5,"up_from":11,"seq":47244640324,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5128,"kb_used_data":128,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":94366712,"statfs":{"total":96636764160,"available":96631513088,"internally_reserved":0,"allocated":131072,"data_stored":37781,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 01:38:05 2023","interfaces":[{"interface":"back","average":{"1min":0.76700000000000002,"5min":0.85799999999999998,"15min":0.79400000000000004},"min":{"1min":0.46999999999999997,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":1.028,"5min":8.109,"15min":8.109},"last":0.70799999999999996},{"interface":"front","average":{"1min":0.82199999999999995,"5min":0.88600000000000001,"15min":0.85999999999999999},"min":{"1min":0.57699999999999996,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":1.0009999999999999,"5min":8.3780000000000001,"15min":8.3780000000000001},"last":0.86699999999999999}]},{"osd":1,"last update":"Sat Sep 30 01:38:05 2023","interfaces":[{"interface":"back","average":{"1min":0.751,"5min":0.78300000000000003,"15min":0.77700000000000002},"min":{"1min":0.628,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":0.96999999999999997,"5min":1.087,"15min":1.087},"last":0.83799999999999997},{"interface":"front","average":{"1min":0.82899999999999996,"5min":0.81399999999999995,"15min":0.81699999999999995},"min":{"1min":0.67100000000000004,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.97899999999999998,"5min":1.5289999999999999,"15min":1.5289999999999999},"last":0.89500000000000002}]},{"osd":2,"last update":"Sat Sep 30 01:38:05 2023","interfaces":[{"interface":"back","average":{"1min":0.83799999999999997,"5min":0.878,"15min":0.86799999999999999},"min":{"1min":0.67200000000000004,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":1.0820000000000001,"5min":4.0759999999999996,"15min":4.0759999999999996},"last":0.66700000000000004},{"interface":"front","average":{"1min":0.872,"5min":0.91500000000000004,"15min":0.86799999999999999},"min":{"1min":0.65400000000000003,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":1.0629999999999999,"5min":8.3109999999999999,"15min":8.3109999999999999},"last":0.79900000000000004}]},{"osd":3,"last update":"Sat Sep 30 01:38:05 2023","interfaces":[{"interface":"back","average":{"1min":0.751,"5min":0.751,"15min":0.78700000000000003},"min":{"1min":0.51600000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.95699999999999996,"5min":1.446,"15min":1.446},"last":0.74099999999999999},{"interface":"front","average":{"1min":0.98899999999999999,"5min":0.94299999999999995,"15min":0.95899999999999996},"min":{"1min":0.84299999999999997,"5min":0.53900000000000003,"15min":0.53900000000000003},"max":{"1min":1.1299999999999999,"5min":1.4790000000000001,"15min":1.4790000000000001},"last":0.98599999999999999}]},{"osd":4,"last update":"Sat Sep 30 01:38:05 2023","interfaces":[{"interface":"back","average":{"1min":0.85299999999999998,"5min":0.88400000000000001,"15min":0.84199999999999997},"min":{"1min":0.626,"5min":0.441,"15min":0.441},"max":{"1min":1.3109999999999999,"5min":8.2289999999999992,"15min":8.2289999999999992},"last":0.95199999999999996},{"interface":"front","average":{"1min":0.85199999999999998,"5min":0.82199999999999995,"15min":0.83399999999999996},"min":{"1min":0.58999999999999997,"5min":0.52100000000000002,"15min":0.52100000000000002},"max":{"1min":1,"5min":1.2310000000000001,"15min":1.2310000000000001},"last":0.92400000000000004}]}]},{"osd":4,"up_from":11,"seq":47244640324,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5128,"kb_used_data":128,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":94366712,"statfs":{"total":96636764160,"available":96631513088,"internally_reserved":0,"allocated":131072,"data_stored":37781,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 01:38:11 2023","interfaces":[{"interface":"back","average":{"1min":0.77500000000000002,"5min":0.77400000000000002,"15min":0.76900000000000002},"min":{"1min":0.54600000000000004,"5min":0.52500000000000002,"15min":0.52500000000000002},"max":{"1min":0.95799999999999996,"5min":1.0169999999999999,"15min":1.0169999999999999},"last":0.85899999999999999},{"interface":"front","average":{"1min":0.79900000000000004,"5min":0.79900000000000004,"15min":0.80000000000000004},"min":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":0.93999999999999995,"5min":1.6439999999999999,"15min":1.6439999999999999},"last":0.73299999999999998}]},{"osd":1,"last update":"Sat Sep 30 01:38:11 2023","interfaces":[{"interface":"back","average":{"1min":0.83299999999999996,"5min":0.79800000000000004,"15min":0.78400000000000003},"min":{"1min":0.58299999999999996,"5min":0.51100000000000001,"15min":0.51100000000000001},"max":{"1min":1.0609999999999999,"5min":1.0609999999999999,"15min":1.0609999999999999},"last":0.75800000000000001},{"interface":"front","average":{"1min":0.77700000000000002,"5min":0.79300000000000004,"15min":0.78500000000000003},"min":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"max":{"1min":1.091,"5min":1.548,"15min":1.548},"last":0.82099999999999995}]},{"osd":2,"last update":"Sat Sep 30 01:38:11 2023","interfaces":[{"interface":"back","average":{"1min":0.78900000000000003,"5min":0.83199999999999996,"15min":0.85999999999999999},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":1.1419999999999999,"5min":1.1950000000000001,"15min":1.1950000000000001},"last":0.78700000000000003},{"interface":"front","average":{"1min":0.96299999999999997,"5min":0.92400000000000004,"15min":0.93600000000000005},"min":{"1min":0.77500000000000002,"5min":0.64800000000000002,"15min":0.64800000000000002},"max":{"1min":1.3260000000000001,"5min":1.3260000000000001,"15min":1.3260000000000001},"last":0.92300000000000004}]},{"osd":3,"last update":"Sat Sep 30 01:38:11 2023","interfaces":[{"interface":"back","average":{"1min":0.66800000000000004,"5min":0.71699999999999997,"15min":0.73299999999999998},"min":{"1min":0.496,"5min":0.496,"15min":0.496},"max":{"1min":0.878,"5min":1.5,"15min":1.5},"last":0.89500000000000002},{"interface":"front","average":{"1min":0.746,"5min":0.78500000000000003,"15min":0.81100000000000005},"min":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"max":{"1min":0.96499999999999997,"5min":1.405,"15min":1.405},"last":0.64600000000000002}]},{"osd":5,"last update":"Sat Sep 30 01:38:11 2023","interfaces":[{"interface":"back","average":{"1min":0.72699999999999998,"5min":0.78100000000000003,"15min":0.77600000000000002},"min":{"1min":0.59899999999999998,"5min":0.54800000000000004,"15min":0.54800000000000004},"max":{"1min":0.97199999999999998,"5min":1.7270000000000001,"15min":1.7270000000000001},"last":0.58699999999999997},{"interface":"front","average":{"1min":0.83999999999999997,"5min":0.88900000000000001,"15min":0.90100000000000002},"min":{"1min":0.61299999999999999,"5min":0.61299999999999999,"15min":0.61299999999999999},"max":{"1min":0.94599999999999995,"5min":1.0960000000000001,"15min":1.0960000000000001},"last":0.68600000000000005}]}]},{"osd":3,"up_from":11,"seq":47244640325,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6096,"kb_used_data":648,"kb_used_omap":0,"kb_used_meta":5440,"kb_avail":94365744,"statfs":{"total":96636764160,"available":96630521856,"internally_reserved":0,"allocated":663552,"data_stored":566709,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5570560},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 01:38:05 2023","interfaces":[{"interface":"back","average":{"1min":0.81200000000000006,"5min":0.79200000000000004,"15min":0.77600000000000002},"min":{"1min":0.60899999999999999,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":0.95299999999999996,"5min":1.052,"15min":1.052},"last":0.95399999999999996},{"interface":"front","average":{"1min":0.77400000000000002,"5min":0.75900000000000001,"15min":0.73799999999999999},"min":{"1min":0.59799999999999998,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":1.018,"5min":1.206,"15min":1.206},"last":0.79700000000000004}]},{"osd":1,"last update":"Sat Sep 30 01:38:05 2023","interfaces":[{"interface":"back","average":{"1min":0.84899999999999998,"5min":0.83499999999999996,"15min":0.83099999999999996},"min":{"1min":0.70899999999999996,"5min":0.56599999999999995,"15min":0.56599999999999995},"max":{"1min":1.083,"5min":1.159,"15min":1.159},"last":0.82299999999999995},{"interface":"front","average":{"1min":0.85699999999999998,"5min":0.79700000000000004,"15min":0.76800000000000002},"min":{"1min":0.60099999999999998,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":1.1140000000000001,"5min":1.137,"15min":1.137},"last":0.76200000000000001}]},{"osd":2,"last update":"Sat Sep 30 01:38:05 2023","interfaces":[{"interface":"back","average":{"1min":0.91700000000000004,"5min":0.89900000000000002,"15min":0.93700000000000006},"min":{"1min":0.64500000000000002,"5min":0.60599999999999998,"15min":0.60599999999999998},"max":{"1min":1.0860000000000001,"5min":1.296,"15min":1.296},"last":0.85899999999999999},{"interface":"front","average":{"1min":0.84699999999999998,"5min":0.88900000000000001,"15min":0.91900000000000004},"min":{"1min":0.65900000000000003,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":1.1559999999999999,"5min":1.46,"15min":1.46},"last":0.89100000000000001}]},{"osd":4,"last update":"Sat Sep 30 01:38:05 2023","interfaces":[{"interface":"back","average":{"1min":0.71199999999999997,"5min":0.73299999999999998,"15min":0.73299999999999998},"min":{"1min":0.55200000000000005,"5min":0.51300000000000001,"15min":0.51300000000000001},"max":{"1min":0.95599999999999996,"5min":1.3220000000000001,"15min":1.3220000000000001},"last":0.67600000000000005},{"interface":"front","average":{"1min":0.82599999999999996,"5min":0.82899999999999996,"15min":0.872},"min":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":1.179,"5min":1.179,"15min":1.179},"last":0.58299999999999996}]},{"osd":5,"last update":"Sat Sep 30 01:38:05 2023","interfaces":[{"interface":"back","average":{"1min":0.82999999999999996,"5min":0.81399999999999995,"15min":0.82799999999999996},"min":{"1min":0.64400000000000002,"5min":0.61099999999999999,"15min":0.61099999999999999},"max":{"1min":1.1439999999999999,"5min":1.3320000000000001,"15min":1.3320000000000001},"last":0.92200000000000004},{"interface":"front","average":{"1min":0.78100000000000003,"5min":0.82599999999999996,"15min":0.86399999999999999},"min":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"max":{"1min":1.054,"5min":1.387,"15min":1.387},"last":0.71899999999999997}]}]},{"osd":2,"up_from":11,"seq":47244640326,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5128,"kb_used_data":128,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":94366712,"statfs":{"total":96636764160,"available":96631513088,"internally_reserved":0,"allocated":131072,"data_stored":37781,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 01:38:07 2023","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.57999999999999996,"15min":0.56599999999999995},"min":{"1min":0.46999999999999997,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.79100000000000004,"5min":0.95099999999999996,"15min":0.95099999999999996},"last":0.57399999999999995},{"interface":"front","average":{"1min":0.66800000000000004,"5min":0.68000000000000005,"15min":0.65800000000000003},"min":{"1min":0.53600000000000003,"5min":0.439,"15min":0.439},"max":{"1min":0.96599999999999997,"5min":2.2029999999999998,"15min":2.2029999999999998},"last":0.66300000000000003}]},{"osd":1,"last update":"Sat Sep 30 01:38:07 2023","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.68400000000000005,"15min":0.70099999999999996},"min":{"1min":0.437,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.85599999999999998,"5min":2.2490000000000001,"15min":2.2490000000000001},"last":0.61099999999999999},{"interface":"front","average":{"1min":0.68799999999999994,"5min":0.65200000000000002,"15min":0.65700000000000003},"min":{"1min":0.46500000000000002,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.80700000000000005,"5min":1.0409999999999999,"15min":1.0409999999999999},"last":0.79800000000000004}]},{"osd":3,"last update":"Sat Sep 30 01:38:07 2023","interfaces":[{"interface":"back","average":{"1min":0.79600000000000004,"5min":0.82099999999999995,"15min":0.83299999999999996},"min":{"1min":0.59399999999999997,"5min":0.59399999999999997,"15min":0.59399999999999997},"max":{"1min":1.0309999999999999,"5min":1.343,"15min":1.343},"last":0.71299999999999997},{"interface":"front","average":{"1min":0.875,"5min":0.91200000000000003,"15min":0.92200000000000004},"min":{"1min":0.63,"5min":0.60499999999999998,"15min":0.60499999999999998},"max":{"1min":1.083,"5min":2.4390000000000001,"15min":2.4390000000000001},"last":0.86499999999999999}]},{"osd":4,"last update":"Sat Sep 30 01:38:07 2023","interfaces":[{"interface":"back","average":{"1min":0.82899999999999996,"5min":0.88300000000000001,"15min":0.90300000000000002},"min":{"1min":0.67400000000000004,"5min":0.64300000000000002,"15min":0.64300000000000002},"max":{"1min":1.0349999999999999,"5min":2.3679999999999999,"15min":2.3679999999999999},"last":0.77100000000000002},{"interface":"front","average":{"1min":0.878,"5min":0.85399999999999998,"15min":0.84999999999999998},"min":{"1min":0.67300000000000004,"5min":0.505,"15min":0.505},"max":{"1min":1.0169999999999999,"5min":1.401,"15min":1.401},"last":0.91300000000000003}]},{"osd":5,"last update":"Sat Sep 30 01:38:07 2023","interfaces":[{"interface":"back","average":{"1min":0.88100000000000001,"5min":0.86499999999999999,"15min":0.86699999999999999},"min":{"1min":0.70099999999999996,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":1.085,"5min":1.3959999999999999,"15min":1.3959999999999999},"last":0.82999999999999996},{"interface":"front","average":{"1min":0.88500000000000001,"5min":0.90600000000000003,"15min":0.92400000000000004},"min":{"1min":0.65900000000000003,"5min":0.61799999999999999,"15min":0.61799999999999999},"max":{"1min":1.091,"5min":1.625,"15min":1.625},"last":0.88800000000000001}]}]},{"osd":1,"up_from":11,"seq":47244640325,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6096,"kb_used_data":648,"kb_used_omap":0,"kb_used_meta":5440,"kb_avail":94365744,"statfs":{"total":96636764160,"available":96630521856,"internally_reserved":0,"allocated":663552,"data_stored":566709,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5570560},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 01:38:08 2023","interfaces":[{"interface":"back","average":{"1min":0.59199999999999997,"5min":0.57699999999999996,"15min":0.57299999999999995},"min":{"1min":0.38400000000000001,"5min":0.317,"15min":0.317},"max":{"1min":0.90500000000000003,"5min":0.92400000000000004,"15min":0.92400000000000004},"last":0.75600000000000001},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.64500000000000002,"15min":0.64900000000000002},"min":{"1min":0.50900000000000001,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.72699999999999998,"5min":2.085,"15min":2.085},"last":0.66500000000000004}]},{"osd":2,"last update":"Sat Sep 30 01:38:08 2023","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.625,"15min":0.66300000000000003},"min":{"1min":0.34399999999999997,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.80200000000000005,"5min":1.159,"15min":1.159},"last":0.54300000000000004},{"interface":"front","average":{"1min":0.64800000000000002,"5min":0.68799999999999994,"15min":0.69699999999999995},"min":{"1min":0.40500000000000003,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.79500000000000004,"5min":1.9930000000000001,"15min":1.9930000000000001},"last":0.61299999999999999}]},{"osd":3,"last update":"Sat Sep 30 01:38:08 2023","interfaces":[{"interface":"back","average":{"1min":0.81299999999999994,"5min":0.81999999999999995,"15min":0.82399999999999995},"min":{"1min":0.65000000000000002,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":1.018,"5min":1.1499999999999999,"15min":1.1499999999999999},"last":0.82599999999999996},{"interface":"front","average":{"1min":0.85299999999999998,"5min":0.83499999999999996,"15min":0.84099999999999997},"min":{"1min":0.66800000000000004,"5min":0.61699999999999999,"15min":0.61699999999999999},"max":{"1min":1.0960000000000001,"5min":1.2929999999999999,"15min":1.2929999999999999},"last":0.84199999999999997}]},{"osd":4,"last update":"Sat Sep 30 01:38:08 2023","interfaces":[{"interface":"back","average":{"1min":0.84899999999999998,"5min":0.85499999999999998,"15min":0.878},"min":{"1min":0.64300000000000002,"5min":0.53600000000000003,"15min":0.53600000000000003},"max":{"1min":0.98999999999999999,"5min":1.415,"15min":1.415},"last":0.88200000000000001},{"interface":"front","average":{"1min":0.874,"5min":0.872,"15min":0.88600000000000001},"min":{"1min":0.70499999999999996,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":0.99199999999999999,"5min":1.381,"15min":1.381},"last":0.95599999999999996}]},{"osd":5,"last update":"Sat Sep 30 01:38:08 2023","interfaces":[{"interface":"back","average":{"1min":0.85599999999999998,"5min":0.89600000000000002,"15min":0.90200000000000002},"min":{"1min":0.625,"5min":0.625,"15min":0.625},"max":{"1min":1.03,"5min":1.395,"15min":1.395},"last":1.0249999999999999},{"interface":"front","average":{"1min":0.96199999999999997,"5min":0.95699999999999996,"15min":0.96299999999999997},"min":{"1min":0.67500000000000004,"5min":0.56299999999999994,"15min":0.56299999999999994},"max":{"1min":1.1379999999999999,"5min":1.4179999999999999,"15min":1.4179999999999999},"last":0.92800000000000005}]}]},{"osd":0,"up_from":11,"seq":47244640326,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5128,"kb_used_data":128,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":94366712,"statfs":{"total":96636764160,"available":96631513088,"internally_reserved":0,"allocated":131072,"data_stored":37781,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Sep 30 01:38:07 2023","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.57799999999999996,"15min":0.59099999999999997},"min":{"1min":0.433,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.75,"5min":0.92900000000000005,"15min":0.92900000000000005},"last":0.66000000000000003},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.63700000000000001,"15min":0.63900000000000001},"min":{"1min":0.45400000000000001,"5min":0.314,"15min":0.314},"max":{"1min":0.81999999999999995,"5min":0.98599999999999999,"15min":0.98599999999999999},"last":0.47699999999999998}]},{"osd":2,"last update":"Sat Sep 30 01:38:07 2023","interfaces":[{"interface":"back","average":{"1min":0.60599999999999998,"5min":0.627,"15min":0.61899999999999999},"min":{"1min":0.41199999999999998,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.86299999999999999,"5min":0.93799999999999994,"15min":0.93799999999999994},"last":0.76900000000000002},{"interface":"front","average":{"1min":0.66300000000000003,"5min":0.71399999999999997,"15min":0.70599999999999996},"min":{"1min":0.45100000000000001,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":0.83699999999999997,"5min":2.069,"15min":2.069},"last":0.53300000000000003}]},{"osd":3,"last update":"Sat Sep 30 01:38:07 2023","interfaces":[{"interface":"back","average":{"1min":0.83199999999999996,"5min":0.83699999999999997,"15min":0.85399999999999998},"min":{"1min":0.67700000000000005,"5min":0.60299999999999998,"15min":0.60299999999999998},"max":{"1min":1.081,"5min":1.1319999999999999,"15min":1.1319999999999999},"last":0.71599999999999997},{"interface":"front","average":{"1min":0.82699999999999996,"5min":0.872,"15min":0.876},"min":{"1min":0.66200000000000003,"5min":0.62,"15min":0.62},"max":{"1min":1.024,"5min":2.4870000000000001,"15min":2.4870000000000001},"last":0.91700000000000004}]},{"osd":4,"last update":"Sat Sep 30 01:38:07 2023","interfaces":[{"interface":"back","average":{"1min":0.83899999999999997,"5min":0.90600000000000003,"15min":0.92000000000000004},"min":{"1min":0.69299999999999995,"5min":0.66300000000000003,"15min":0.66300000000000003},"max":{"1min":1.0720000000000001,"5min":2.589,"15min":2.589},"last":0.82099999999999995},{"interface":"front","average":{"1min":0.82399999999999995,"5min":0.85799999999999998,"15min":0.90200000000000002},"min":{"1min":0.67900000000000005,"5min":0.54900000000000004,"15min":0.54900000000000004},"max":{"1min":1.0189999999999999,"5min":1.413,"15min":1.413},"last":0.84999999999999998}]},{"osd":5,"last update":"Sat Sep 30 01:38:07 2023","interfaces":[{"interface":"back","average":{"1min":0.85999999999999999,"5min":0.89300000000000002,"15min":0.92400000000000004},"min":{"1min":0.68300000000000005,"5min":0.58799999999999997,"15min":0.58799999999999997},"max":{"1min":1.1220000000000001,"5min":1.3200000000000001,"15min":1.3200000000000001},"last":0.79600000000000004},{"interface":"front","average":{"1min":0.88900000000000001,"5min":0.94799999999999995,"15min":0.96399999999999997},"min":{"1min":0.749,"5min":0.73999999999999999,"15min":0.73999999999999999},"max":{"1min":1.0609999999999999,"5min":2.4279999999999999,"15min":2.4279999999999999},"last":0.879}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-09-30T01:38:35.131 DEBUG:teuthology.orchestra.run.smithi088:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2023-09-30T01:38:35.234 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:35.230+0000 7fcefee78700 10 monclient: get_monmap_and_config 2023-09-30T01:38:35.235 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:35.230+0000 7fcefee78700 10 monclient: build_initial_monmap 2023-09-30T01:38:35.235 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:35.230+0000 7fcefee78700 10 monclient: monmap: 2023-09-30T01:38:35.235 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2023-09-30T01:38:35.235 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:38:35.235 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:38:35.234250+0000 2023-09-30T01:38:35.235 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:38:35.234250+0000 2023-09-30T01:38:35.235 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2023-09-30T01:38:35.235 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:38:35.235 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a-legacy 2023-09-30T01:38:35.235 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.107:6789/0 mon.noname-f-legacy 2023-09-30T01:38:35.235 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.noname-b 2023-09-30T01:38:35.236 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.noname-g 2023-09-30T01:38:35.236 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.noname-c 2023-09-30T01:38:35.236 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.noname-h 2023-09-30T01:38:35.236 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.noname-d 2023-09-30T01:38:35.236 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.noname-i 2023-09-30T01:38:35.236 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.noname-e 2023-09-30T01:38:35.236 INFO:teuthology.orchestra.run.smithi088.stderr:9: v2:172.21.15.88:3300/0 mon.noname-a 2023-09-30T01:38:35.236 INFO:teuthology.orchestra.run.smithi088.stderr:10: v2:172.21.15.107:3300/0 mon.noname-f 2023-09-30T01:38:35.236 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:38:35.236 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:35.230+0000 7fcefee78700 1 Processor -- start 2023-09-30T01:38:35.236 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:35.230+0000 7fcefee78700 1 -- start start 2023-09-30T01:38:35.236 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:35.230+0000 7fcefee78700 10 monclient: init 2023-09-30T01:38:35.243 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:35.230+0000 7fcefee78700 10 monclient: _reopen_session rank -1 2023-09-30T01:38:35.243 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:35.230+0000 7fcefee78700 10 monclient: _add_conns ranks=[2,0,1,3,4,8,7,5,9,10,6] 2023-09-30T01:38:35.243 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:35.230+0000 7fcefee78700 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fcef80a3380 0x7fcef80a48b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:38:35.243 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:35.230+0000 7fcefee78700 10 monclient(hunting): picked mon.noname-b con 0x7fcef80a3380 addr [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] 2023-09-30T01:38:35.244 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:35.230+0000 7fcefee78700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fcef809ffd0 addr v1:172.21.15.88:6789/0 2023-09-30T01:38:35.244 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:35.230+0000 7fcefee78700 10 monclient(hunting): picked mon.noname-f-legacy con 0x7fcef80a03a0 addr v1:172.21.15.107:6789/0 2023-09-30T01:38:35.244 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:35.230+0000 7fcefee78700 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcef804bb20 con 0x7fcef809ffd0 2023-09-30T01:38:35.244 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:35.230+0000 7fcefee78700 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcef8098480 con 0x7fcef80a03a0 2023-09-30T01:38:35.244 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:35.230+0000 7fcefee78700 10 monclient(hunting): start opening mon connection 2023-09-30T01:38:35.244 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:35.230+0000 7fcefee78700 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcef8097ab0 con 0x7fcef80a3380 2023-09-30T01:38:35.244 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:35.230+0000 7fcefee78700 10 monclient(hunting): _renew_subs 2023-09-30T01:38:35.244 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:35.230+0000 7fcefee78700 10 monclient(hunting): authenticate will time out at 1187.271835s 2023-09-30T01:38:35.244 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:35.234+0000 7fcefde76700 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fcef80a3380 0x7fcef80a48b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:38:35.244 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:35.234+0000 7fcefd675700 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7fcef809ffd0 0x7fcef81333b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:36434/0 (socket says 172.21.15.88:36434) 2023-09-30T01:38:35.244 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:35.234+0000 7fcefd675700 1 -- 172.21.15.88:0/2763226664 learned_addr learned my addr 172.21.15.88:0/2763226664 (peer_addr_for_me v1:172.21.15.88:0/0) 2023-09-30T01:38:35.244 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:35.234+0000 7fcefde76700 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fcef80a3380 0x7fcef80a48b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3301/0 says I am v2:172.21.15.88:44426/0 (socket says 172.21.15.88:44426) 2023-09-30T01:38:35.245 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:35.234+0000 7fcefde76700 10 monclient(hunting): get_auth_request con 0x7fcef80a3380 auth_method 0 2023-09-30T01:38:35.245 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:35.234+0000 7fcefde76700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:38:35.245 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:35.234+0000 7fcefde76700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:38:35.245 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:35.234+0000 7fcefde76700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:38:35.245 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:35.234+0000 7fcefde76700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:38:35.245 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:35.234+0000 7fcefde76700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:38:35.245 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:35.234+0000 7fcefde76700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:38:35.245 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:35.234+0000 7fcefde76700 10 monclient(hunting): handle_auth_done global_id 4979 payload 274 2023-09-30T01:38:35.245 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:35.234+0000 7fcefde76700 1 -- 172.21.15.88:0/2763226664 >> v1:172.21.15.107:6789/0 conn(0x7fcef80a03a0 legacy=0x7fcef8133ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:38:35.245 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:35.234+0000 7fcefde76700 1 -- 172.21.15.88:0/2763226664 >> v1:172.21.15.88:6789/0 conn(0x7fcef809ffd0 legacy=0x7fcef81333b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:38:35.245 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:35.234+0000 7fcefde76700 10 monclient: _finish_hunting 0 2023-09-30T01:38:35.245 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:35.234+0000 7fcefde76700 1 monclient: found mon.noname-b 2023-09-30T01:38:35.246 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:35.234+0000 7fcefde76700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:38:35.246 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:35.234+0000 7fcefde76700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.88:3301/0 2023-09-30T01:38:35.246 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:35.234+0000 7fcefde76700 1 -- 172.21.15.88:0/2763226664 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcef80a3750 con 0x7fcef80a3380 2023-09-30T01:38:35.246 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:35.234+0000 7fcefde76700 1 --2- 172.21.15.88:0/2763226664 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fcef80a3380 0x7fcef80a48b0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fcee8009e90 tx=0x7fcee8009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=216abf3e48b8a77f server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:38:35.246 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:35.234+0000 7fcefce74700 1 -- 172.21.15.88:0/2763226664 <== mon.2 v2:172.21.15.88:3301/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7fcee800e070 con 0x7fcef80a3380 2023-09-30T01:38:35.246 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:35.234+0000 7fcefce74700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:38:35.246 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:35.234+0000 7fcefce74700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T01:38:35.246 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:35.234+0000 7fcefce74700 10 monclient: dump: 2023-09-30T01:38:35.246 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:38:35.246 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:38:35.246 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:38:35.247 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:38:35.247 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:38:35.247 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:38:35.247 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:38:35.247 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:38:35.247 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:38:35.247 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:38:35.247 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:38:35.247 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:38:35.247 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:38:35.247 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:38:35.247 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:38:35.248 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:38:35.248 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:35.234+0000 7fcefce74700 10 monclient: _finish_auth 0 2023-09-30T01:38:35.248 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:35.234+0000 7fcefce74700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:38:35.248 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:35.234+0000 7fcefee78700 5 monclient: authenticate success, global_id 4979 2023-09-30T01:38:35.248 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:35.234+0000 7fcefee78700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:38:35.248 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:35.234+0000 7fcefce74700 1 -- 172.21.15.88:0/2763226664 <== mon.2 v2:172.21.15.88:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcee800bc30 con 0x7fcef80a3380 2023-09-30T01:38:35.248 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:35.234+0000 7fcefce74700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:38:35.248 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:35.234+0000 7fcefee78700 10 monclient: get_monmap_and_config success 2023-09-30T01:38:35.248 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:35.234+0000 7fcefee78700 4 set_mon_vals no callback set 2023-09-30T01:38:35.248 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:35.234+0000 7fcefce74700 1 -- 172.21.15.88:0/2763226664 <== mon.2 v2:172.21.15.88:3301/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7fcee800bda0 con 0x7fcef80a3380 2023-09-30T01:38:35.248 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:35.234+0000 7fcefce74700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:38:35.248 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:35.234+0000 7fcefce74700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T01:38:35.249 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:35.234+0000 7fcefce74700 10 monclient: dump: 2023-09-30T01:38:35.249 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:38:35.249 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:38:35.249 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:38:35.249 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:38:35.249 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:38:35.249 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:38:35.249 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:38:35.249 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:38:35.249 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:38:35.249 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:38:35.249 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:38:35.250 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:38:35.250 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:38:35.250 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:38:35.250 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:38:35.250 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:38:35.250 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:35.234+0000 7fcefee78700 10 monclient: shutdown 2023-09-30T01:38:35.250 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:35.234+0000 7fcefee78700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:38:35.250 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:35.234+0000 7fcefee78700 1 -- 172.21.15.88:0/2763226664 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fcef80a3380 msgr2=0x7fcef80a48b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:38:35.250 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:35.234+0000 7fcefee78700 1 --2- 172.21.15.88:0/2763226664 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fcef80a3380 0x7fcef80a48b0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fcee8009e90 tx=0x7fcee8009780 comp rx=0 tx=0).stop 2023-09-30T01:38:35.250 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:35.234+0000 7fcefee78700 1 -- 172.21.15.88:0/2763226664 shutdown_connections 2023-09-30T01:38:35.250 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:35.234+0000 7fcefee78700 1 --2- 172.21.15.88:0/2763226664 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fcef80a3380 0x7fcef80a48b0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:38:35.251 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:35.234+0000 7fcefee78700 1 -- 172.21.15.88:0/2763226664 >> 172.21.15.88:0/2763226664 conn(0x7fcef806e5e0 msgr2=0x7fcef806e9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:38:35.251 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:35.234+0000 7fcefee78700 1 -- 172.21.15.88:0/2763226664 shutdown_connections 2023-09-30T01:38:35.251 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:35.234+0000 7fcefee78700 1 -- 172.21.15.88:0/2763226664 wait complete. 2023-09-30T01:38:35.251 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:35.234+0000 7fcefee78700 10 monclient: build_initial_monmap 2023-09-30T01:38:35.251 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:35.234+0000 7fcefee78700 10 monclient: monmap: 2023-09-30T01:38:35.251 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2023-09-30T01:38:35.251 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:38:35.251 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 0.000000 2023-09-30T01:38:35.251 INFO:teuthology.orchestra.run.smithi088.stderr:created 0.000000 2023-09-30T01:38:35.251 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2023-09-30T01:38:35.251 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 1 2023-09-30T01:38:35.251 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.noname-a 2023-09-30T01:38:35.252 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.noname-f 2023-09-30T01:38:35.252 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.noname-b 2023-09-30T01:38:35.252 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.noname-g 2023-09-30T01:38:35.252 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.noname-c 2023-09-30T01:38:35.252 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.noname-h 2023-09-30T01:38:35.252 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.noname-d 2023-09-30T01:38:35.252 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.noname-i 2023-09-30T01:38:35.252 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.noname-e 2023-09-30T01:38:35.252 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:38:35.252 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:35.234+0000 7fcefee78700 1 Processor -- start 2023-09-30T01:38:35.252 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:35.234+0000 7fcefee78700 1 -- start start 2023-09-30T01:38:35.252 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:35.234+0000 7fcefee78700 10 monclient: init 2023-09-30T01:38:35.253 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:35.238+0000 7fcefee78700 10 monclient: _reopen_session rank -1 2023-09-30T01:38:35.253 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:35.238+0000 7fcefee78700 10 monclient: _add_conns ranks=[2,5,6,7,0,3,1,8,4] 2023-09-30T01:38:35.253 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:35.238+0000 7fcefee78700 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fcef809ffd0 0x7fcef8131180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:38:35.253 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:35.238+0000 7fcefee78700 10 monclient(hunting): picked mon.noname-b con 0x7fcef809ffd0 addr [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] 2023-09-30T01:38:35.253 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:35.238+0000 7fcefee78700 1 --2- >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7fcef80a03a0 0x7fcef811cf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:38:35.253 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:35.238+0000 7fcefee78700 10 monclient(hunting): picked mon.noname-h con 0x7fcef80a03a0 addr [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] 2023-09-30T01:38:35.253 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:35.238+0000 7fcefee78700 1 --2- >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7fcef80a3380 0x7fcef811d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:38:35.253 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:35.238+0000 7fcefee78700 10 monclient(hunting): picked mon.noname-d con 0x7fcef80a3380 addr [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] 2023-09-30T01:38:35.253 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:35.238+0000 7fcefee78700 10 monclient(hunting): start opening mon connection 2023-09-30T01:38:35.253 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:35.238+0000 7fcefee78700 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcef81359b0 con 0x7fcef809ffd0 2023-09-30T01:38:35.253 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:35.238+0000 7fcefee78700 10 monclient(hunting): start opening mon connection 2023-09-30T01:38:35.253 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:35.238+0000 7fcefee78700 1 -- --> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] -- mon_getmap magic: 0 v1 -- 0x7fcef80a0ac0 con 0x7fcef80a03a0 2023-09-30T01:38:35.254 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:35.238+0000 7fcefee78700 10 monclient(hunting): start opening mon connection 2023-09-30T01:38:35.254 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:35.238+0000 7fcefee78700 1 -- --> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] -- mon_getmap magic: 0 v1 -- 0x7fcef8097ce0 con 0x7fcef80a3380 2023-09-30T01:38:35.254 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:35.238+0000 7fcefee78700 10 monclient(hunting): _renew_subs 2023-09-30T01:38:35.254 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:35.238+0000 7fcefee78700 10 monclient(hunting): authenticate will time out at 1187.276134s 2023-09-30T01:38:35.254 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:35.238+0000 7fcefde76700 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fcef809ffd0 0x7fcef8131180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:38:35.254 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:35.238+0000 7fcefde76700 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fcef809ffd0 0x7fcef8131180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3301/0 says I am v2:172.21.15.88:44434/0 (socket says 172.21.15.88:44434) 2023-09-30T01:38:35.254 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:35.238+0000 7fcefe677700 1 --2- >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7fcef80a3380 0x7fcef811d4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:38:35.254 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:35.238+0000 7fcefde76700 1 -- 172.21.15.88:0/1036828418 learned_addr learned my addr 172.21.15.88:0/1036828418 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-09-30T01:38:35.254 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:35.238+0000 7fcefde76700 10 monclient(hunting): get_auth_request con 0x7fcef809ffd0 auth_method 0 2023-09-30T01:38:35.254 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:35.238+0000 7fcefde76700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:38:35.254 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:35.238+0000 7fcefde76700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:38:35.254 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:35.238+0000 7fcefde76700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:38:35.255 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:35.238+0000 7fcefe677700 10 monclient(hunting): get_auth_request con 0x7fcef80a3380 auth_method 0 2023-09-30T01:38:35.255 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:35.238+0000 7fcefe677700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:38:35.255 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:35.238+0000 7fcefe677700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:38:35.255 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:35.238+0000 7fcefe677700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:38:35.255 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:35.238+0000 7fcefde76700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:38:35.255 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:35.238+0000 7fcefde76700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:38:35.255 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:35.238+0000 7fcefde76700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:38:35.255 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:35.238+0000 7fcefe677700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:38:35.255 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:35.238+0000 7fcefe677700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:38:35.255 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:35.238+0000 7fcefd675700 1 --2- 172.21.15.88:0/1036828418 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7fcef80a03a0 0x7fcef811cf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:38:35.255 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:35.238+0000 7fcefe677700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:38:35.256 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:35.238+0000 7fcefd675700 10 monclient(hunting): get_auth_request con 0x7fcef80a03a0 auth_method 0 2023-09-30T01:38:35.256 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:35.238+0000 7fcefd675700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:38:35.256 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:35.238+0000 7fcefd675700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:38:35.256 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:35.238+0000 7fcefd675700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:38:35.256 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:35.238+0000 7fcefd675700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:38:35.256 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:35.238+0000 7fcefd675700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:38:35.256 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:35.238+0000 7fcefd675700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:38:35.256 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:35.238+0000 7fcefe677700 10 monclient(hunting): handle_auth_done global_id 4884 payload 824 2023-09-30T01:38:35.256 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:35.238+0000 7fcefe677700 1 -- 172.21.15.88:0/1036828418 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7fcef80a03a0 msgr2=0x7fcef811cf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:38:35.256 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:35.238+0000 7fcefe677700 1 --2- 172.21.15.88:0/1036828418 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7fcef80a03a0 0x7fcef811cf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:38:35.257 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:35.238+0000 7fcefe677700 1 -- 172.21.15.88:0/1036828418 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fcef809ffd0 msgr2=0x7fcef8131180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:38:35.257 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:35.238+0000 7fcefe677700 1 --2- 172.21.15.88:0/1036828418 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fcef809ffd0 0x7fcef8131180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:38:35.257 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:35.238+0000 7fcefe677700 10 monclient: _finish_hunting 0 2023-09-30T01:38:35.257 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:35.238+0000 7fcefe677700 1 monclient: found mon.noname-d 2023-09-30T01:38:35.257 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:35.238+0000 7fcefe677700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:38:35.257 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:35.238+0000 7fcefe677700 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.88:3303/0 2023-09-30T01:38:35.257 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:35.238+0000 7fcefe677700 1 -- 172.21.15.88:0/1036828418 --> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcee8009a20 con 0x7fcef80a3380 2023-09-30T01:38:35.257 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:35.238+0000 7fcefe677700 1 --2- 172.21.15.88:0/1036828418 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7fcef80a3380 0x7fcef811d4a0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fcef000c4b0 tx=0x7fcef0007e60 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:38:35.257 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:35.238+0000 7fcef5ffb700 1 -- 172.21.15.88:0/1036828418 <== mon.6 v2:172.21.15.88:3303/0 1 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7fcef000ce60 con 0x7fcef80a3380 2023-09-30T01:38:35.257 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:35.238+0000 7fcef5ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:38:35.257 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:35.238+0000 7fcef5ffb700 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2023-09-30T01:38:35.258 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:35.238+0000 7fcef5ffb700 10 monclient: dump: 2023-09-30T01:38:35.258 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:38:35.258 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:38:35.258 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:38:35.258 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:38:35.258 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:38:35.258 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:38:35.258 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:38:35.258 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:38:35.258 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:38:35.258 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:38:35.258 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:38:35.259 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:38:35.259 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:38:35.259 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:38:35.259 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:38:35.259 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:38:35.259 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:35.238+0000 7fcef5ffb700 10 monclient: _finish_auth 0 2023-09-30T01:38:35.259 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:35.238+0000 7fcef5ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:38:35.259 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:35.238+0000 7fcefee78700 5 monclient: authenticate success, global_id 4884 2023-09-30T01:38:35.259 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:35.238+0000 7fcef5ffb700 1 -- 172.21.15.88:0/1036828418 <== mon.6 v2:172.21.15.88:3303/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcef0011a10 con 0x7fcef80a3380 2023-09-30T01:38:35.259 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:35.238+0000 7fcefee78700 10 monclient: _renew_subs 2023-09-30T01:38:35.259 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:35.238+0000 7fcefee78700 10 monclient: _send_mon_message to mon.d at v2:172.21.15.88:3303/0 2023-09-30T01:38:35.259 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:35.238+0000 7fcefee78700 1 -- 172.21.15.88:0/1036828418 --> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcef811ff90 con 0x7fcef80a3380 2023-09-30T01:38:35.259 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:35.238+0000 7fcef5ffb700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:38:35.260 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:35.238+0000 7fcef5ffb700 1 -- 172.21.15.88:0/1036828418 <== mon.6 v2:172.21.15.88:3303/0 3 ==== mon_map magic: 0 v1 ==== 1037+0+0 (secure 0 0 0) 0x7fcef00102e0 con 0x7fcef80a3380 2023-09-30T01:38:35.260 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:35.238+0000 7fcef5ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:38:35.260 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:35.238+0000 7fcef5ffb700 10 monclient: got monmap 1 from mon.d (according to old e1) 2023-09-30T01:38:35.260 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:35.238+0000 7fcef7fff700 4 set_mon_vals no callback set 2023-09-30T01:38:35.260 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:35.238+0000 7fcef5ffb700 10 monclient: dump: 2023-09-30T01:38:35.260 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2023-09-30T01:38:35.260 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 650badb5-899b-4472-88a6-cfdccee24d52 2023-09-30T01:38:35.260 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2023-09-30T01:32:16.800064+0000 2023-09-30T01:38:35.260 INFO:teuthology.orchestra.run.smithi088.stderr:created 2023-09-30T01:32:16.800064+0000 2023-09-30T01:38:35.260 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 17 (quincy) 2023-09-30T01:38:35.260 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2023-09-30T01:38:35.260 INFO:teuthology.orchestra.run.smithi088.stderr:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-09-30T01:38:35.261 INFO:teuthology.orchestra.run.smithi088.stderr:1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.f 2023-09-30T01:38:35.261 INFO:teuthology.orchestra.run.smithi088.stderr:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.b 2023-09-30T01:38:35.261 INFO:teuthology.orchestra.run.smithi088.stderr:3: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.g 2023-09-30T01:38:35.261 INFO:teuthology.orchestra.run.smithi088.stderr:4: [v2:172.21.15.88:3302/0,v1:172.21.15.88:6791/0] mon.c 2023-09-30T01:38:35.261 INFO:teuthology.orchestra.run.smithi088.stderr:5: [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] mon.h 2023-09-30T01:38:35.261 INFO:teuthology.orchestra.run.smithi088.stderr:6: [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] mon.d 2023-09-30T01:38:35.261 INFO:teuthology.orchestra.run.smithi088.stderr:7: [v2:172.21.15.107:3303/0,v1:172.21.15.107:6792/0] mon.i 2023-09-30T01:38:35.261 INFO:teuthology.orchestra.run.smithi088.stderr:8: [v2:172.21.15.88:3304/0,v1:172.21.15.88:6793/0] mon.e 2023-09-30T01:38:35.261 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-09-30T01:38:35.261 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:35.238+0000 7fcefee78700 10 monclient: _renew_subs 2023-09-30T01:38:35.261 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:35.238+0000 7fcefee78700 10 monclient: _send_mon_message to mon.d at v2:172.21.15.88:3303/0 2023-09-30T01:38:35.261 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:35.238+0000 7fcefee78700 1 -- 172.21.15.88:0/1036828418 --> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcef811f3d0 con 0x7fcef80a3380 2023-09-30T01:38:35.261 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:35.238+0000 7fcef5ffb700 1 -- 172.21.15.88:0/1036828418 <== mon.6 v2:172.21.15.88:3303/0 4 ==== mgrmap(e 5) v1 ==== 40144+0+0 (secure 0 0 0) 0x7fcef0011b80 con 0x7fcef80a3380 2023-09-30T01:38:35.262 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:35.238+0000 7fcef5ffb700 1 --2- 172.21.15.88:0/1036828418 >> v2:172.21.15.107:6812/29926 conn(0x7fcee4033360 0x7fcee4035810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:38:35.262 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:35.238+0000 7fcef5ffb700 1 -- 172.21.15.88:0/1036828418 <== mon.6 v2:172.21.15.88:3303/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 3502+0+0 (secure 0 0 0) 0x7fcef00422e0 con 0x7fcef80a3380 2023-09-30T01:38:35.262 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:35.238+0000 7fcefee78700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:38:35.262 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:35.238+0000 7fcefee78700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:38:35.262 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:35.238+0000 7fcefee78700 10 monclient: _send_mon_message to mon.d at v2:172.21.15.88:3303/0 2023-09-30T01:38:35.262 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:35.238+0000 7fcefee78700 1 -- 172.21.15.88:0/1036828418 --> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcef8120740 con 0x7fcef80a3380 2023-09-30T01:38:35.262 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:35.238+0000 7fcefde76700 1 --2- 172.21.15.88:0/1036828418 >> v2:172.21.15.107:6812/29926 conn(0x7fcee4033360 0x7fcee4035810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:38:35.262 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:35.238+0000 7fcefde76700 10 monclient: get_auth_request con 0x7fcee4033360 auth_method 0 2023-09-30T01:38:35.263 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:35.242+0000 7fcefde76700 1 --2- 172.21.15.88:0/1036828418 >> v2:172.21.15.107:6812/29926 conn(0x7fcee4033360 0x7fcee4035810 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fcee8009de0 tx=0x7fcee8019000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:38:35.263 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:35.246+0000 7fcef5ffb700 1 -- 172.21.15.88:0/1036828418 <== mon.6 v2:172.21.15.88:3303/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fcef0017030 con 0x7fcef80a3380 2023-09-30T01:38:35.263 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:35.246+0000 7fcef5ffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:38:35.263 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:35.246+0000 7fcef5ffb700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:38:35.423 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:35.418+0000 7fcefee78700 10 start_mon_command cmd=[{"prefix": "config set", "who": "global", "name": "mon_health_to_clog", "value": "false"}] 2023-09-30T01:38:35.423 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:35.418+0000 7fcefee78700 10 monclient: _send_command 2 [{"prefix": "config set", "who": "global", "name": "mon_health_to_clog", "value": "false"}] 2023-09-30T01:38:35.424 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:35.418+0000 7fcefee78700 10 monclient: _send_mon_message to mon.d at v2:172.21.15.88:3303/0 2023-09-30T01:38:35.424 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:35.418+0000 7fcefee78700 1 -- 172.21.15.88:0/1036828418 --> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] -- mon_command([{prefix=config set, name=mon_health_to_clog}] v 0) v1 -- 0x7fcef80710d0 con 0x7fcef80a3380 2023-09-30T01:38:35.432 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:35.430+0000 7fcef5ffb700 1 -- 172.21.15.88:0/1036828418 <== mon.6 v2:172.21.15.88:3303/0 7 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7fcef0018450 con 0x7fcef80a3380 2023-09-30T01:38:35.433 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:35.430+0000 7fcef5ffb700 10 monclient: handle_config config(1 keys) v1 2023-09-30T01:38:35.433 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:35.430+0000 7fcef7fff700 4 set_mon_vals no callback set 2023-09-30T01:38:35.433 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:35.430+0000 7fcef7fff700 10 set_mon_vals mon_health_to_clog = false 2023-09-30T01:38:35.434 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:35.430+0000 7fcef5ffb700 1 -- 172.21.15.88:0/1036828418 <== mon.6 v2:172.21.15.88:3303/0 8 ==== mon_command_ack([{prefix=config set, name=mon_health_to_clog}]=0 v2)=0 v2) v1 ==== 123+0+0 (secure 0 0 0) 0x7fcef0017020 con 0x7fcef80a3380 2023-09-30T01:38:35.434 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:35.430+0000 7fcef5ffb700 10 monclient: handle_mon_command_ack 2 [{"prefix": "config set", "who": "global", "name": "mon_health_to_clog", "value": "false"}] 2023-09-30T01:38:35.435 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:35.430+0000 7fcef5ffb700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:38:35.437 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:35.434+0000 7fcefee78700 1 -- 172.21.15.88:0/1036828418 >> v2:172.21.15.107:6812/29926 conn(0x7fcee4033360 msgr2=0x7fcee4035810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:38:35.437 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:35.434+0000 7fcefee78700 1 --2- 172.21.15.88:0/1036828418 >> v2:172.21.15.107:6812/29926 conn(0x7fcee4033360 0x7fcee4035810 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fcee8009de0 tx=0x7fcee8019000 comp rx=0 tx=0).stop 2023-09-30T01:38:35.438 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:35.434+0000 7fcefee78700 10 monclient: shutdown 2023-09-30T01:38:35.438 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:35.434+0000 7fcefee78700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:38:35.438 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:35.434+0000 7fcefee78700 1 -- 172.21.15.88:0/1036828418 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7fcef80a3380 msgr2=0x7fcef811d4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:38:35.438 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:35.434+0000 7fcefee78700 1 --2- 172.21.15.88:0/1036828418 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7fcef80a3380 0x7fcef811d4a0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fcef000c4b0 tx=0x7fcef0007e60 comp rx=0 tx=0).stop 2023-09-30T01:38:35.439 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:35.434+0000 7fcefee78700 1 -- 172.21.15.88:0/1036828418 shutdown_connections 2023-09-30T01:38:35.439 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:35.434+0000 7fcefee78700 1 --2- 172.21.15.88:0/1036828418 >> v2:172.21.15.107:6812/29926 conn(0x7fcee4033360 0x7fcee4035810 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:38:35.439 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:35.434+0000 7fcefee78700 1 --2- 172.21.15.88:0/1036828418 >> [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] conn(0x7fcef80a3380 0x7fcef811d4a0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:38:35.439 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:35.434+0000 7fcefee78700 1 --2- 172.21.15.88:0/1036828418 >> [v2:172.21.15.107:3302/0,v1:172.21.15.107:6791/0] conn(0x7fcef80a03a0 0x7fcef811cf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:38:35.439 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:35.434+0000 7fcefee78700 1 --2- 172.21.15.88:0/1036828418 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fcef809ffd0 0x7fcef8131180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:38:35.440 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:35.434+0000 7fcefee78700 1 -- 172.21.15.88:0/1036828418 >> 172.21.15.88:0/1036828418 conn(0x7fcef806e5e0 msgr2=0x7fcef80a2a70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:38:35.440 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:35.434+0000 7fcefee78700 1 -- 172.21.15.88:0/1036828418 shutdown_connections 2023-09-30T01:38:35.440 INFO:teuthology.orchestra.run.smithi088.stderr:2023-09-30T01:38:35.434+0000 7fcefee78700 1 -- 172.21.15.88:0/1036828418 wait complete. 2023-09-30T01:38:35.447 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/github.com_ceph_ceph-c_93dce5e17918f8aca65993a7a2dc40233a47b481/qa/tasks/ceph.py", line 1895, in task healthy(ctx=ctx, config=dict(cluster=config['cluster'])) File "/home/teuthworker/src/github.com_ceph_ceph-c_93dce5e17918f8aca65993a7a2dc40233a47b481/qa/tasks/ceph.py", line 1472, in healthy manager.wait_until_healthy(timeout=300) File "/home/teuthworker/src/github.com_ceph_ceph-c_93dce5e17918f8aca65993a7a2dc40233a47b481/qa/tasks/ceph_manager.py", line 3189, in wait_until_healthy assert time.time() - start < timeout, \ AssertionError: timeout expired in wait_until_healthy 2023-09-30T01:38:35.449 INFO:teuthology.misc:Shutting down mds daemons... 2023-09-30T01:38:35.449 INFO:teuthology.misc:Shutting down osd daemons... 2023-09-30T01:38:35.449 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2023-09-30T01:38:35.449 INFO:teuthology.orchestra.run:waiting for 300 2023-09-30T01:38:35.520 INFO:tasks.ceph.osd.0:Stopped 2023-09-30T01:38:35.520 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2023-09-30T01:38:35.520 INFO:teuthology.orchestra.run:waiting for 300 2023-09-30T01:38:35.614 INFO:tasks.ceph.osd.1:Stopped 2023-09-30T01:38:35.614 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2023-09-30T01:38:35.614 INFO:teuthology.orchestra.run:waiting for 300 2023-09-30T01:38:35.696 INFO:tasks.ceph.osd.2:Stopped 2023-09-30T01:38:35.697 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2023-09-30T01:38:35.697 INFO:teuthology.orchestra.run:waiting for 300 2023-09-30T01:38:35.767 INFO:tasks.ceph.osd.3:Stopped 2023-09-30T01:38:35.767 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2023-09-30T01:38:35.767 INFO:teuthology.orchestra.run:waiting for 300 2023-09-30T01:38:35.840 INFO:tasks.ceph.osd.4:Stopped 2023-09-30T01:38:35.841 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2023-09-30T01:38:35.841 INFO:teuthology.orchestra.run:waiting for 300 2023-09-30T01:38:35.931 INFO:tasks.ceph.osd.5:Stopped 2023-09-30T01:38:35.931 INFO:teuthology.misc:Shutting down mgr daemons... 2023-09-30T01:38:35.931 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2023-09-30T01:38:35.932 INFO:teuthology.orchestra.run:waiting for 300 2023-09-30T01:38:35.951 INFO:tasks.ceph.mgr.x:Stopped 2023-09-30T01:38:35.952 INFO:teuthology.misc:Shutting down mon daemons... 2023-09-30T01:38:35.952 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2023-09-30T01:38:35.952 INFO:teuthology.orchestra.run:waiting for 300 2023-09-30T01:38:35.971 INFO:tasks.ceph.mon.a:Stopped 2023-09-30T01:38:35.971 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2023-09-30T01:38:35.971 INFO:teuthology.orchestra.run:waiting for 300 2023-09-30T01:38:36.028 INFO:tasks.ceph.mon.b:Stopped 2023-09-30T01:38:36.029 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2023-09-30T01:38:36.029 INFO:teuthology.orchestra.run:waiting for 300 2023-09-30T01:38:36.047 INFO:tasks.ceph.mon.c:Stopped 2023-09-30T01:38:36.047 DEBUG:tasks.ceph.mon.d:waiting for process to exit 2023-09-30T01:38:36.047 INFO:teuthology.orchestra.run:waiting for 300 2023-09-30T01:38:36.067 INFO:tasks.ceph.mon.d:Stopped 2023-09-30T01:38:36.067 DEBUG:tasks.ceph.mon.e:waiting for process to exit 2023-09-30T01:38:36.067 INFO:teuthology.orchestra.run:waiting for 300 2023-09-30T01:38:36.082 INFO:tasks.ceph.mon.e:Stopped 2023-09-30T01:38:36.083 DEBUG:tasks.ceph.mon.f:waiting for process to exit 2023-09-30T01:38:36.083 INFO:teuthology.orchestra.run:waiting for 300 2023-09-30T01:38:36.101 INFO:tasks.ceph.mon.f:Stopped 2023-09-30T01:38:36.101 DEBUG:tasks.ceph.mon.g:waiting for process to exit 2023-09-30T01:38:36.101 INFO:teuthology.orchestra.run:waiting for 300 2023-09-30T01:38:36.116 INFO:tasks.ceph.mon.g:Stopped 2023-09-30T01:38:36.116 DEBUG:tasks.ceph.mon.h:waiting for process to exit 2023-09-30T01:38:36.117 INFO:teuthology.orchestra.run:waiting for 300 2023-09-30T01:38:36.135 INFO:tasks.ceph.mon.h:Stopped 2023-09-30T01:38:36.135 DEBUG:tasks.ceph.mon.i:waiting for process to exit 2023-09-30T01:38:36.135 INFO:teuthology.orchestra.run:waiting for 300 2023-09-30T01:38:36.152 INFO:tasks.ceph.mon.i:Stopped 2023-09-30T01:38:36.152 INFO:tasks.ceph:Checking cluster log for badness... 2023-09-30T01:38:36.152 DEBUG:teuthology.orchestra.run.smithi088:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v ScrubResult | egrep -v 'scrub mismatch' | egrep -v 'overall HEALTH_' | egrep -v '\(MGR_DOWN\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(SLOW_OPS\)' | egrep -v 'slow request' | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'overall HEALTH_' | egrep -v '\(MON_DOWN\)' | egrep -v '\(OSDMAP_FLAGS\)' | head -n 1 2023-09-30T01:38:36.173 INFO:teuthology.orchestra.run.smithi088.stdout:1696037575.6718426 mgr.x (mgr.4108) 1 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.x: 'PgAutoscaler' object has no attribute 'noautoscale' 2023-09-30T01:38:36.174 WARNING:tasks.ceph:Found errors (ERR|WRN|SEC) in cluster log 2023-09-30T01:38:36.174 DEBUG:teuthology.orchestra.run.smithi088:> sudo egrep '\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v ScrubResult | egrep -v 'scrub mismatch' | egrep -v 'overall HEALTH_' | egrep -v '\(MGR_DOWN\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(SLOW_OPS\)' | egrep -v 'slow request' | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'overall HEALTH_' | egrep -v '\(MON_DOWN\)' | egrep -v '\(OSDMAP_FLAGS\)' | head -n 1 2023-09-30T01:38:36.190 DEBUG:teuthology.orchestra.run.smithi088:> sudo egrep '\[ERR\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v ScrubResult | egrep -v 'scrub mismatch' | egrep -v 'overall HEALTH_' | egrep -v '\(MGR_DOWN\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(SLOW_OPS\)' | egrep -v 'slow request' | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'overall HEALTH_' | egrep -v '\(MON_DOWN\)' | egrep -v '\(OSDMAP_FLAGS\)' | head -n 1 2023-09-30T01:38:36.205 INFO:teuthology.orchestra.run.smithi088.stdout:1696037575.6718426 mgr.x (mgr.4108) 1 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.x: 'PgAutoscaler' object has no attribute 'noautoscale' 2023-09-30T01:38:36.207 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi088.front.sepia.ceph.com 2023-09-30T01:38:36.208 DEBUG:teuthology.orchestra.run.smithi088:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2023-09-30T01:38:36.546 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi088.front.sepia.ceph.com 2023-09-30T01:38:36.547 DEBUG:teuthology.orchestra.run.smithi088:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2023-09-30T01:38:36.785 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi088.front.sepia.ceph.com 2023-09-30T01:38:36.785 DEBUG:teuthology.orchestra.run.smithi088:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2023-09-30T01:38:36.984 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi107.front.sepia.ceph.com 2023-09-30T01:38:36.984 DEBUG:teuthology.orchestra.run.smithi107:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2023-09-30T01:38:37.172 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi107.front.sepia.ceph.com 2023-09-30T01:38:37.172 DEBUG:teuthology.orchestra.run.smithi107:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2023-09-30T01:38:37.403 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi107.front.sepia.ceph.com 2023-09-30T01:38:37.404 DEBUG:teuthology.orchestra.run.smithi107:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2023-09-30T01:38:37.559 INFO:tasks.ceph:Archiving mon data... 2023-09-30T01:38:37.561 DEBUG:teuthology.misc:Transferring archived files from smithi088:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi/7406999/data/mon.a.tgz 2023-09-30T01:38:37.561 DEBUG:teuthology.orchestra.run.smithi088:> mktemp 2023-09-30T01:38:37.567 INFO:teuthology.orchestra.run.smithi088.stdout:/tmp/tmp.zK0GwaHmej 2023-09-30T01:38:37.568 DEBUG:teuthology.orchestra.run.smithi088:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.zK0GwaHmej 2023-09-30T01:38:37.675 DEBUG:teuthology.orchestra.run.smithi088:> sudo chmod 0666 /tmp/tmp.zK0GwaHmej 2023-09-30T01:38:37.700 DEBUG:teuthology.orchestra.remote:smithi088:/tmp/tmp.zK0GwaHmej is 228KB 2023-09-30T01:38:37.757 DEBUG:teuthology.orchestra.run.smithi088:> rm -fr /tmp/tmp.zK0GwaHmej 2023-09-30T01:38:37.764 DEBUG:teuthology.misc:Transferring archived files from smithi088:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi/7406999/data/mon.b.tgz 2023-09-30T01:38:37.764 DEBUG:teuthology.orchestra.run.smithi088:> mktemp 2023-09-30T01:38:37.771 INFO:teuthology.orchestra.run.smithi088.stdout:/tmp/tmp.ljY4raEJ3n 2023-09-30T01:38:37.771 DEBUG:teuthology.orchestra.run.smithi088:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.ljY4raEJ3n 2023-09-30T01:38:37.831 DEBUG:teuthology.orchestra.run.smithi088:> sudo chmod 0666 /tmp/tmp.ljY4raEJ3n 2023-09-30T01:38:37.898 DEBUG:teuthology.orchestra.remote:smithi088:/tmp/tmp.ljY4raEJ3n is 228KB 2023-09-30T01:38:37.958 DEBUG:teuthology.orchestra.run.smithi088:> rm -fr /tmp/tmp.ljY4raEJ3n 2023-09-30T01:38:37.965 DEBUG:teuthology.misc:Transferring archived files from smithi088:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi/7406999/data/mon.c.tgz 2023-09-30T01:38:37.965 DEBUG:teuthology.orchestra.run.smithi088:> mktemp 2023-09-30T01:38:37.971 INFO:teuthology.orchestra.run.smithi088.stdout:/tmp/tmp.YDV5tcnjXr 2023-09-30T01:38:37.971 DEBUG:teuthology.orchestra.run.smithi088:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.YDV5tcnjXr 2023-09-30T01:38:38.030 DEBUG:teuthology.orchestra.run.smithi088:> sudo chmod 0666 /tmp/tmp.YDV5tcnjXr 2023-09-30T01:38:38.049 DEBUG:teuthology.orchestra.remote:smithi088:/tmp/tmp.YDV5tcnjXr is 228KB 2023-09-30T01:38:38.110 DEBUG:teuthology.orchestra.run.smithi088:> rm -fr /tmp/tmp.YDV5tcnjXr 2023-09-30T01:38:38.116 DEBUG:teuthology.misc:Transferring archived files from smithi088:/var/lib/ceph/mon/ceph-d to /home/teuthworker/archive/yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi/7406999/data/mon.d.tgz 2023-09-30T01:38:38.116 DEBUG:teuthology.orchestra.run.smithi088:> mktemp 2023-09-30T01:38:38.123 INFO:teuthology.orchestra.run.smithi088.stdout:/tmp/tmp.5rar0ln5CS 2023-09-30T01:38:38.123 DEBUG:teuthology.orchestra.run.smithi088:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-d -- . > /tmp/tmp.5rar0ln5CS 2023-09-30T01:38:38.181 DEBUG:teuthology.orchestra.run.smithi088:> sudo chmod 0666 /tmp/tmp.5rar0ln5CS 2023-09-30T01:38:38.251 DEBUG:teuthology.orchestra.remote:smithi088:/tmp/tmp.5rar0ln5CS is 227KB 2023-09-30T01:38:38.271 DEBUG:teuthology.orchestra.run.smithi088:> rm -fr /tmp/tmp.5rar0ln5CS 2023-09-30T01:38:38.278 DEBUG:teuthology.misc:Transferring archived files from smithi088:/var/lib/ceph/mon/ceph-e to /home/teuthworker/archive/yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi/7406999/data/mon.e.tgz 2023-09-30T01:38:38.278 DEBUG:teuthology.orchestra.run.smithi088:> mktemp 2023-09-30T01:38:38.329 INFO:teuthology.orchestra.run.smithi088.stdout:/tmp/tmp.LSnt3hl3TK 2023-09-30T01:38:38.330 DEBUG:teuthology.orchestra.run.smithi088:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-e -- . > /tmp/tmp.LSnt3hl3TK 2023-09-30T01:38:38.391 DEBUG:teuthology.orchestra.run.smithi088:> sudo chmod 0666 /tmp/tmp.LSnt3hl3TK 2023-09-30T01:38:38.415 DEBUG:teuthology.orchestra.remote:smithi088:/tmp/tmp.LSnt3hl3TK is 227KB 2023-09-30T01:38:38.478 DEBUG:teuthology.orchestra.run.smithi088:> rm -fr /tmp/tmp.LSnt3hl3TK 2023-09-30T01:38:38.484 DEBUG:teuthology.misc:Transferring archived files from smithi107:/var/lib/ceph/mon/ceph-f to /home/teuthworker/archive/yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi/7406999/data/mon.f.tgz 2023-09-30T01:38:38.485 DEBUG:teuthology.orchestra.run.smithi107:> mktemp 2023-09-30T01:38:38.491 INFO:teuthology.orchestra.run.smithi107.stdout:/tmp/tmp.nIFGfKH1J0 2023-09-30T01:38:38.492 DEBUG:teuthology.orchestra.run.smithi107:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-f -- . > /tmp/tmp.nIFGfKH1J0 2023-09-30T01:38:38.592 DEBUG:teuthology.orchestra.run.smithi107:> sudo chmod 0666 /tmp/tmp.nIFGfKH1J0 2023-09-30T01:38:38.615 DEBUG:teuthology.orchestra.remote:smithi107:/tmp/tmp.nIFGfKH1J0 is 228KB 2023-09-30T01:38:38.678 DEBUG:teuthology.orchestra.run.smithi107:> rm -fr /tmp/tmp.nIFGfKH1J0 2023-09-30T01:38:38.685 DEBUG:teuthology.misc:Transferring archived files from smithi107:/var/lib/ceph/mon/ceph-g to /home/teuthworker/archive/yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi/7406999/data/mon.g.tgz 2023-09-30T01:38:38.685 DEBUG:teuthology.orchestra.run.smithi107:> mktemp 2023-09-30T01:38:38.692 INFO:teuthology.orchestra.run.smithi107.stdout:/tmp/tmp.4txbpC9XEy 2023-09-30T01:38:38.692 DEBUG:teuthology.orchestra.run.smithi107:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-g -- . > /tmp/tmp.4txbpC9XEy 2023-09-30T01:38:38.751 DEBUG:teuthology.orchestra.run.smithi107:> sudo chmod 0666 /tmp/tmp.4txbpC9XEy 2023-09-30T01:38:38.774 DEBUG:teuthology.orchestra.remote:smithi107:/tmp/tmp.4txbpC9XEy is 228KB 2023-09-30T01:38:38.838 DEBUG:teuthology.orchestra.run.smithi107:> rm -fr /tmp/tmp.4txbpC9XEy 2023-09-30T01:38:38.845 DEBUG:teuthology.misc:Transferring archived files from smithi107:/var/lib/ceph/mon/ceph-h to /home/teuthworker/archive/yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi/7406999/data/mon.h.tgz 2023-09-30T01:38:38.845 DEBUG:teuthology.orchestra.run.smithi107:> mktemp 2023-09-30T01:38:38.851 INFO:teuthology.orchestra.run.smithi107.stdout:/tmp/tmp.cay8uRX1J9 2023-09-30T01:38:38.851 DEBUG:teuthology.orchestra.run.smithi107:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-h -- . > /tmp/tmp.cay8uRX1J9 2023-09-30T01:38:38.917 DEBUG:teuthology.orchestra.run.smithi107:> sudo chmod 0666 /tmp/tmp.cay8uRX1J9 2023-09-30T01:38:38.942 DEBUG:teuthology.orchestra.remote:smithi107:/tmp/tmp.cay8uRX1J9 is 228KB 2023-09-30T01:38:39.003 DEBUG:teuthology.orchestra.run.smithi107:> rm -fr /tmp/tmp.cay8uRX1J9 2023-09-30T01:38:39.009 DEBUG:teuthology.misc:Transferring archived files from smithi107:/var/lib/ceph/mon/ceph-i to /home/teuthworker/archive/yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi/7406999/data/mon.i.tgz 2023-09-30T01:38:39.010 DEBUG:teuthology.orchestra.run.smithi107:> mktemp 2023-09-30T01:38:39.016 INFO:teuthology.orchestra.run.smithi107.stdout:/tmp/tmp.4Nr4xCGdXA 2023-09-30T01:38:39.016 DEBUG:teuthology.orchestra.run.smithi107:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-i -- . > /tmp/tmp.4Nr4xCGdXA 2023-09-30T01:38:39.082 DEBUG:teuthology.orchestra.run.smithi107:> sudo chmod 0666 /tmp/tmp.4Nr4xCGdXA 2023-09-30T01:38:39.103 DEBUG:teuthology.orchestra.remote:smithi107:/tmp/tmp.4Nr4xCGdXA is 228KB 2023-09-30T01:38:39.162 DEBUG:teuthology.orchestra.run.smithi107:> rm -fr /tmp/tmp.4Nr4xCGdXA 2023-09-30T01:38:39.169 INFO:tasks.ceph:Cleaning ceph cluster... 2023-09-30T01:38:39.169 DEBUG:teuthology.orchestra.run.smithi088:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2023-09-30T01:38:39.171 DEBUG:teuthology.orchestra.run.smithi107:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2023-09-30T01:38:39.185 INFO:tasks.ceph:Checking for errors in any valgrind logs... 2023-09-30T01:38:39.186 DEBUG:teuthology.orchestra.run.smithi088:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2023-09-30T01:38:39.229 DEBUG:teuthology.orchestra.run.smithi107:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2023-09-30T01:38:39.255 INFO:teuthology.orchestra.run.smithi088.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2023-09-30T01:38:39.256 INFO:teuthology.orchestra.run.smithi107.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2023-09-30T01:38:39.263 INFO:tasks.ceph:Archiving crash dumps... 2023-09-30T01:38:39.265 DEBUG:teuthology.misc:Transferring archived files from smithi088:/var/lib/ceph/crash to /home/teuthworker/archive/yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi/7406999/remote/smithi088/crash 2023-09-30T01:38:39.265 DEBUG:teuthology.orchestra.run.smithi088:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2023-09-30T01:38:39.283 DEBUG:teuthology.misc:Transferring archived files from smithi107:/var/lib/ceph/crash to /home/teuthworker/archive/yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi/7406999/remote/smithi107/crash 2023-09-30T01:38:39.284 DEBUG:teuthology.orchestra.run.smithi107:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2023-09-30T01:38:39.322 INFO:tasks.ceph:Compressing logs... 2023-09-30T01:38:39.323 DEBUG:teuthology.orchestra.run.smithi088:> sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-09-30T01:38:39.325 DEBUG:teuthology.orchestra.run.smithi107:> sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-09-30T01:38:40.746 INFO:tasks.ceph:Archiving logs... 2023-09-30T01:38:40.746 DEBUG:teuthology.misc:Transferring archived files from smithi088:/var/log/ceph to /home/teuthworker/archive/yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi/7406999/remote/smithi088/log 2023-09-30T01:38:40.747 DEBUG:teuthology.orchestra.run.smithi088:> sudo tar c -f - -C /var/log/ceph -- . 2023-09-30T01:38:41.057 DEBUG:teuthology.misc:Transferring archived files from smithi107:/var/log/ceph to /home/teuthworker/archive/yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi/7406999/remote/smithi107/log 2023-09-30T01:38:41.058 DEBUG:teuthology.orchestra.run.smithi107:> sudo tar c -f - -C /var/log/ceph -- . 2023-09-30T01:38:41.211 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_93dce5e17918f8aca65993a7a2dc40233a47b481/qa/tasks/ceph.py", line 1895, in task healthy(ctx=ctx, config=dict(cluster=config['cluster'])) File "/home/teuthworker/src/github.com_ceph_ceph-c_93dce5e17918f8aca65993a7a2dc40233a47b481/qa/tasks/ceph.py", line 1472, in healthy manager.wait_until_healthy(timeout=300) File "/home/teuthworker/src/github.com_ceph_ceph-c_93dce5e17918f8aca65993a7a2dc40233a47b481/qa/tasks/ceph_manager.py", line 3189, in wait_until_healthy assert time.time() - start < timeout, \ AssertionError: timeout expired in wait_until_healthy 2023-09-30T01:38:41.278 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=d8ba4ebfdcfc436a8db1f3fa32bf4e29 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_93dce5e17918f8aca65993a7a2dc40233a47b481/qa/tasks/ceph.py", line 1895, in task healthy(ctx=ctx, config=dict(cluster=config['cluster'])) File "/home/teuthworker/src/github.com_ceph_ceph-c_93dce5e17918f8aca65993a7a2dc40233a47b481/qa/tasks/ceph.py", line 1472, in healthy manager.wait_until_healthy(timeout=300) File "/home/teuthworker/src/github.com_ceph_ceph-c_93dce5e17918f8aca65993a7a2dc40233a47b481/qa/tasks/ceph_manager.py", line 3189, in wait_until_healthy assert time.time() - start < timeout, \ AssertionError: timeout expired in wait_until_healthy 2023-09-30T01:38:41.280 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2023-09-30T01:38:41.294 DEBUG:teuthology.run_tasks:Unwinding manager install 2023-09-30T01:38:41.322 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2023-09-30T01:38:41.322 DEBUG:teuthology.orchestra.run.smithi088:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-09-30T01:38:41.333 DEBUG:teuthology.orchestra.run.smithi107:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-09-30T01:38:41.398 DEBUG:teuthology.run_tasks:Unwinding manager clock 2023-09-30T01:38:41.406 INFO:teuthology.task.clock:Checking final clock skew... 2023-09-30T01:38:41.407 DEBUG:teuthology.orchestra.run.smithi088:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-09-30T01:38:41.409 DEBUG:teuthology.orchestra.run.smithi107:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-09-30T01:38:41.431 INFO:teuthology.orchestra.run.smithi088.stdout: remote refid st t when poll reach delay offset jitter 2023-09-30T01:38:41.431 INFO:teuthology.orchestra.run.smithi088.stdout:============================================================================== 2023-09-30T01:38:41.431 INFO:teuthology.orchestra.run.smithi088.stdout:*hv01.front.sepi 44.4.53.2 2 u 43 64 377 0.065 1.256 0.804 2023-09-30T01:38:41.431 INFO:teuthology.orchestra.run.smithi088.stdout:+hv02.front.sepi 63.231.80.2 3 u 49 64 377 0.074 1.339 0.798 2023-09-30T01:38:41.432 INFO:teuthology.orchestra.run.smithi088.stdout:+hv03.front.sepi 74.6.168.72 3 u 49 64 377 0.091 0.697 0.698 2023-09-30T01:38:41.432 INFO:teuthology.orchestra.run.smithi088.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2023-09-30T01:38:41.433 INFO:teuthology.orchestra.run.smithi107.stdout: remote refid st t when poll reach delay offset jitter 2023-09-30T01:38:41.433 INFO:teuthology.orchestra.run.smithi107.stdout:============================================================================== 2023-09-30T01:38:41.433 INFO:teuthology.orchestra.run.smithi107.stdout:*hv01.front.sepi 44.4.53.2 2 u 42 64 377 0.081 1.336 0.703 2023-09-30T01:38:41.434 INFO:teuthology.orchestra.run.smithi107.stdout:+hv02.front.sepi 63.231.80.2 3 u 47 64 377 0.054 1.034 0.427 2023-09-30T01:38:41.434 INFO:teuthology.orchestra.run.smithi107.stdout:+hv03.front.sepi 74.6.168.72 3 u 41 64 377 0.082 -0.267 0.844 2023-09-30T01:38:41.434 INFO:teuthology.orchestra.run.smithi107.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2023-09-30T01:38:41.434 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2023-09-30T01:38:41.443 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2023-09-30T01:38:41.444 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2023-09-30T01:38:41.472 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2023-09-30T01:38:41.507 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2023-09-30T01:38:41.541 INFO:teuthology.task.internal:Duration was 860.437548 seconds 2023-09-30T01:38:41.542 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2023-09-30T01:38:41.573 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2023-09-30T01:38:41.573 DEBUG:teuthology.orchestra.run.smithi088:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-09-30T01:38:41.576 DEBUG:teuthology.orchestra.run.smithi107:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-09-30T01:38:41.627 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2023-09-30T01:38:41.627 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi088.front.sepia.ceph.com 2023-09-30T01:38:41.628 DEBUG:teuthology.orchestra.run.smithi088:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-09-30T01:38:41.640 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi107.front.sepia.ceph.com 2023-09-30T01:38:41.640 DEBUG:teuthology.orchestra.run.smithi107:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-09-30T01:38:41.655 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2023-09-30T01:38:41.655 DEBUG:teuthology.orchestra.run.smithi088:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-09-30T01:38:41.689 DEBUG:teuthology.orchestra.run.smithi107:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-09-30T01:38:41.723 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2023-09-30T01:38:41.724 DEBUG:teuthology.orchestra.run.smithi088:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-09-30T01:38:41.726 DEBUG:teuthology.orchestra.run.smithi107:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-09-30T01:38:41.831 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2023-09-30T01:38:41.841 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2023-09-30T01:38:41.841 DEBUG:teuthology.orchestra.run.smithi088:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-09-30T01:38:41.857 DEBUG:teuthology.orchestra.run.smithi107:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-09-30T01:38:41.873 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2023-09-30T01:38:41.883 DEBUG:teuthology.orchestra.run.smithi088:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-09-30T01:38:41.901 DEBUG:teuthology.orchestra.run.smithi107:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-09-30T01:38:41.914 INFO:teuthology.orchestra.run.smithi088.stdout:kernel.core_pattern = core 2023-09-30T01:38:41.930 INFO:teuthology.orchestra.run.smithi107.stdout:kernel.core_pattern = core 2023-09-30T01:38:41.960 DEBUG:teuthology.orchestra.run.smithi088:> test -e /home/ubuntu/cephtest/archive/coredump 2023-09-30T01:38:41.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-09-30T01:38:41.967 DEBUG:teuthology.orchestra.run.smithi107:> test -e /home/ubuntu/cephtest/archive/coredump 2023-09-30T01:38:41.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-09-30T01:38:41.972 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2023-09-30T01:38:41.982 INFO:teuthology.task.internal:Transferring archived files... 2023-09-30T01:38:41.984 DEBUG:teuthology.misc:Transferring archived files from smithi088:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi/7406999/remote/smithi088 2023-09-30T01:38:41.984 DEBUG:teuthology.orchestra.run.smithi088:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-09-30T01:38:42.005 DEBUG:teuthology.misc:Transferring archived files from smithi107:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi/7406999/remote/smithi107 2023-09-30T01:38:42.006 DEBUG:teuthology.orchestra.run.smithi107:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-09-30T01:38:42.038 INFO:teuthology.task.internal:Removing archive directory... 2023-09-30T01:38:42.038 DEBUG:teuthology.orchestra.run.smithi088:> rm -rf -- /home/ubuntu/cephtest/archive 2023-09-30T01:38:42.045 DEBUG:teuthology.orchestra.run.smithi107:> rm -rf -- /home/ubuntu/cephtest/archive 2023-09-30T01:38:42.051 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2023-09-30T01:38:42.062 INFO:teuthology.task.internal:Not uploading archives. 2023-09-30T01:38:42.062 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2023-09-30T01:38:42.090 INFO:teuthology.task.internal:Tidying up after the test... 2023-09-30T01:38:42.091 DEBUG:teuthology.orchestra.run.smithi088:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-09-30T01:38:42.093 DEBUG:teuthology.orchestra.run.smithi107:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-09-30T01:38:42.098 INFO:teuthology.orchestra.run.smithi088.stdout: 924112 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 30 01:38 /home/ubuntu/cephtest 2023-09-30T01:38:42.101 INFO:teuthology.orchestra.run.smithi107.stdout: 924112 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 30 01:38 /home/ubuntu/cephtest 2023-09-30T01:38:42.103 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2023-09-30T01:38:42.123 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2023-09-30T01:38:42.190 INFO:teuthology.nuke:Checking targets against current locks 2023-09-30T01:38:42.216 DEBUG:teuthology.nuke:shortname: smithi088 2023-09-30T01:38:42.216 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-09-30T01:38:42.242 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi088.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi/7406999', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '20.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-09-30 01:14:57.489390', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFyT/r7xuKfmg50w0OdQebr4A/ptJo1Zk6DyZNZNZM/q5rW+sznh+wvHqeT3e4YtCdZLaXyvMms8o8u8H5nXOYM='} 2023-09-30T01:38:42.244 DEBUG:teuthology.nuke:shortname: smithi107 2023-09-30T01:38:42.244 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-09-30T01:38:42.270 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi107.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi/7406999', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '20.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-09-30 01:14:57.490731', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFyT/r7xuKfmg50w0OdQebr4A/ptJo1Zk6DyZNZNZM/q5rW+sznh+wvHqeT3e4YtCdZLaXyvMms8o8u8H5nXOYM='} 2023-09-30T01:38:42.294 INFO:teuthology.orchestra.console.smithi088:Power off 2023-09-30T01:38:42.294 DEBUG:teuthology.orchestra.console.smithi088:pexpect command: ipmitool -H smithi088.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-09-30T01:38:42.317 DEBUG:teuthology.orchestra.console.smithi088:power off output: Chassis Power Control: Down/Off 2023-09-30T01:38:42.317 DEBUG:teuthology.orchestra.console.smithi088:pexpect command: ipmitool -H smithi088.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-30T01:38:42.335 INFO:teuthology.orchestra.console.smithi107:Power off 2023-09-30T01:38:42.336 DEBUG:teuthology.orchestra.console.smithi107:pexpect command: ipmitool -H smithi107.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-09-30T01:38:42.345 DEBUG:teuthology.orchestra.console.smithi088:check power output: Chassis Power is on 2023-09-30T01:38:42.361 DEBUG:teuthology.orchestra.console.smithi107:power off output: Chassis Power Control: Down/Off 2023-09-30T01:38:42.362 DEBUG:teuthology.orchestra.console.smithi107:pexpect command: ipmitool -H smithi107.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-30T01:38:42.387 DEBUG:teuthology.orchestra.console.smithi107:check power output: Chassis Power is on 2023-09-30T01:38:46.346 DEBUG:teuthology.orchestra.console.smithi088:pexpect command: ipmitool -H smithi088.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-30T01:38:46.388 DEBUG:teuthology.orchestra.console.smithi107:pexpect command: ipmitool -H smithi107.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-30T01:38:46.457 DEBUG:teuthology.orchestra.console.smithi088:check power output: Chassis Power is on 2023-09-30T01:38:46.501 DEBUG:teuthology.orchestra.console.smithi107:check power output: Chassis Power is on 2023-09-30T01:38:50.462 DEBUG:teuthology.orchestra.console.smithi088:pexpect command: ipmitool -H smithi088.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-30T01:38:50.501 DEBUG:teuthology.orchestra.console.smithi107:pexpect command: ipmitool -H smithi107.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-30T01:38:50.573 DEBUG:teuthology.orchestra.console.smithi088:check power output: Chassis Power is on 2023-09-30T01:38:50.611 DEBUG:teuthology.orchestra.console.smithi107:check power output: Chassis Power is on 2023-09-30T01:38:54.574 DEBUG:teuthology.orchestra.console.smithi088:pexpect command: ipmitool -H smithi088.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-30T01:38:54.612 DEBUG:teuthology.orchestra.console.smithi107:pexpect command: ipmitool -H smithi107.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-30T01:38:54.684 DEBUG:teuthology.orchestra.console.smithi088:check power output: Chassis Power is off 2023-09-30T01:38:54.721 DEBUG:teuthology.orchestra.console.smithi107:check power output: Chassis Power is off 2023-09-30T01:38:54.786 INFO:teuthology.orchestra.console.smithi088:Power off completed 2023-09-30T01:38:54.823 INFO:teuthology.orchestra.console.smithi107:Power off completed 2023-09-30T01:38:54.927 INFO:teuthology.run:Summary data: description: rados/monthrash/{ceph clusters/9-mons mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-hybrid rados supported-random-distro$/{ubuntu_latest} thrashers/many workloads/rados_mon_osdmap_prune} duration: 860.4375479221344 failure_reason: '"1696037575.6718426 mgr.x (mgr.4108) 1 : cluster [ERR] Unhandled exception from module ''pg_autoscaler'' while running on mgr.x: ''PgAutoscaler'' object has no attribute ''noautoscale''" in cluster log' flavor: default owner: scheduled_yuriw@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=d8ba4ebfdcfc436a8db1f3fa32bf4e29 status: fail success: false 2023-09-30T01:38:54.928 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-09-30T01:38:55.015 INFO:teuthology.run:FAIL